As I work on the last pieces needed to launch the YDN on Courant next month, I finally got around to implementing the email engine for Courant and the YDN site.
There were a few basic requirements necessitated by the quantity of emails that the YDN handles on a daily basis (on the order of several thousand subscribers):
- Process the email queue asynchronously from the web server process. Sending thousands of emails takes a long time, and we don’t want to tie up our web server during that time.
- When administrators/editors submit new mass email jobs (e.g., daily headlines), they should not have to wait around for the server to process that request. The page should quickly load and the processing should be done async/out-of-band.
- Since emails could potentially be customized, avoid reliance on bcc: field. Essentially, each user gets their own email object in the system.
For the email engine itself, I started with the django-mailer app as my foundation. It stores emails in a queue in the database, and a command-line management command is run by cron every x minutes to process queued emails. I also took some patches from various forks to handle reuse of the SMTP connection, streamlined HTML+Text email creation, and a limit on the number of emails the management command processes at one time.
On the interface side, I created a Django admin action for both issues and articles that will render templates for the text and HTML versions and then create a django-mailer Message instance for each subscriber. I quickly discovered that this only meets the above requirements for very small subscriber sets. With 5000 subscribers, it took about 45 seconds on my dev VM to generate all the emails and save them in the database; while that was happening, the browser would wait for a response, possibly timing out in the interim.
This clearly didn’t meet requirement #2, so I had to add another step to the process. When the admin action executes, it would create the email contents and store that in a new model, MessageJob, which had a simple TextField containing a delimited list of email addresses to send to. Then I created a new management command that will process MessageJobs into all the individual Messages when called by cron.
So now the execution flows looks like this (see images at end of post):
- Editor checks box next to an issue, selects “Send email update” admin action option, and hits “Go” (submits the action form)
- A MessageJob is generated containing the text and HTML versions of the email, the subject line, from address, and delimited list of recipients. Page loads for editor and displays message confirming that job has been queued.
- System cron soon calls “manage.py process_mail_jobs”, which pulls out any unprocessed MessageJobs and creates a new Message instance for each recipient.
- System cron soon calls “manage.py send_mail” which attempts to send all the Messages in the queue in the database. If any fail to send, they will be retried when cron calls “manage.py retry_deferred” (standard django-mailer behavior).
Those three cron jobs get called every minute (configurable if you want less frequently, though there is almost no penalty for calling them if the queues are empty), either by manually setting your user crontab or by using an app like django-chronograph (which will come pre-configured with Courant News).
So the modified django-mailer system with the admin actions integration meets all of the requirements we had defined for the email engine. The next step will be to implement the email subscription spec that Rob and I wrote back in December, though that work may be deferred to version 2.
I have not yet done extensive performance tests comparing use of a local mail server (e.g., Exim) vs. an external service (e.g., Gmail), but that’s something I’ll be exploring in the not-too-distant future. Even if the email sending process takes a long time, the users of the website and admin should not notice, and system performance should be well within acceptable bounds. User registration, password reset and other emails will have higher priorities and jump to the top of the queue, so even while processing a large subscription queue those critical messages will still be (almost) immediately sent out.
I’m holding off on checking this work into the public Courant News repository as I currently have a mess of Nando stuff interleaved, and I need to sort that out first. I also want to add some forms and views for handling creation of email subscriptions on the public website, so maybe once that is complete I’ll push through a checkin.