On September 2nd, the Yale Daily News published its first issue of the fall 2009 semester. Although appearing to the casual observer to be just another issue, there was one huge difference: it was running on the new Courant News online publishing platform. Just one week later, Yale graduate student Annie Le went missing. The following ten days resulted in enormous national and international coverage of the case and a record surge in traffic to our website. Courant News played a huge role in our outstanding coverage and lack of downtime during the traffic spikes.
Chronology
After being a missing persons case for almost 5 days, Annie Le’s body was found and suddenly it became a homicide investigation. When we published the breaking news at 8:51pm, our server was immediately slammed by an unusually large swell in traffic; in the few hours remaining in that Sunday night, we had twice as many website visitors as we typically get in an entire weekday. As the night wore on, I kept updating our editorial staff on the impressive numbers: 6,000; 8,000; 13,000 hits in 70 minutes. I eventually went to bed proud that we had survived the spike without any problems, but I was in for a surprise.
Out of curiosity before heading to breakfast on Monday morning, I decided to check on the server’s health. Server utilization was at 100%, and the server was really straining. I immediately went to the Drudge Report and found that they had placed a link to us at the top of their front page. Out of all the national coverage available, the editor(s) at the Drudge Report had decided linked to us, and the flood gates were opened.
For the first six hours after Drudge posting the link, we received over 70,000 pageviews per hour before traffic slowly declined to “only” 30,000 pageviews per hour by the end of the day. In the 48 hours following, we handled a total of 1.1 million pageviews; in the 10 day period starting when she went missing, we saw over 2 million pageviews. At peak traffic, we were serving 30Mbps in data from our server, which continued for several hours. Despite the 3000% increase in traffic, we had zero downtime and our site was fully operational the entire time.
Courant News’ Role
In the last week of August, the YDN Editor-in-Chief and I decided to make the switch to Courant News for our first issue instead of waiting a bit longer to refine it a bit more. One of the aspects of Courant that had not yet been properly tested was performance; Courant News was designed with the lessons learned from operating a high traffic site and surviving Drudge’s in the past, but no special effort had been made to optimize performance yet. Fortunately, the few steps that we had taken in the spring were sufficient, and the system performed like an absolute champ throughout the spikes.
Hourly traffic during Drudge Report coverage (blue); Typical weekday traffic (green)
One of the tricks that Courant employs is a full-page cache for anonymous (non-logged-in) users: when an anonymous user attempts to access a page, our load balancer (nginx) serves them a cached version directly from RAM using memcached. Nginx is amazing, and can handle enormous amounts of traffic with minimal server resource usage; however, Django (served by an Apache server instance), is more resource intensive, and would quickly be bogged down by too many simultaneous requests. Having nginx serve the full page caches to the swarms of anonymous Drudge Report referrals meant that very few requests were being passed through to Courant (mostly our EIC and MEs who were adding new content as the day went on).
Nginx requests/second log during Drudge Report spike
No downtime is great, but ultimately it’s all about the content, and our editors and reporters did an outstanding job covering the Annie Le case. Unlike many main stream media outlets that published unsubstantiated rumors, the Yale Daily News supplied top notch coverage and provided unique angles that only Yale students can provide (such as photos from inside the basement before it was identified as the scene of the crime).
One of the new features that Courant News brought to the YDN site was the ability to post multiple media elements in a given article. We made judicious use of this capability, including upwards of three or four items on many articles. Our old system only allowed a single photo on each article, which would have crippled our ability to cover this story.
Another key capability was the ability to use multiple templates for articles and the homepage. We created a new “Big Photo” article template to highlight the top media item on many of our stories. We also created a number of new homepage templates to highlight our breaking news coverage (note: because we only archive one version of the homepage per day, I cannot show some of the additional templates we used).
To support the amount of content we were publishing, Courant News allowed us to give limited access to the administrative interface to our Photo Editors and some Production & Design staffers, who helped the EIC and MEs upload content and publish new information in a timely manner. This distribution of work is something that we would like to continue going forward, eliminating the excuse of publishing extra content online being too much work at the end of the night.
Finally, Courant News included new email and analytics tracking systems, which allowed us to push breaking news updates to our email subscribers and track engagement from emails and our Twitter updates. Such data nicely complements our Google Analytics reports regarding readership engagement and has provided insight into ways we can improve our coverage in the future.
Conclusion
Moving to a new CMS and publishing platform can be a risky endeavor, but Courant News has pulled its weight and played a critical supporting role in our ability to cover this story and survive the massive influx of traffic it brought us. With a promising future ahead of it, including the digital newsroom and other exciting new features, Courant News will help us at the Yale Daily News innovate and experiment with our website in the coming years. I see many interesting projects in our future this year, and I look forward to helping other news organizations take the next step and join us in developing the Courant News platform for the betterment of all college news organizations.
