Max Cutler

On Sep­tem­ber 2nd, the Yale Daily News pub­lished its first issue of the fall 2009 semes­ter. Although appear­ing to the casual observer to be just another issue, there was one huge dif­fer­ence: it was run­ning on the new Courant News online pub­lish­ing plat­form. Just one week later, Yale grad­u­ate stu­dent Annie Le went miss­ing. The fol­low­ing ten days resulted in enor­mous national and inter­na­tional cov­er­age of the case and a record surge in traf­fic to our web­site. Courant News played a huge role in our out­stand­ing cov­er­age and lack of down­time dur­ing the traf­fic spikes.

Chronol­ogy

After being a miss­ing per­sons case for almost 5 days, Annie Le’s body was found and sud­denly it became a homi­cide inves­ti­ga­tion. When we pub­lished the break­ing news at 8:51pm, our server was imme­di­ately slammed by an unusu­ally large swell in traf­fic; in the few hours remain­ing in that Sun­day night, we had twice as many web­site vis­i­tors as we typ­i­cally get in an entire week­day. As the night wore on, I kept updat­ing our edi­to­r­ial staff on the impres­sive num­bers: 6,000; 8,000; 13,000 hits in 70 min­utes. I even­tu­ally went to bed proud that we had sur­vived the spike with­out any prob­lems, but I was in for a surprise.

Out of curios­ity before head­ing to break­fast on Mon­day morn­ing, I decided to check on the server’s health. Server uti­liza­tion was at 100%, and the server was really strain­ing. I imme­di­ately went to the Drudge Report and found that they had placed a link to us at the top of their front page. Out of all the national cov­er­age avail­able, the editor(s) at the Drudge Report had decided linked to us, and the flood gates were opened.

For the first six hours after Drudge post­ing the link, we received over 70,000 pageviews per hour before traf­fic slowly declined to “only” 30,000 pageviews per hour by the end of the day. In the 48 hours fol­low­ing, we han­dled a total of 1.1 mil­lion pageviews; in the 10 day period start­ing when she went miss­ing, we saw over 2 mil­lion pageviews. At peak traf­fic, we were serv­ing 30Mbps in data from our server, which con­tin­ued for sev­eral hours. Despite the 3000% increase in traf­fic, we had zero down­time and our site was fully oper­a­tional the entire time.

Courant News’ Role

In the last week of August, the YDN Editor-in-Chief and I decided to make the switch to Courant News for our first issue instead of wait­ing a bit longer to refine it a bit more. One of the aspects of Courant that had not yet been prop­erly tested was per­for­mance; Courant News was designed with the lessons learned from oper­at­ing a high traf­fic site and sur­viv­ing Drudge’s in the past, but no spe­cial effort had been made to opti­mize per­for­mance yet. For­tu­nately, the few steps that we had taken in the spring were suf­fi­cient, and the sys­tem per­formed like an absolute champ through­out the spikes.

Hourly traf­fic dur­ing Drudge Report cov­er­age (blue); Typ­i­cal week­day traf­fic (green)

One of the tricks that Courant employs is a full-page cache for anony­mous (non-logged-in) users: when an anony­mous user attempts to access a page, our load bal­ancer (nginx) serves them a cached ver­sion directly from RAM using mem­cached. Nginx is amaz­ing, and can han­dle enor­mous amounts of traf­fic with min­i­mal server resource usage; how­ever, Django (served by an Apache server instance), is more resource inten­sive, and would quickly be bogged down by too many simul­ta­ne­ous requests. Hav­ing nginx serve the full page caches to the swarms of anony­mous Drudge Report refer­rals meant that very few requests were being passed through to Courant (mostly our EIC and MEs who were adding new con­tent as the day went on).

Nginx requests/second log dur­ing Drudge Report spike

No down­time is great, but ulti­mately it’s all about the con­tent, and our edi­tors and reporters did an out­stand­ing job cov­er­ing the Annie Le case. Unlike many main stream media out­lets that pub­lished unsub­stan­ti­ated rumors, the Yale Daily News sup­plied top notch cov­er­age and pro­vided unique angles that only Yale stu­dents can pro­vide (such as pho­tos from inside the base­ment before it was iden­ti­fied as the scene of the crime).

One of the new fea­tures that Courant News brought to the YDN site was the abil­ity to post mul­ti­ple media ele­ments in a given arti­cle. We made judi­cious use of this capa­bil­ity, includ­ing upwards of three or four items on many arti­cles. Our old sys­tem only allowed a sin­gle photo on each arti­cle, which would have crip­pled our abil­ity to cover this story.

Another key capa­bil­ity was the abil­ity to use mul­ti­ple tem­plates for arti­cles and the home­page. We cre­ated a new “Big Photo” arti­cle tem­plate to high­light the top media item on many of our sto­ries. We also cre­ated a num­ber of new home­page tem­plates to high­light our break­ing news cov­er­age (note: because we only archive one ver­sion of the home­page per day, I can­not show  some of the addi­tional tem­plates we used).

To sup­port the amount of con­tent we were pub­lish­ing, Courant News allowed us to give lim­ited access to the admin­is­tra­tive inter­face to our Photo Edi­tors and some Pro­duc­tion & Design staffers, who helped the EIC and MEs upload con­tent and pub­lish new infor­ma­tion in a timely man­ner. This dis­tri­b­u­tion of work is some­thing that we would like to con­tinue going for­ward, elim­i­nat­ing the excuse of pub­lish­ing extra con­tent online being too much work at the end of the night.

Finally, Courant News included new email and ana­lyt­ics track­ing sys­tems, which allowed us to push break­ing news updates to our email sub­scribers and track engage­ment from emails and our Twit­ter updates. Such data nicely com­ple­ments our Google Ana­lyt­ics reports regard­ing read­er­ship engage­ment and has pro­vided insight into ways we can improve our cov­er­age in the future.

Con­clu­sion

Mov­ing to a new CMS and pub­lish­ing plat­form can be a risky endeavor, but Courant News has pulled its weight and played a crit­i­cal sup­port­ing role in our abil­ity to cover this story and sur­vive the mas­sive influx of traf­fic it brought us. With a promis­ing future ahead of it, includ­ing the dig­i­tal news­room and other excit­ing new fea­tures, Courant News will help us at the Yale Daily News inno­vate and exper­i­ment with our web­site in the com­ing years. I see many inter­est­ing projects in our future this year, and I look for­ward to help­ing other news orga­ni­za­tions take the next step and join us in devel­op­ing the Courant News plat­form for the bet­ter­ment of all col­lege news organizations.