Context

During the fall semester we made significant progress with Courant: switched to Haystack for search, improved multimedia browse-ability, added the headers app for improved mobile and alternative versions of a site, implemented powerful automated caching busting, refactored the staff app, added OpenX integration, added mapping and basic geo functionality, started work on Nando, added batch uploading of photos, and continuously increased performance.

But in early January, Rob and I had a serious conversation about whether it still made sense to develop Courant. Although the YDN was using it, no other developers had joined the team, and we had doubts about the project’s long-term survivability.

Our last best hope was to convince the team at CoPress to help support clients running on Courant. I met with them at their team meet-up in Philadelphia over MLK weekend and tried to feel out their intentions and plans for the next year. There was still some glimmer of hope, and I considered joining their support team to make this a reality.

Two weeks later, the YDN Business and Online departments hosted a conference for college news organizations, at which I gave a presentation about Courant. Two of the attendees were enthusiastic about the idea of migrating to Courant, and I’ve been in talks with them in the intervening weeks. We even wrote a CP4 to Courant data migration script while working on a prototype installation for one of the potential partners.

The Decision

Shortly thereafter, CoPress decided to shut down their operations. Without CoPress, and without any other potential developers to take over the reins, we had to make a tough decision: continue developing the project and potentially transition these two new sites to the platform without long-term support plans, or abandon the project.

Having worked on this project for over 16 months, pouring countless hours of my life into its development, it was an extraordinarily difficult decision. But, in the end, I feel it would be irresponsible to continue, especially knowing that I am going to have to leave anyways after my term at the YDN expires in October.

I’m currently working to transition the YDN off of Courant to a commercial system; although the front-runner is currently Ellington (from which Django was originally extracted), a final decision has yet to be made and it will likely be several months before I can talk publicly about our choice.

In the meantime, there may be occasional bug fixes to Courant, but substantial development of new features will not occur. We will leave the website and code online at least through the summer, and will likely move the code to Google Code or Github before retiring the website. Please feel free to use the code, but know that we will no longer be providing support of any form.

Conclusion

Had we pursued grant funding or other avenues of development, perhaps Courant could have lived up to its full potential. I truly believe it could have been a major player in changing the landscape of college news. I’m very proud of the work we did to create a generic and customizable, yet still very powerful, solution tailored specifically to the needs of college news organizations. Components like the get tag, dynamic/custom fields, the headers app, and the cache system are beyond the scope of most existing college news CMSes, and represent months of analysis and planning. The system was battle-tested against outrageous traffic spikes and proved to be quite resilient. We are not the first, and will surely not be the last, team to attempt to create a new CMS option for the college news world, and I wish others the best of luck.

I owe immense thanks to Rob and Paul for their contributions to the project over the last 18 months; their support has been invaluable, and I hope to work with them again in the future. I also want to thank Andrew Spittle for his assistance on designs for Nando, it’s unfortunate that they never saw the light of day. Thanks to everyone who has been supportive and encouraging over the past year, especially those who contributed to discussions on our mailing list and privately via email or IM.

I look forward to seeing how college news organizations respond to the current crisis in the media industry, and I hope they learn to adapt and look forward instead of backwards. My stint in the college news world has been enjoyable and a great learning experience, but I’m ready to apply my skills to new areas of interest. If you need an experienced developer for your software or robotics project, please get in touch.

Multimedia

Being both an early riser and three hours ahead by virtue of time zones, I started my day working on some YDN tasks and left Courant for the afternoon. First order of business was finishing the new multimedia landing page. We’ve been without that page since launching on Courant at the beginning of the semester, and the version before was barely functionality as it was. Zeke Miller, the new Online Editor, was the driving force for the design while brainstorming and gathering feedback from the rest of the Online desk. I’m very pleased with the results, but have opened a thread over at the CoPress forums to get feedback from the wider college news community.

Search

Next on the docket was finishing the transition from Sphinx to Haystack for the Courant News search system. Sphinx was very nice, but Haystack opens new doors and fits better with the Courant News project philosophy; it allows for the use of multiple search engine backends, so that sites can use whatever is most appropriate for their situation.

For example, if a site cannot install a powerful Java system like Solr, then they can use the pure-Python Whoosh backend (which is also nice for development use). In practice, I ran into problems while trying to use the Xapian (fast C++ engine) backend and had to switch back to the resource-heavy Solr. Hopefully as Haystack and its Xapian backend mature the situation will improve.

By the end of the afternoon, I had finished implementing it in Courant and had completed the templates for the YDN search system. Please feel free to explore. I’m especially proud of the filtering options, which allowing for filtering of the query results by section, staff members, content type (article, photo, video, etc.), and date range. It makes it very easy to find specific content that you are looking for, or even to just get a broader sense of how popular a given topic is.

Solr’s relevancy engine is giving disproportionate weight to photos because they aren’t as long as articles, which is not very desirable. However, I’m still a Solr newbie and am trying to explore options for resolving that issue. Once I figure out how to do it with Solr, I’ll have to push the Haystack folks to expose any requisite functionality.

Headers App

Rob’s big project, and his first major code contribution in several months, was the implementation of an app we’ve been designing for the past few days. He’s going to write a more in-depth post, so I won’t step on his toes too much and go too deeply into it right now; essentially, it allows you to vary the display of your news site based on HTTP header values, most importantly User-Agent and Referer [sic].

The biggest example is to show a different set of templates to visitors with mobile User-Agent strings by showing them .mobile templates (using the FileExtension middleware and render() function in Courant News). You can also define a cascade of extensions, so that you only have to customize certain parts of your site if you want, and show the standard .html version elsewhere; this could be interesting if you wanted to take advantage of new HTML5 functionality in browsers (User-Agents) that support it, while not having to duplicate templates on other pages (or if you want to customize only certain pages on your mobile site for iPhone users).

It also lets you simply set extra template context variables, so that you can show special content to people coming from certain referrers (e.g., Facebook or Twitter). There are countless uses for such functionality, I’ll let you come up with your own.

You can browse the headers app in our code repository; it only requires a bit more work before we will declare it complete, hopefully that will happen in the next several days. When Rob writes his post, I’ll update with a link.

Conclusion

Today was a productive day, with 9 commits to the Courant News repository. Hopefully the pace will keep up, and once these last few features are finalized the path will be clear to begin work on Nando, which will be the real cornerstone of the Courant News platform.

Chronology

After being a missing persons case for almost 5 days, Annie Le’s body was found and suddenly it became a homicide investigation. When we published the breaking news at 8:51pm, our server was immediately slammed by an unusually large swell in traffic; in the few hours remaining in that Sunday night, we had twice as many website visitors as we typically get in an entire weekday. As the night wore on, I kept updating our editorial staff on the impressive numbers: 6,000; 8,000; 13,000 hits in 70 minutes. I eventually went to bed proud that we had survived the spike without any problems, but I was in for a surprise.

Out of curiosity before heading to breakfast on Monday morning, I decided to check on the server’s health. Server utilization was at 100%, and the server was really straining. I immediately went to the Drudge Report and found that they had placed a link to us at the top of their front page. Out of all the national coverage available, the editor(s) at the Drudge Report had decided linked to us, and the flood gates were opened.

For the first six hours after Drudge posting the link, we received over 70,000 pageviews per hour before traffic slowly declined to “only” 30,000 pageviews per hour by the end of the day. In the 48 hours following, we handled a total of 1.1 million pageviews; in the 10 day period starting when she went missing, we saw over 2 million pageviews. At peak traffic, we were serving 30Mbps in data from our server, which continued for several hours. Despite the 3000% increase in traffic, we had zero downtime and our site was fully operational the entire time.

Courant News’ Role

In the last week of August, the YDN Editor-in-Chief and I decided to make the switch to Courant News for our first issue instead of waiting a bit longer to refine it a bit more. One of the aspects of Courant that had not yet been properly tested was performance; Courant News was designed with the lessons learned from operating a high traffic site and surviving Drudge’s in the past, but no special effort had been made to optimize performance yet. Fortunately, the few steps that we had taken in the spring were sufficient, and the system performed like an absolute champ throughout the spikes.

Hourly traffic during Drudge Report coverage (blue); Typical weekday traffic (green)

One of the tricks that Courant employs is a full-page cache for anonymous (non-logged-in) users: when an anonymous user attempts to access a page, our load balancer (nginx) serves them a cached version directly from RAM using memcached. Nginx is amazing, and can handle enormous amounts of traffic with minimal server resource usage; however, Django (served by an Apache server instance), is more resource intensive, and would quickly be bogged down by too many simultaneous requests. Having nginx serve the full page caches to the swarms of anonymous Drudge Report referrals meant that very few requests were being passed through to Courant (mostly our EIC and MEs who were adding new content as the day went on).

Nginx requests/second log during Drudge Report spike

No downtime is great, but ultimately it’s all about the content, and our editors and reporters did an outstanding job covering the Annie Le case. Unlike many main stream media outlets that published unsubstantiated rumors, the Yale Daily News supplied top notch coverage and provided unique angles that only Yale students can provide (such as photos from inside the basement before it was identified as the scene of the crime).

One of the new features that Courant News brought to the YDN site was the ability to post multiple media elements in a given article. We made judicious use of this capability, including upwards of three or four items on many articles. Our old system only allowed a single photo on each article, which would have crippled our ability to cover this story.

Another key capability was the ability to use multiple templates for articles and the homepage. We created a new “Big Photo” article template to highlight the top media item on many of our stories. We also created a number of new homepage templates to highlight our breaking news coverage (note: because we only archive one version of the homepage per day, I cannot show  some of the additional templates we used).

To support the amount of content we were publishing, Courant News allowed us to give limited access to the administrative interface to our Photo Editors and some Production & Design staffers, who helped the EIC and MEs upload content and publish new information in a timely manner. This distribution of work is something that we would like to continue going forward, eliminating the excuse of publishing extra content online being too much work at the end of the night.

Finally, Courant News included new email and analytics tracking systems, which allowed us to push breaking news updates to our email subscribers and track engagement from emails and our Twitter updates. Such data nicely complements our Google Analytics reports regarding readership engagement and has provided insight into ways we can improve our coverage in the future.

Conclusion

Moving to a new CMS and publishing platform can be a risky endeavor, but Courant News has pulled its weight and played a critical supporting role in our ability to cover this story and survive the massive influx of traffic it brought us. With a promising future ahead of it, including the digital newsroom and other exciting new features, Courant News will help us at the Yale Daily News innovate and experiment with our website in the coming years. I see many interesting projects in our future this year, and I look forward to helping other news organizations take the next step and join us in developing the Courant News platform for the betterment of all college news organizations.

YDN Headlines Email Screenshot

YDN

Since the launch of the current YDN website in January 2007, we have had a headlines email subscription available for public consumption. Visitors could either sign up with just an email address, or if they had a YDN.com account there was an option in their user settings to subscribe. We currently have several thousand subscribers to our email list, which encompasses both the daily (nightly) headlines and breaking news stories. To the right is the email for today’s issue.

As I’ve discussed before, our website currently runs on a custom system built on the CakePHP framework. Because sending thousands of emails at once is a time consuming process, we use a mail queue system to spread out the work. When the editors click the “Email headlines” link in the admin interface at the end of the night, we queue up all the emails in a table in our database. We then use a cron job to pull out a certain number of emails from the queue each minute and send them through our outgoing mail server. It can take up to 30 or 40 minutes to process all the emails in this manner, but it prevents overloading or slowing of the server, and the emails are not really time sensitive.

Courant News

For Courant News, we have also planned to implement a queue-based mail system. django-mailer plans to do alot of what we want, but is still in an early stage of development. It is part of the popular Pinax project, so I’m sure development will continue for the forseeable future. The queue is just part of the email system, however, and we have some ideas about how to make emails more useful and customizable.

The basic premise is that we wanted people to be able to subscribe to emails like they can subscribe to RSS feeds, where they can choose to only receive content from certain sections, or by certain authors, or with certain tags, etc, etc. For example, a parent of one of the staff reporters might want to get an email digest of any articles his/her son/daughter writes, while not necessarily wanting to have to filter through all the rest of that day’s paper’s contents. Or someone may only be interested in news in a certain section, such as Sports. The user would also be able to choose a periodicity of the emails, choosing to receive such aggregations daily, weekly, or monthly.

While many younger people are comfortable using RSS feeds, many less tech-savvy students, alumni, and parents are still unaware of or unsure how to use RSS feeds. Others, like myself, may simply prefer emails to digging through an RSS client. By giving such visitors the ability to subscribe to emails with exactly the content they want, we can increase the likelihood that they will follow those links to our site and become frequent visitors. By not forcing them to sift through a bunch of links they probably don’t care about, we can reduce the friction associated with browsing our content and help them find what they are looking for.

We’ve currently been exploring how to implement such functionality, and have several ideas. Email alerts/subscriptions are one of the next things on the docket for actual implemention for Courant News, so I welcome any feedback or additional ideas and suggestions from the community.

Sometime last summer, Robert Baskin (@rsbaskin), my former “boss” and mentor at the Yale Daily News, and I talked about an idea we had for a startup company: create an online publishing platform for college news organizations. The YDN has run on a custom CakePHP-based CMS since January 2007, which Rob built with Henry Corrigan-Gibbs from scratch. Last spring (April 2008), we held a web conference for the Ivy League newspapers, and the big takeaway was that open source solutions were not good enough; we needed a CMS designed for college news orgs, by college news orgs. College Publisher was a blight upon the college news landscape, and we thought we could create a viable alternative.

Fast forward a few months, and we brought Paul O’Shannessy on board. Paul was finishing his masters degree at CMU where he worked for the TCPulse. With dreams of a successful company that Paul and Rob could work for after graduating this year, we worked hard for several months, and even had a week-long meetup and code session in the Philadelphia area over our winter break. We courted a few clients, and started refining our marketing materials. We had designed and started building a hosted platform-as-a-service, which was designed to be fully managable through a web-based admin interface ,while still allowing programmers the ability to add new functionality thanks to Django‘s awesome app-based design.

And then we looked harder at the numbers, and thought about why we were doing this, and it all started to unravel. After some long discussions, we decided to abandon the business venture, and we all agreed to allow the code we had labored over to go into the open source realm. Around this time, in late January, I approached the YDN formally about switching to the new system. It had been built with the lessons-learned from the YDN’s existing CMS, and encompassed a number of improvements that the Editorial staff had requested. The decision was made to move forward with switching to the new platform, but only under the condition that it launched right after spring break to give time for our new blog to take off before the end of the semester.

With money no longer in the picture, and the reality of the senior essays and full time jobs coming to the fore, Rob and Paul had to step away from the project, leaving me to work on it mostly alone. I took some time off in February to focus on school, but returned and worked hard during March to get Courant and the new YDN site up to snuff for launch. As recently as last week, I thought I could maybe finish things in time for our planned April 5 transition. Once the YDN site was up, I’d focus on truly open sourcing Courant and getting the ball rolling on that front.

A few days ago, I realized that I wasn’t going to finish in time, and that I couldn’t, in good conscience, put the site out there in its current state. We have a working, stable website now, and there was no real reason to rush the launch other than for the sake of the new blog and some new features. As the sole active member of the YDN web staff, I had to make the hard decision to postpone the launch until later this summer to allow for finishing all the features and doing proper testing.

What this means for Courant is that we no longer have our pioneer site at release, but it means I can give the project more of my attention and get other people involved. I will be attending the BarCamp NewsInnovation in Philly this month, and my goal is to have the project site fully up by then, including posting all of the documentation and plans from when the project was in the commercial startup phase. Courant News isn’t quite ready to support a full site yet, but it’s close and just requires a few more weeks of work. Hopefully, if other developers and UX people get involved, the project can begin to generate a community of its own and gain some traction.  We’ve tried our best to make Courant customizable and suitable for just about any type of college news website, and have a few features that I’ve not seen elsewhere. In the coming days and weeks, I’ll begin posting about them here and on the official Courant News blog once I find time to set that up. Thanks to all who have expressed in interest in Courant, I’ll be sure to keep the news flowing on a regular basis from here on out.

Update (April 6, 2009): At the request of the YDN, I have removed some details about internal affairs.