Context

During the fall semester we made significant progress with Courant: switched to Haystack for search, improved multimedia browse-ability, added the headers app for improved mobile and alternative versions of a site, implemented powerful automated caching busting, refactored the staff app, added OpenX integration, added mapping and basic geo functionality, started work on Nando, added batch uploading of photos, and continuously increased performance.

But in early January, Rob and I had a serious conversation about whether it still made sense to develop Courant. Although the YDN was using it, no other developers had joined the team, and we had doubts about the project’s long-term survivability.

Our last best hope was to convince the team at CoPress to help support clients running on Courant. I met with them at their team meet-up in Philadelphia over MLK weekend and tried to feel out their intentions and plans for the next year. There was still some glimmer of hope, and I considered joining their support team to make this a reality.

Two weeks later, the YDN Business and Online departments hosted a conference for college news organizations, at which I gave a presentation about Courant. Two of the attendees were enthusiastic about the idea of migrating to Courant, and I’ve been in talks with them in the intervening weeks. We even wrote a CP4 to Courant data migration script while working on a prototype installation for one of the potential partners.

The Decision

Shortly thereafter, CoPress decided to shut down their operations. Without CoPress, and without any other potential developers to take over the reins, we had to make a tough decision: continue developing the project and potentially transition these two new sites to the platform without long-term support plans, or abandon the project.

Having worked on this project for over 16 months, pouring countless hours of my life into its development, it was an extraordinarily difficult decision. But, in the end, I feel it would be irresponsible to continue, especially knowing that I am going to have to leave anyways after my term at the YDN expires in October.

I’m currently working to transition the YDN off of Courant to a commercial system; although the front-runner is currently Ellington (from which Django was originally extracted), a final decision has yet to be made and it will likely be several months before I can talk publicly about our choice.

In the meantime, there may be occasional bug fixes to Courant, but substantial development of new features will not occur. We will leave the website and code online at least through the summer, and will likely move the code to Google Code or Github before retiring the website. Please feel free to use the code, but know that we will no longer be providing support of any form.

Conclusion

Had we pursued grant funding or other avenues of development, perhaps Courant could have lived up to its full potential. I truly believe it could have been a major player in changing the landscape of college news. I’m very proud of the work we did to create a generic and customizable, yet still very powerful, solution tailored specifically to the needs of college news organizations. Components like the get tag, dynamic/custom fields, the headers app, and the cache system are beyond the scope of most existing college news CMSes, and represent months of analysis and planning. The system was battle-tested against outrageous traffic spikes and proved to be quite resilient. We are not the first, and will surely not be the last, team to attempt to create a new CMS option for the college news world, and I wish others the best of luck.

I owe immense thanks to Rob and Paul for their contributions to the project over the last 18 months; their support has been invaluable, and I hope to work with them again in the future. I also want to thank Andrew Spittle for his assistance on designs for Nando, it’s unfortunate that they never saw the light of day. Thanks to everyone who has been supportive and encouraging over the past year, especially those who contributed to discussions on our mailing list and privately via email or IM.

I look forward to seeing how college news organizations respond to the current crisis in the media industry, and I hope they learn to adapt and look forward instead of backwards. My stint in the college news world has been enjoyable and a great learning experience, but I’m ready to apply my skills to new areas of interest. If you need an experienced developer for your software or robotics project, please get in touch.

We performed survey research on eight of the most discussed new business models for news, although we tried not to prescribe any of them as “the” solution. As countless others have pointed out, there will not be any single solution that will work for any or all news organizations, so it is instead important to understand how each might contribute to a new news organization structure. We hope to update the site periodically as events unfold in the industry, and will continue to post to our Publish2 newsgroup as we come across relevant new content.

We hope our research and summaries are useful to interested observers, especially those who may not have been following the issues at hand as closely as we have. Feel free to give feedback and let us know what you think.

Session #1: Who Uses the News and How?

Session #2: Preserving Local Journalism

Session #3: Publicly Owned and Operated Media

Session #4: Quest for Pay Models (minus most of Q&A)

This page will be updated throughout the day.

Full-page caching

The simplest caching strategy is to cache the output of entire pages and serve them directly to users before they hit your real code.

The recommended deployment strategy for Courant News employs an nginx proxy server in front of Apache running mod_wsgi. Fortunately, nginx has a module for memcached integration which can be used to return objects directly from memcached without hitting Apache (and all the relatively heavy Python code) at all.

Our nginx config file might contain the following:

location /
{
  if ($request_method = POST) {
    proxy_pass http://backend;
    break;
  }

  if ($http_cookie ~* "sessionid=.{32}") {
    proxy_pass http://backend;
    break;
  }
  add_header Memcached True;
  default_type "text/html; charset=utf-8";
  set $memcached_key "/ydn-$uri";
  memcached_pass localhost:11211;
  error_page 404 502 = /django;
}

location /django {
  proxy_pass http://backend;
  include /etc/nginx/proxy.conf;
  break;
}

What this does is check memcached for a fullpage cache if this is not a POST request (form submission) and the user is not logged in (has a session cookie). If it can’t find the entry in the cache, it just passes it along to Apache.

When the YDN got Drudge’d, all the users were anonymous users, and nginx could serve them the cached pages in its sleep. The site remained responsive for the logged-in staff members who had to update content and moderate comments.

Full-page caching is great, but most sites customize the site for visitors with site accounts.

Template Caching

If nginx doesn’t serve up a full-page cache, the request gets passed on to the actual code powering the site. The final HTML is generated by processing template files and then returned to the visitor’s browser.

Because there is often a fair amount of logic and database querying involved with processing the templates, the server load can be high if every visitor had to regenerate the final HTML every time. So we can cache fragments of our templates that we know involve alot of work and avoid that overhead.

Django has a template tag to handle this, and it does a pretty good job. The tag is given an expiration time, a name, and a set of extra variables that the cache should vary on. Courant News’ caching system is based on Django’s, but includes two extra pieces of functionality: anti-dogpiling and automatic cache invalidation.

Anti-Dogpiling

Caching can help performance greatly, but the cache itself has to be created before it can be used. Generally, one wants a cache to be created the first time it is viewed.

However, page views are not instantaneous; it may take several tens or hundreds of milliseconds to process a page and create a cached version of it. During this time, many more visitors will hit the same page, not find a cached entry, and want to create the cache themselves. All of a sudden, you have dozens of people generating the same cache, until eventually one of them finishes and subsequent visitors can use the cached version.

This can cause undue burden on the server and lead to performance troubles if a cache expires in the middle of a large traffic spike. Often we are simply invalidating an existing cache instead of creating a new one from scratch. In that case, we can do soft-expiration of the cache.

Normally when a cache is invalidated, it is simply deleted altogether, and future requests for it will cause it to be regenerated. Instead, we can have only the first person after a cache expiration generate a new version, while giving subsequent visitors the old cache until the new one is done being generated.

This is what we have implemented in Courant News, originally based on this snippet. It essentially uses an additional cache key to create a second, earlier, soft expiration time, and uses some logic to determine whether a given user should generate the new cache or be served the stale copy.

Cache Invalidation

Most sites don’t stay the same forever, so at some point it is necessary to clear the caches so that new content or modifications can appear on the site.

There’s a famous quote by Phil Karlton that “[t]here are only two hard things in Computer Science: cache invalidation and naming things.”

Because there may be links to a given article on dozens of pages on your site, there is no easy way to know which of those pages need to be regenerated when that article’s headline changes or a new article should replace it.

One solution is to simply set a low cache duration, so that the cache expires in a short amount of time (e.g., 30 minutes or an hour). However, what if you have breaking news that needs to appear immediately? Or what if you have an embarrassing typo in an article that you need to fix ASAP?

A really simple answer is to have the ability to flush the entire site cache. This indiscriminately deletes all entries in the cache, which can cause a sudden spike in server usage as all your visitors cause the caches to be regenerated.

This is what we’ve been using at the YDN for the past few years, and this is workable but a bit overkill.

Automated Template Fragment Cache Invalidation

Ideally, we would prefer that the system magically know what pages contained what content objects, and then whenever those objects changed would invalidate all of those caches. This is an extraordinarily difficult problem to solve for all cases, but we have attempted to solve the most common cases, and leave full cache clearing as a fallback for the unusual cases.

Our implementation hinges on the concept of tracking the “dependencies” of each template fragment. Because it is impossible to algorithmically inspect a template’s output and determine which content objects were used to generate it, we instead rely on the template author to provide some explicit help.

If a template cache block depends on only a single object, template authors can use an extra ‘for’ clause on the end of the cache tag:

{% cache 3600 article_page article.id for article %}
...
{% endcache %}

This tells the caching system that, if the object called ‘article’ changes, it should invalidate this cache block.

However, often a block will contain many objects, such as a section listing or most popular box. In those cases, additional ‘cache_dep’ template tags can be used inside the ‘cache’ tag to register these additional objects.

{% cache 3600 section_age section.id for section %}
  <ul>
    {% for article in section.articles.all %}
      {% cache_dep article %}
      <li>{{ article.heading }}</li>
    {% endfor %}
  </ul>
{% endcache %}

In this case, if ‘section’ or any of its articles get changed, this template fragment cache will be invalidated and regenerated.

While this adds some extra clutter to templates, we feel it is a reasonable trade-off if performance is of importance to a site.

In any case, the caching system will listen for post_save signals from objects and then look them up in a special database table. If we find any template fragments depending on this object, we invalidate them, and we invalidate the full-page cache for that fragment’s page.

We have also included an admin action that allow editors to manually tell an object to expire its caches.

Conclusion

If there is anything I’ve learned from my time at the YDN the past 2+ years, it’s been that caching is one of the most important aspects of designing a high traffic website. We’ve taken our years of experience optimizing caching for a news site and implemented a system for Courant News that will allow other sites using the platform take advantage of our experience and ensure their sites can survive large traffic spikes with aplomb.

In the future I’ll write more about our recommended deployment environment for Courant News and how that can influence site performance. If you are curious, you can find the Courant News caching code on our code site.

Multimedia

Being both an early riser and three hours ahead by virtue of time zones, I started my day working on some YDN tasks and left Courant for the afternoon. First order of business was finishing the new multimedia landing page. We’ve been without that page since launching on Courant at the beginning of the semester, and the version before was barely functionality as it was. Zeke Miller, the new Online Editor, was the driving force for the design while brainstorming and gathering feedback from the rest of the Online desk. I’m very pleased with the results, but have opened a thread over at the CoPress forums to get feedback from the wider college news community.

Search

Next on the docket was finishing the transition from Sphinx to Haystack for the Courant News search system. Sphinx was very nice, but Haystack opens new doors and fits better with the Courant News project philosophy; it allows for the use of multiple search engine backends, so that sites can use whatever is most appropriate for their situation.

For example, if a site cannot install a powerful Java system like Solr, then they can use the pure-Python Whoosh backend (which is also nice for development use). In practice, I ran into problems while trying to use the Xapian (fast C++ engine) backend and had to switch back to the resource-heavy Solr. Hopefully as Haystack and its Xapian backend mature the situation will improve.

By the end of the afternoon, I had finished implementing it in Courant and had completed the templates for the YDN search system. Please feel free to explore. I’m especially proud of the filtering options, which allowing for filtering of the query results by section, staff members, content type (article, photo, video, etc.), and date range. It makes it very easy to find specific content that you are looking for, or even to just get a broader sense of how popular a given topic is.

Solr’s relevancy engine is giving disproportionate weight to photos because they aren’t as long as articles, which is not very desirable. However, I’m still a Solr newbie and am trying to explore options for resolving that issue. Once I figure out how to do it with Solr, I’ll have to push the Haystack folks to expose any requisite functionality.

Headers App

Rob’s big project, and his first major code contribution in several months, was the implementation of an app we’ve been designing for the past few days. He’s going to write a more in-depth post, so I won’t step on his toes too much and go too deeply into it right now; essentially, it allows you to vary the display of your news site based on HTTP header values, most importantly User-Agent and Referer [sic].

The biggest example is to show a different set of templates to visitors with mobile User-Agent strings by showing them .mobile templates (using the FileExtension middleware and render() function in Courant News). You can also define a cascade of extensions, so that you only have to customize certain parts of your site if you want, and show the standard .html version elsewhere; this could be interesting if you wanted to take advantage of new HTML5 functionality in browsers (User-Agents) that support it, while not having to duplicate templates on other pages (or if you want to customize only certain pages on your mobile site for iPhone users).

It also lets you simply set extra template context variables, so that you can show special content to people coming from certain referrers (e.g., Facebook or Twitter). There are countless uses for such functionality, I’ll let you come up with your own.

You can browse the headers app in our code repository; it only requires a bit more work before we will declare it complete, hopefully that will happen in the next several days. When Rob writes his post, I’ll update with a link.

Conclusion

Today was a productive day, with 9 commits to the Courant News repository. Hopefully the pace will keep up, and once these last few features are finalized the path will be clear to begin work on Nando, which will be the real cornerstone of the Courant News platform.

Chronology

After being a missing persons case for almost 5 days, Annie Le’s body was found and suddenly it became a homicide investigation. When we published the breaking news at 8:51pm, our server was immediately slammed by an unusually large swell in traffic; in the few hours remaining in that Sunday night, we had twice as many website visitors as we typically get in an entire weekday. As the night wore on, I kept updating our editorial staff on the impressive numbers: 6,000; 8,000; 13,000 hits in 70 minutes. I eventually went to bed proud that we had survived the spike without any problems, but I was in for a surprise.

Out of curiosity before heading to breakfast on Monday morning, I decided to check on the server’s health. Server utilization was at 100%, and the server was really straining. I immediately went to the Drudge Report and found that they had placed a link to us at the top of their front page. Out of all the national coverage available, the editor(s) at the Drudge Report had decided linked to us, and the flood gates were opened.

For the first six hours after Drudge posting the link, we received over 70,000 pageviews per hour before traffic slowly declined to “only” 30,000 pageviews per hour by the end of the day. In the 48 hours following, we handled a total of 1.1 million pageviews; in the 10 day period starting when she went missing, we saw over 2 million pageviews. At peak traffic, we were serving 30Mbps in data from our server, which continued for several hours. Despite the 3000% increase in traffic, we had zero downtime and our site was fully operational the entire time.

Courant News’ Role

In the last week of August, the YDN Editor-in-Chief and I decided to make the switch to Courant News for our first issue instead of waiting a bit longer to refine it a bit more. One of the aspects of Courant that had not yet been properly tested was performance; Courant News was designed with the lessons learned from operating a high traffic site and surviving Drudge’s in the past, but no special effort had been made to optimize performance yet. Fortunately, the few steps that we had taken in the spring were sufficient, and the system performed like an absolute champ throughout the spikes.

Hourly traffic during Drudge Report coverage (blue); Typical weekday traffic (green)

One of the tricks that Courant employs is a full-page cache for anonymous (non-logged-in) users: when an anonymous user attempts to access a page, our load balancer (nginx) serves them a cached version directly from RAM using memcached. Nginx is amazing, and can handle enormous amounts of traffic with minimal server resource usage; however, Django (served by an Apache server instance), is more resource intensive, and would quickly be bogged down by too many simultaneous requests. Having nginx serve the full page caches to the swarms of anonymous Drudge Report referrals meant that very few requests were being passed through to Courant (mostly our EIC and MEs who were adding new content as the day went on).

Nginx requests/second log during Drudge Report spike

No downtime is great, but ultimately it’s all about the content, and our editors and reporters did an outstanding job covering the Annie Le case. Unlike many main stream media outlets that published unsubstantiated rumors, the Yale Daily News supplied top notch coverage and provided unique angles that only Yale students can provide (such as photos from inside the basement before it was identified as the scene of the crime).

One of the new features that Courant News brought to the YDN site was the ability to post multiple media elements in a given article. We made judicious use of this capability, including upwards of three or four items on many articles. Our old system only allowed a single photo on each article, which would have crippled our ability to cover this story.

Another key capability was the ability to use multiple templates for articles and the homepage. We created a new “Big Photo” article template to highlight the top media item on many of our stories. We also created a number of new homepage templates to highlight our breaking news coverage (note: because we only archive one version of the homepage per day, I cannot show  some of the additional templates we used).

To support the amount of content we were publishing, Courant News allowed us to give limited access to the administrative interface to our Photo Editors and some Production & Design staffers, who helped the EIC and MEs upload content and publish new information in a timely manner. This distribution of work is something that we would like to continue going forward, eliminating the excuse of publishing extra content online being too much work at the end of the night.

Finally, Courant News included new email and analytics tracking systems, which allowed us to push breaking news updates to our email subscribers and track engagement from emails and our Twitter updates. Such data nicely complements our Google Analytics reports regarding readership engagement and has provided insight into ways we can improve our coverage in the future.

Conclusion

Moving to a new CMS and publishing platform can be a risky endeavor, but Courant News has pulled its weight and played a critical supporting role in our ability to cover this story and survive the massive influx of traffic it brought us. With a promising future ahead of it, including the digital newsroom and other exciting new features, Courant News will help us at the Yale Daily News innovate and experiment with our website in the coming years. I see many interesting projects in our future this year, and I look forward to helping other news organizations take the next step and join us in developing the Courant News platform for the betterment of all college news organizations.

Background

As you might have gathered from my other blog posts, I am, first and foremost, a developer. I’ve done stints as the combo designer-and-developer for sites, but I’ve learned that my design skills are lacking and have tried to find more talented partners when working on projects that require interfaces.

Last week, the first site was launched on the Courant News platform. There are still a slew of template bugs that I’m working through, but the platform has performed admirably over the past week. It currently uses a more-or-less stock Django admin interface with the grappelli skin. However, the ultimate goal is that of the digital newsroom and Nando.

The Job

I am seeking a web/UX designer to work with me on designing the Nando interface. For those not familiar, Nando is the codename for the admin interface in the Courant News publishing platform for college news sites; it combines standard CRUD content management with a digital newsroom and the ability to manage an organization’s site from the web interface without touching code. For full details, you can read through the spec and the first round of community comments.

Because Courant News is an open source project, we don’t have the money to pay a huge amount for this work, but I am willing to work out monetary and other forms of compensation and negotiate terms of the work with interested individuals. When Nando is complete, Courant News will represent one of the best options for college news sites who embrace the web-first mentality and want to push the boundaries of online journalism; by working on designing Nando, your mark will be left on all of these endeavors and will touch members of the next generation of journalists.

Contact Me

So if you are interested or have questions, please send me an email (maxcutler AT gmail DOT com) or find me on AIM (S/N: Goaliexam).

There is certainly no shortage of short URL services in existence, with bit.ly currently being the most popular. However, as the recent existential crisis at tr.im has shown, trusting these services is sometimes not the best idea. In addition, there are numerous advantages (enumerated elsewhere) to hosting your short URLs on your own domain name.

Referral Tracking

As news orgs strive to find ways of attracting traffic to their websites, evaluating the effectiveness of distribution through different mediums is critical. How much traffic comes from email newsletters? Twitter? Facebook? the print edition?

Some or all of such mediums will not pass HTTP referrer headings, which means it is difficult or impossible to track using normal methodology. If you are using Google Analytics or similar service, the best you can likely do is add an HTTP GET parameter to the end of the link and convert that on the server side. However, it is easy for users to strip this off and is far from foolproof.

Alternatively, we can generate a different short URL for each of the target mediums and then track hits before/during the redirection step. That’s the route we chose for our app.

Short URL Types

There are a few strategies to generating actual short URL strings. Most of the popular services just use incremental keys that are encoded in base-32 or similar (converts a number to a string consisting of A-Z, a-z, 0–9 and perhaps some punctuation characters).

That works well with huge volume, as a single individual is unlikely to generate consecutive URL tokens. However, when you host your own URL shortener, all URLs generated on a given day will look very similar using this approach, which may or may not be problematic.

I, personally, prefer more diverse URLs for tokens created in a sequence. Since the point of per-medium URLs is to reliably track referrers, we don’t want users to easily guess what the URL for another medium is and thus spoof our service (or just accidentally mistype them).

Therefore, our short URL app supports both generation types: consecutive sequence (GHn8n, GHn8o, GHn8p) or a pseudo-random generator (GHn8n, SfMgo, ednto).

If you want a custom URL for a special case, you can manually create a new short URL through the admin interface and manually specify whatever URL value you desire.

Primary vs. Secondary Domain

The YDN has a very long domain name, so we want to use a shorter domain name (goydn.com) for our short URLs. However, some orgs, such as David’s DailyUW.com may want to just use their normal domain. Thus our app supports both scenarios.

If you want to use your primary domain, we recommend you put it under some prefix, such as /go/ or /g/, to avoid collisions with other valid URLs (e.g., /news/). However, if you want to play with fire, you can configure your short URLs to work at the top level of your primary domain without problems.

Do note that short URLs will only get checked if the page requested would have otherwise returned a 404 error.

App Models

The first model in our app is ShorturlMedium, which is simply a named object representing a given medium like “Print”, “Twitter” or whatever your org needs.

The core of our app is the Shorturl model, which maps a ShorturlMedium to a content object (using Django’s content type app) and stores the actual short URL.

Finally, we have the ShorturlHit model, which tracks hits for a given per-medium short URL. I have not yet generated any report generation tools, but that should be an easy work item in the future and will surely be a part of Nando.

Template Tags

To put short URL links on your pages, we have a simple template tag to get (and generate, if necessary) one for a given content object (article, page, event, etc.).

{% load shorturls %}
{% shorturl obj Twitter %}

That’s it. The first parameter is the content object variable, and the second is the medium you want it to run in.

Conclusion

Our short URL app is pretty simple, but is configurable to meet the needs of most college news deployment scenarios. Because it ties into your content behind the scenes, it won’t break when you change your normal URL structures and supports any new models you add to the system. In a few months, after we use the system in production, I’ll post a followup analyzing the analytics data we gather from it.

Code can be found in the Courant News code repo. Parts were based on django-ittybitty, but we strayed far enough away in purpose and implementation that it didn’t make sense to just treat our code as a patch. Let me know if you have any questions.

Goals/Requirements

There were a few basic requirements necessitated by the quantity of emails that the YDN handles on a daily basis (on the order of several thousand subscribers):

1. Process the email queue asynchronously from the web server process. Sending thousands of emails takes a long time, and we don’t want to tie up our web server during that time.
2. When administrators/editors submit new mass email jobs (e.g., daily headlines), they should not have to wait around for the server to process that request. The page should quickly load and the processing should be done async/out-of-band.
3. Since emails could potentially be customized, avoid reliance on bcc: field. Essentially, each user gets their own email object in the system.

Implementation

For the email engine itself, I started with the django-mailer app as my foundation. It stores emails in a queue in the database, and a command-line management command is run by cron every x minutes to process queued emails. I also took some patches from various forks to handle reuse of the SMTP connection, streamlined HTML+Text email creation, and a limit on the number of emails the management command processes at one time.

On the interface side, I created a Django admin action for both issues and articles that will render templates for the text and HTML versions and then create  a django-mailer Message instance for each subscriber. I quickly discovered that this only meets the above requirements for very small subscriber sets. With 5000 subscribers, it took about 45 seconds on my dev VM to generate all the emails and save them in the database; while that was happening, the browser would wait for a response, possibly timing out in the interim.

This clearly didn’t meet requirement #2, so I had to add another step to the process. When the admin action executes, it would create the email contents and store that in a new model, MessageJob, which had a simple TextField containing a delimited list of email addresses to send to. Then I created a new management command that will process MessageJobs into all the individual Messages when called by cron.

So now the execution flows looks like this (see images at end of post):

1. Editor checks box next to an issue, selects “Send email update” admin action option, and hits “Go” (submits the action form)
2. A MessageJob is generated containing the text and HTML versions of the email, the subject line, from address, and delimited list of recipients. Page loads for editor and displays message confirming that job has been queued.
3. System cron soon calls “manage.py process_mail_jobs”, which pulls out any unprocessed MessageJobs and creates a new Message instance for each recipient.
4. System cron soon calls “manage.py send_mail” which attempts to send all the Messages in the queue in the database. If any fail to send, they will be retried when cron calls “manage.py retry_deferred” (standard django-mailer behavior).

Those three cron jobs get called every minute (configurable if you want less frequently, though there is almost no penalty for calling them if the queues are empty), either by manually setting your user crontab or by using an app like django-chronograph (which will come pre-configured with Courant News).

Conclusion

So the modified django-mailer system with the admin actions integration meets all of the requirements we had defined for the email engine. The next step will be to implement the email subscription spec that Rob and I wrote back in December, though that work may be deferred to version 2.

I have not yet done extensive performance tests comparing use of a local mail server (e.g., Exim) vs. an external service (e.g., Gmail), but that’s something I’ll be exploring in the not-too-distant future. Even if the email sending process takes a long time, the users of the website and admin should not notice, and system performance should be well within acceptable bounds. User registration, password reset and other emails will have higher priorities and jump to the top of the queue, so even while processing a large subscription queue those critical messages will still be (almost) immediately sent out.

I’m holding off on checking this work into the public Courant News repository as I currently have a mess of Nando stuff interleaved, and I need to sort that out first. I also want to add some forms and views for handling creation of email subscriptions on the public website, so maybe once that is complete I’ll push through a checkin.

Over the past 6 months I’ve been installing new builds of Windows 7 every month or two on one of my laptops.

General Programs

The set of programs I install after each wipe:

• Microsoft Office (productivity suite)
• Firefox & Opera (web browsers)
• Trillian (multi-protocol IM client)
• Twhirl (Twitter client)
• Pandora desktop client (internet radio)
• iTunes (music for personal music device)
• Adobe Reader or Foxit (PDF reader)
• 7-zip (zip/tar archive creator/extractor)
• Sling Media Player (watch home TV from across internet)

Development Tools

I also have a bunch of development tools:

• WinSCP (FTP/SCP client)
• Putty (SSH client)
• XAMPP (LAMP stack)
• Python (programming language)
• VMWare Workstation (virtual machine environment)
• Komodo IDE (programming environment)
• TortoiseSVN and GIT (source control integration)
• Adobe Creative Suite (Photoshop, etc.)

There’s probably some that I’m missing, but those are all the ones that I use most often.

Conclusion

I think it’s safe to say that it’s near impossible to replace my development setup within my browser, but honestly “most” of the population doesn’t do such work.

On the general software side, I could probably get by with just a browser. Twitter, IM, Pandora, Office Web/Google Apps, etc. could replace many of the programs that I listed. I’d still have some issues with zip files and watching TV through my SlingBox, but my general needs could be satisfied. Yet I currently decide to use the desktop versions of these apps, because I can run them nicely in the background, with OS notifications, and automatic startup at system boot.

I’m curious to see what other people use and how readily they could forgo everything but a browser.