Multimedia

Being both an early riser and three hours ahead by virtue of time zones, I started my day working on some YDN tasks and left Courant for the afternoon. First order of business was finishing the new multimedia landing page. We’ve been without that page since launching on Courant at the beginning of the semester, and the version before was barely functionality as it was. Zeke Miller, the new Online Editor, was the driving force for the design while brainstorming and gathering feedback from the rest of the Online desk. I’m very pleased with the results, but have opened a thread over at the CoPress forums to get feedback from the wider college news community.

Search

Next on the docket was finishing the transition from Sphinx to Haystack for the Courant News search system. Sphinx was very nice, but Haystack opens new doors and fits better with the Courant News project philosophy; it allows for the use of multiple search engine backends, so that sites can use whatever is most appropriate for their situation.

For example, if a site cannot install a powerful Java system like Solr, then they can use the pure-Python Whoosh backend (which is also nice for development use). In practice, I ran into problems while trying to use the Xapian (fast C++ engine) backend and had to switch back to the resource-heavy Solr. Hopefully as Haystack and its Xapian backend mature the situation will improve.

By the end of the afternoon, I had finished implementing it in Courant and had completed the templates for the YDN search system. Please feel free to explore. I’m especially proud of the filtering options, which allowing for filtering of the query results by section, staff members, content type (article, photo, video, etc.), and date range. It makes it very easy to find specific content that you are looking for, or even to just get a broader sense of how popular a given topic is.

Solr’s relevancy engine is giving disproportionate weight to photos because they aren’t as long as articles, which is not very desirable. However, I’m still a Solr newbie and am trying to explore options for resolving that issue. Once I figure out how to do it with Solr, I’ll have to push the Haystack folks to expose any requisite functionality.

Headers App

Rob’s big project, and his first major code contribution in several months, was the implementation of an app we’ve been designing for the past few days. He’s going to write a more in-depth post, so I won’t step on his toes too much and go too deeply into it right now; essentially, it allows you to vary the display of your news site based on HTTP header values, most importantly User-Agent and Referer [sic].

The biggest example is to show a different set of templates to visitors with mobile User-Agent strings by showing them .mobile templates (using the FileExtension middleware and render() function in Courant News). You can also define a cascade of extensions, so that you only have to customize certain parts of your site if you want, and show the standard .html version elsewhere; this could be interesting if you wanted to take advantage of new HTML5 functionality in browsers (User-Agents) that support it, while not having to duplicate templates on other pages (or if you want to customize only certain pages on your mobile site for iPhone users).

It also lets you simply set extra template context variables, so that you can show special content to people coming from certain referrers (e.g., Facebook or Twitter). There are countless uses for such functionality, I’ll let you come up with your own.

You can browse the headers app in our code repository; it only requires a bit more work before we will declare it complete, hopefully that will happen in the next several days. When Rob writes his post, I’ll update with a link.

Conclusion

Today was a productive day, with 9 commits to the Courant News repository. Hopefully the pace will keep up, and once these last few features are finalized the path will be clear to begin work on Nando, which will be the real cornerstone of the Courant News platform.

For your techies, some background: We have been using Subversion (svn) for source control, but there’s been a push to move towards git. The biggest holdup had been our dependence on svn:externals to pull in some other projects without having to fork them, which is a pretty poor reason. As a result, I’ve been seeking ways to get rid of that last vestige of svn so that we could allow people to use git if they prefer that. We intend to continue using svn for Courant’s core codebase, but we appreciate the fact that many devs prefer to use git for their workflow, so we don’t want to preclude that.

As a result, I’ve been seeking a solution to simultaneously get rid of svn:externals while also easing the process of installing and using Courant. Some posts by the famous djangonaut James Bennett, along with looking at projects like Pinax, convinced me that such a solution existed in pip with virtualenv. Virtualenv allows you to create an isolated environment on your computer in which you can install python packages without it polluting the rest of your system, while pip allows you to install python packages from the Chesseshop or from a repository (supports svn, git, bzr, and hg currently).

Now for everyone: To use Courant, you first have to install python (2.5 recommended), svn, git, and whatever database engine you’d like to use (we strongly recommend PostgreSQL, but MySQL works too). From there, you just download one file from the Courant servers, courant-boot.py. Running courant-boot.py with some options will install Courant and all of its python dependencies, including Django and many others, inside a fresh virtualenv. It can also set up a clean copy of our sample project, or it can checkout your own site project from an external repository.

python courant-boot.py myvirtualenv -n website -p default

That one command will create a new virtualenv called “myvirtualenv,” install everything, and make a copy of the “default” sample project in a folder called “website.” For the YDN, I can replace “default” with the URL to our private SVN repository, and it will check out the Courant-powered version of our site and be all ready to go. I don’t think it could be much simpler. It should work perfectly well across all the major operating systems, and I have personally tested it on Ubuntu and Windows. I’ll get Paul to verify it works on OSX before release.

More detailed installation instructions, including a number of other options and instructions for how to set things up for developing on Courant itself, will be included in the project documentation. This was one of our last hurdles before public release, so once we finalize some remaining details and get team consensus, we will be opening up the project site. Stay tuned.

One common way, adopted by the Django built-in admin app, as well as  a number of common Django reusable apps like django-tagging and django-mptt, is that of a registry system. I’ve been joking with one of my Courant cohorts, Robert Baskin (@rsbaskin), on twitter about registries, and I thought it was time to let everyone else in on the discussion.

An Example

One way to manage functionality across various content types is to define the set of content types in your settings file. A recent example is django-shorturls, which lets you generate short URLs for your content by specifiying the content types to expose with a given prefix/abbreviation.

# in settings.py
SHORTEN_MODELS = {
    'A': 'myapp.animal',
    'V': 'myapp.vegetable',
    'M': 'myapp.mineral'
}

This works wonderfully when you have full control of the codebase, but if we were to include django-shorturls in Courant itself, it would require news orgs to modify our code* to add in their own custom content types or tweak how our standard configuration works.

You can solve this problem by creating a registry system. Each model can then register itself with the URL shortening app, passing along the prefix parameter of its choice. For example:

# in myapp/models.py
from django.db import models
from courant.core.shorturls import shorturls
 
class Planet(models.Model):
    name = models.CharField(max_length=100)
    ...
shorturls.register(Planet, 'P')

In the courant.core.shorturls app, it keeps track of all the models that register with it, and then can use this registry in its internal code. It also means that if you want to change the prefix for, say, the built-in Article model from ‘A’ to ‘S’ (for story), you could do this:

# in myapp/models.py or any other place that will get automatically run by Django, such as an __init__.py
from courant.core.news.models import Article
from courant.core.shorturls import shorturls
 
shorturls.unregister(Article)
shorturls.register(Article, 'S')

In this manner you can tweak the default configuration, while also hooking your own new content types into our functionality.

The Proliferation Problem

The short URL app example above is a nice, clear example of the types of situations where registries make sense. But as I mentioned in the beginning of this post, there are a number of other apps that use registries, both within Courant and in the general Django community. There’s the Django admin, django-tagging, django-mptt (hierarchical relationships), and django-comment-utils on the outside. We also have the get tag which I described previously and a registry for the search system (upcoming post once a few bugs are fixed). Our Article model currently looks like this (much simplified):

# courant/core/news/models.py
from django.db import models
from courant.core.discussions.moderation import moderator, CourantModerator
from courant.core.gettag import gettag
from courant.core.search import search
 
class Article(models.Model):
    heading = ...
    ...
moderator.register(Article, CourantModerator)
gettag.register(Article, name_field='heading')
search.register(Article,
                fields=('heading', 'subheading', 'summary', 'body'),
                filter_fields=('section', 'display_type','status'),
                date_field='published_at',
                use_delta=True)

We haven’t actually written the short URLs app I described above, but likely will, in which case you can tack on yet another registration call there. As you can see, this starts to rapidly build up for the more commonly used models, although that’s admittedly a rather small percentage of all of our models.

For those who follow Rob or I on twitter, you may have noticed us joking about meta-registries, which would be a registry to help manage all of these individual registries. How that would actually work is up for debate, and is really nothing more than an inside joke (though not so inside anymore).

Alternatives?

In light of this potential problem of runaway registry creation, we’ve been considering some other options. For some of the more complicated registries, like for search, we’d most likely be better served by going to a declarative syntax like models themselves. django-haystack has taken this approach, and I actually much prefer it in many respects, and just haven’t yet gotten around to building something similar to work with django-sphinx (our search tool of choice; explanation of that decision for that future post on search). It still requires a small registry of its own, but the registration call is handled in a separate file from the model itself, similar to how the admin system works.

Another option could be to tack additional options onto the models’ internal Meta classes and customize the python metaprogramming that Django does to build python objects from your model definitions. A registration process would still be occuring behind the scenes, but you wouldn’t be required to interact with it directly. For the above Article example, it might look like this (again, the surrounding parts of the model are much simplified):

# courant/core/news/models.py
from django.db import models
from courant.core.discussions.moderation import CourantModerator
 
class Article(models.Model):
    heading = ...
    ...
 
    class Meta:
        # standard Django meta options
        ordering = '-published_at'
 
        # custom Courant meta options
        short_url_prefix = 'A'
        get_tag = {'name_field': 'heading'}
        moderator = CourantModerator
    ...

This is maybe slightly cleaner because you don’t need all the imports at the top of the file, but I’m not sure it gains you much in the end. I personally rather like the explicitness of importing everything and manually registering it. Note that the get_tag meta option in this example uses a python dictionary because there are actually a number of other optional parameters that you can pass to it, and you wouldn’t want to have a meta option for every single possible parameter. My biggest hesitation with going this route is that it probably involves mucking with metaprogramming and doing some behind-the-scenes magic, which I don’t really think is a favorable cost/benefit tradeoff.

Conclusion

I hope that explains some of my twitter ramblings over the past couple weeks, and gives some additional insight into how Courant will enable customization and extension of the platform without forking the code. Please post comments if you have any questions or thoughts or opinions.

* Technically, this example resides in the settings file, which the news org has full control over anyways and thus would have no problem modifying. But I think it makes more sense to define the prefixes with the models they are related to, and not hidden away in a setting.

(Note: This post is written from my perspective as the lead developer at a news organization, and only secondarily from my position as one of the leads of a news CMS project.)

Perhaps the fundamental problem is that open source software is generally built to scratch one’s own itch. The beauty of the OSS model is that it allows other people with the same itch to join your effort and not reinvent the wheel.  The potential downside is that your itch is likely not exactly the same as my itch, and if I build the software just how I need it, it will likely not be as ideal for you as it is for me. There’s nothing wrong with me building something to scratch my own itch, that’s my prerogative as a software developer. But if I want you and other people to use or contribute to my software project, I’m going to have to take your needs into consideration.

The Approaches

How is this achieved? One way is the lowest common denominator (LCD) approach, where we figure out what requirements we both share, and then just build those. Maybe I need some extra things, and you need some different set of extra things, but we build to the LCD and then we can privately add the features that only we need. This is often achieved through a plug-in or extension system, where the base software has a number of hooks and entry points for external code to plug into. WordPress works on a model like this, and it has developed an enormous plug-in ecosystem where people share their extensions for the benefit of the community.

Another way is to build a system around a set of generic tools, upon which you can build whatever functionality you need. Systems like Drupal with its CCK fall into this category, where you have a few basic generic content types upon which all the other modules can be built. This can be an extraordinarily powerful system, allowing users to build almost whatever they want from these basic building blocks. Sure, it might take some more effort than if everything was pre-assembled like in WordPress, but once it’s all set up it, it is exactly what you want it to be.

Yet another approach is to build “opinionated” software. The leaders of the project devise some philosophy or vision around which the product is built, and if that vision doesn’t align with a particular user’s, then that user can live with it or go find another piece of software that does. Such software is often of excellent quality and does exactly what the leaders envision, but at the expense of building something that is applicable to a much wider audience.

The Problem

Each of these approaches have their own pros and cons, which have been discussed to death elsewhere and which I don’t wish to rehash here.  Instead, I want to look at how I feel creators of various news CMSes are failing their audiences, or at least not living up to their potential.

I have no desire to publicly bash any one project in particular, because I do not think that that helps anyone, and there are honestly too many projects for me to cover in a single blog post. But there are a number of projects with sizable teams behind them, and with either funding or professional backing, and it saddens me to see them follow the trajectories they are on.

I understand that news organizations come in all shapes and sizes. There are daily and weekly papers, magazines, online-only, blogs, etc, etc. Each of these has slightly different needs and workflows. Perhaps each of them needs their own toolsets, or perhaps they simply need to adapt.

My problem is that I think it is myopic and borderline arrogant for teams to build a piece of software for their particular organization and then market it to the rest of the world as a generic product. Some projects even delude themselves into believing they are building a generic product, only to continue building it to their own set of needs. There’s nothing wrong with building a CMS for your news organization; tons of college news organizations have done so, and are more than happy with the results.

But if you aren’t building in considerations for how other organizations operate and might need to customize your system, then you are on the wrong road. When I see HTML snippets for rendering bits of content buried in code, it makes me want to weep. Because I know that if I tried to use that system for my news organization, I would be unable to control that aspect of the output without forking the project and using a customized version of the codebase. When I see a hard-coded workflow system, I wonder whether that can really fit the workflow of my news organization; whether I will need to rip it apart to bend it to my will, or else force my editorial staff to change their ways. When I see the set of fields on the article/story content type, and realize that they are missing something crucial to my news organization’s needs, I wonder how much work it will take to add what I need and whether that will break the rest of the system.

The Solution

It is all too easy to start and design a project with noble intentions and then get derailed as you actually build the product. Ideally, the team already consists of individuals from varied backgrounds, who can explain how organizations they have worked with/for would want to use the software. This is a great start, and can lead to a healthy design process.

When you actually start writing code, you need to be building multiple projects that use it simultaneously. Only by actually using the code to create something can you discover the shortcomings and problems. And by building multiple, different projects on top of your platform, you begin to see how different organizations have different needs and how you can adapt your platform to meet these varied requirements. The same person shouldn’t be doing multiple projects, however, you need multiple people with varied backgrounds making different projects across the full spectrum of your target market. Otherwise, you can too easily fall into the trap of creating opinionated software that meets the needs of that single sample project.

If you really want a generic platform with a solid community, there has to be consideration given to extending your product and using it in conjunction with other software. Many news CMSes are being built on Django, which is appropriate given its genesis, and which also has a strong movement for “reusable apps.” There’s quite a large collection of 3rd party reusable Django apps out on Google Code, Github, and other sites, and any CMS should be built so as to allow for use of 3rd party apps. They can act as orthogonal “plug-ins” of sorts, but can also act as a way for users to customize a CMS by overriding or enhancing certain bits of built-in functionality. If your article handling isn’t quite what I need, allow me to write my own and have your code use my extension. That’s frictionless software, and that lets me do my job. Sensible, comprehensive ways to build on your platform are a must. Ultimately your CMS should be acting as a platform or framework for me to build my news site on, so give me the power to build it to my requirements.

Conclusion

There’s certainly a place for opinionated software in the news CMS domain, and some organizations may want or need to create their own solutions. But if your aim is to build a platform for other organizations to adopt, and you haven’t thought about how they can bend it to their will without forking the project, then you’ve failed at your task in a pretty fundamental way. If I can’t get what I need from your software with minimal friction, I’ll move on and find something else. News organizations need to focus on the news and the journalism, and perhaps on experimenting with new technology. Your software shouldn’t be getting in their way at all, and should actually be making it even easier to do their job as they want to do it.

All I’m asking is for you to have an open mind. Find someone at another organization and have them look at your project and see how well it would fit their needs. See all the little things that don’t quite align, and find ways to remove such friction. Because all the little details matter as much as (if not more than) the high-level conceptual ones. CMSes should be enablers, please get them out of my way and let me do what my news organizations wants it to do.