Better and Consistent Headers

For some reason, almost every screen in the app uses a header that consists of a relatively large WordPress logo and then the typical page header text. That makes some sense for the blog selection screen (Fig. 1), but not for inner screens (Figs. 3 & 4) when the user is well-aware that they are using the WordPress app.

On the panorama screen, the blog’s name takes prime billing (Fig. 2), which makes total sense and should be replicated (with a smaller font size/height) for all other “interior” screens in the app. If the branding is still desired, we could maybe keep a small circular WP logo in the top-left or top-right corner of the screen.

For non-panorama pages, I’d also like to enable the system tray (clock, battery, wireless signal, etc.). This eats up some vertical screen space, but I think most users prefer to see it when using apps. It would also allow us to use the system progress indicator (see below).

Figure 1

Figure 2

Figure 3

Figure 4

Use Native Controls

When the WP7 app was originally made, the Windows Phone 7 SDK was pretty new and the ecosystem was relatively weak. As such, the original developer(s) was forced to implement custom controls for certain tasks, and they reverted to emulating the controls implemented/used by the WordPress for Android app. With the Silverlight Toolkit, that is no longer necessary, and it would be a great improvement to use native-like controls instead.

Multi-select lists

For comment moderation and category selection (Fig. 5), the app employs a multiple selection list control. The current design toggles a blue background for “selected” list items, where native WP7 apps like email use a checkbox paradigm. Figure 6 shows the multi-select list control from the Silverlight Toolkit.

Figure 5 - Category Selection

Figure 6 - Toolkit Multiselect Control

 Progress indicators

Another Android-ism used throughout the app is progress spinners (Fig. 7). For the 1.2 release, we added a WP7-style progress bar to the panorama screen, but most other screens use a modal spinner animation. With the WP7.5 SDK, Microsoft made it super-easy to use native progress indicators by building it into the system tray.

One advantage of the spinner was that we could load the underlying page controls and they would be impossible to interact with until the spinner disappeared. If we switch to using the system-tray ProgressIndicator, then we will likely want to hide or disable controls while data is loading or an action is being performed (or whatever else is causing a progress indicator).

Figure 7 - Spinner

Figure 8 - ProgressIndicator

Context menus

On the blog selection screen, you can currently delete a blog by choosing “delete blog” from the appbar menu (hidden under the ‘…’) and then choose the desired blog from a modal pop-up list. It would make a ton more sense to just support the tap-and-hold context menu paradigm as supplied by the Silverlight Toolkit. I would have the context menus contain “pin” and “delete” actions.

A similar pop-up list exists for pages and posts in the panorama, though it’s less clear that a context menu makes sense in that context. But that’s a non-issue, because…

Panorama Cleanup

Perhaps the biggest concession made to the WP7 app when porting it from Android was the use of the iconic panorama control. It opens to a grid of tiles for the most common actions (see Fig. 2 above), and has additional columns for comments, posts, pages, and statistics. While there is a separate “moderate comments” screen (see Fig. 4 above), the panorama is the only way to access the functionality for posts, pages, and stats.

The WP7/Metro design guidelines call for panoramas to be like “magazine covers”, with a very small amount of content that entices users to dig deeper into the content on additional screens. But the WordPress app’s panorama has heavy lists for comments (see scrollbar in Fig. 9), pages, and posts; this is not only in violation of the design guidelines, but also makes the panorama slow to load and heavy on memory usage.

Figure 9 - Panorama Long List

Recently, I’ve been attempting to discern usage patterns for users of the various WordPress mobile apps, in addition to mobile apps for other blogging systems/services (Tumblr, Blogger, SquareSpace, etc.). As far as I can tell, the primary scenarios for mobile blog wrangling are new post creation, comment moderation, and light tweaking of recent posts.

Curiously, all of the WordPress mobile apps give equal weight to the four top-level items (comments, posts, pages, stats), perhaps because that’s what the WordPress XML-RPC API supports. But I’m in favor of optimizing the UI for the most important use-cases, and relegating some of that functionality to deeper parts of the app.

My proposal is to keep the first actions screen, but with two groups of tiles:

1. “Actions”: Add a post, Moderate comments, View stats, and Blog Settings
2. “Manage”: Posts, Pages, Comments, Users (once supported by XML-RPC API), <one tile for each custom post type (once they are supported by XML-RPC API)>

I would have three columns in addition to the actions screen: unmoderated comments, drafts, and recent posts. These correspond with the scenarios I identified above, and would include a maximum of 3-5 items each, with “more…” links leading to the corresponding archive management pivot screens.

Content-type management pages

This light-weight panorama would require new screens for management of the archives for each content type (page, post, comments, etc.). I envision these using the pivot control, allowing pivoting by the ‘status’ field (all/draft/published for posts, unmoderated/approved/rejected for comments, etc.).

The appbar on these pages would contain “add”, “search”, and “refresh” buttons. Tapping a list item would open an edit/moderate page for that item, and tap-and-hold would open a context menu for actions like “delete”, “preview”, “view comments”, etc. (replacing the pop-up modal menu used in the panorama now for posts and pages).

Using the new Mango SQL CE database support and paging for XML-RPC in WordPress 3.4 (hopefully), we can load the entire archive of a blog (probably only the most important fields like title, author, status, format, date, tags, and categories; notably, not the full body contents) and support infinite-scroll on these pivot pages. This would also back the “search” operation, and we could add a feature to the post editing page akin to the internal linking feature in WP core admin.

About & Feedback Page

The app currently has a page that shows when unexpected exceptions occur, and has links to some sources of help. This is a decent start, but the app really needs a proper ‘about’ page, with links to it in the appbar of the blog selection page and the panorama page.

The page would contain the current app version information and links to common sources of help and feedback, such as the official forums, official FAQ, and marketplace review form. We’ve been getting a number of bad app reviews in the past months, but not much feedback on the forums so that we can actually identify and address users’ issues.

Live Tiles

In the last release, we added support for per-blog live tiles on the start screen. But these tiles are still static images, with nothing “live” about them. The tiles should probably update with the number of new comments, number of views today (if stats are enabled/allowed), and maybe the text of the most recent comment on the back. This will require some design work from Isaac to make it look pretty. I like the way Evernote has tackled this in their Mango release, maybe we can try something similar.

Animations

The WordPress app is painfully static, especially when compared with the first-party apps that are full of subtle animations that make the apps feel alive. The app would feel much more polished and enjoyable if we spent some time carefully designing proper animations and transitions. The Silverlight Toolkit makes transitions super easy, and animations aren’t particularly difficult either. This will require some serious design collaboration with Isaac, since it’s critical to get animations just right or else they will just annoy users and detract from the experience.

Conclusion

The end result of all of these proposed changes would be a noticeable change to the app’s feel and flow, which is why I’ve tagged them for version ’2.0′.

One of the Windows Phone marketing slogans was “designed to get you in and out and back to life”. I know the WP7 team had a strong scenario-focused design effort, and it really shows when you use the OS. I’d really like to see the WordPress for Windows Phone app streamline and orient itself such that it’s fast and easy to tackle the most common tasks.

The most complex page of the app is the post/page editor, which also requires some serious rethinking. This post was too long, so I’ve saved my thoughts on the editor another post soon.

All apps compiled for Mango support the new “fast switching” capability, which suspends the app instead of immediately tombstoning it when the user only leaves the app temporarily (e.g., for a phone call or to check email).

To take advantage of fast switching, we’re trying to push out a quick update (1.3) for Mango users of the WordPress for Windows Phone app. We also added two new features: per-blog live tiles and picture hub integration.

In the current release, when you launch the app you choose which blog you want to manage. For the 1.3 release, there will be a new “pin” button in the appbar on the blog panorama that will let you pin that blog directly to your start screen. This lets you easily jump into your favorite blog(s) right away. In a future update we hope to add new comment counts and other information to the live tiles to make them even more useful.

One of the common use-cases for on-the-go blogging is uploading photos taken with your smartphone. The 1.3 release will now include integration with the Photo hub’s “Share” functionality. When you take a photo, click the “…” menu and choose “share.” Your email and social networking accounts will show in the list, and now WordPress will too. When you choose to share with WordPress, the app will launch, you can select the appropriate blog, and you’ll be taken right into the “New Post” page with the photo already attached and waiting for you to write your post.

I hope the app will be submitted for certification soon, and we’ll have a post on the official blog when it becomes available in the marketplace.

Background

The WordPress mobile apps are primarily developed and maintained by the Mobile team at Automattic, although all of the apps are themselves open source. At present, there are three developers and one designer on the Mobile team, and they try to spread their efforts among the six mobile apps.

Each app has its own P2 development blog and Trac instance, and anyone from the general population can contribute patches and file tickets. In practice, however, it appears that the vast majority of the work is done by the Automattic employees.

I wish more people would contribute to these projects, as they are a great opportunity to get involved in open source on a smaller scale, since these apps are much less complex than WordPress itself.

WP7 App Beginnings

Windows Phone 7, which was a reset of Microsoft’s mobile strategy, launched worldwide in October/November of 2010. In December 2010, the WordPress for Windows Phone app was first released. Version 1.1, which fixed a set of important bugs, was released shortly thereafter.

The app was a “partnership between iSoftStone, Microsoft and [Automattic]“. As far as I can tell (as an outsider), the app was primarily developed by iSoftStone, a Chinese outsourcing company. The app was a port of the WordPress app for Android, with a light UI skin to match the WP7 “Metro” design style, including the popular panorama control.

Once the app was “complete” and iSoftStone had delivered the code, their involvement was essentially done. From there, the Automattic team filed tickets for several bugs, but never really moved forward on v1.2.

Version 1.2

Around mid-May 2011, on the eve of purchasing a WP7 device for my personal use, I decided to see how I could help out. As a developer with a fair amount of C# experience who was going to be using a WP7 device as my primary phone, I thought I was well-positioned to lend some support to the app development.

It appeared that the app had been abandoned, though I was told that it was just on hiatus until some process issues were worked out. So I started playing with the app and discovered a number of annoying bugs, some of which didn’t have tickets yet.

The app had not been updated since the v1.1 release in December, and since then a WP7 OS update (NoDo) had come out with copy/paste support and some performance improvements, neither of which the app properly supported. So I worked on some patches to solve those shortcomings and put them up on the trac.

Soon after, the Automattic team started contributing too and the commits started flowing. The iSoftStone dev fixed his final outstanding bugs, and I contributed patches where I could to help the Automattic team take care of the rest.

Version 1.2 was finally released at the beginning of August, see the release post for details on the features added. At this time, I was given commit access, and thus officially joined the dev team.

Prepping for Mango

With the v1.2 release, we had fixed almost all of the tickets in Trac and most of the glaring flaws in the app. However, there is still a ton of room for improvement. August was a mini-vacation for me, so I had hoped to get some good work done on the app in anticipation of the Mango launch this fall.

On August 22, the WP7 app store opened the door for apps targeting the Mango release (WP7.5), which offers a large number of new features and APIs. This includes support for multitasking, multiple live tiles per app, background agents, built-in SQL CE databases, and more. Consumers will be able to get Mango phones within the next month or two, and I wanted to have an update ready for them.

Most of my work on the app to-date had been relatively superficial changes, and I had not yet dug deep into the internals. However, to implement a database backend or improve live-tile support, I would have to make deep changes. And, boy, was I in for a shock.

Architecture?!

I’ve worked on a healthy variety of software projects, from pet projects to large enterprise applications at companies like Microsoft. I’ve become fond of concepts like Separation of Concerns and Do Not Repeat Yourself, and basic architecture patterns like MVC (or its Silverlight/WPF variant, MVVM).

The WordPress for Windows Phone app codebase is a total mess. XAML code-behinds gleefully call XML-RPC methods or into the global singleton containing all the app’s data structures. The XML-RPC methods themselves are a convoluted mess of string interpolation and concatenation, using string templates from multiple sources (some in RESX files, some inline with logic). Because the data is in a singleton and code-behinds, designers can’t work on the app in Blend. In short, highly coupled code with almost no separation of concerns.

Perhaps the original developers were under intense time pressure, and had to sacrifice maintainable architecture to hit a tight deadline. Been there, done that. However, given that this was primarily the work of an outsourcing firm, my suspicions are that the truth lies elsewhere…

How To Go Forward

So, given the state of the code, what’s the right way to proceed? I see three options:

Option #1 is to start piecemeal refactoring the app towards a “proper” MVVM architecture. Use MVVMLight or Caliburn, and start moving code from XAML code-behinds to ViewModels. In the process, decouple the data layer too, so that it can eventually be replaced with a SQL CE database backend. I imagine this would take a few months at a reasonable pace; but a few months means that Mango users will have to wait for fast-switching support and all the other niceties that Mango apps get over NoDo apps.

Option #2 would be to do a minor update for Mango, perhaps just multiple tiles (per-blog) and a background agent to keep the data fresh. Thus we could do a v1.3 Mango release to tide people over until we can get around to the work in option #1.

Option #3 would be to just soldier on and learn to live with code duplication and tight coupling. This makes it harder to make changes, but maybe releases could happen sooner. If we stumble along long enough, maybe we won’t have any users or Windows Phone will radically change and require a rewrite anyway.

The code snob in me would prefer #1, and I may get started on that on my own. The pragmatist in me would prefer #2. The developer in me would want to quit a project that would choose #3.

I suspect that next week at BUILD we will learn that WinPhone apps will have an easy path towards compatibility with Windows 8; a properly architectured MVVM app would make it super-easy to bring the WP7 app over to W8, or even to existing Windows versions by writing a WPF UI layer over the same ViewModels and data backends. Just another reason to go for option #1 or #2.

Conclusion

Sadly, WP7′s marketshare just isn’t very high, and Automattic has a very small team of mobile developers. I understand and respect their decision to work primarily on the iOS and Android mobile apps. Isaac, Dan, and Dan have been extremely gracious and helpful to me over the past several months, and my complaints/criticisms are not aimed at them.

My academic studies prevent me from pouring significant time into the project, so the app has been sitting since the v1.2 release. If there are other Microsoft stack developers out there who are interested, please post on the dev blog and get involved! With some more manpower, I think we can do great things with this app.

In the short-term, I’ll likely start work on an MVVM refactor and post that work on GitHub. If it doesn’t work out, then no harm done. If it does, and the rest of the team approves, we can merge that back into the primary SVN repository.

Beyond the app’s internal workings, I also have many thoughts about its user interface design. With custom post type support (hopefully) coming in XML-RPC in WordPress 3.4, I have some ideas for how to prepare the app to support more than just posts, pages, and comments. I also have ideas for how to make it better match the WP7 styling and behaviors, support rich text editing, and more. Look for my next post on that topic next week.

Microsoft

In March, I was still in Redmond/Bellevue, Washington, working through my third internship at Microsoft with the Dynamics CRM team. I got to continue working on a set of functionality that I started on during my previous Summer 2010 internship, and which will hopefully ship in the next release. I also got to see the wheels turning on some longer term work, and I’m super excited about where the CRM product is heading.

So excited, in fact, that I’ve accepted Microsoft’s offer to join the team full-time after completion of my degree this school year. As it stands, it looks like I’ll be moving to Seattle in September 2012.

Back to School

After my adventures with Camayak and Microsoft, it was time to return to my studies and finish my degree. I spent the summer here at Yale in New Haven, CT, taking several courses to get back into academic mode.

Today was the first day of classes of the fall semester, and it’s exciting to be back among so many passionate students. I miss many of my friends from the Class of 2011, but will soon get to better know my new Class of 2012 peers and more underclassmen.

World of WordPress

With my previous Yale extracurricular involvements terminated, I finally spent some time digging into WordPress this summer in my “spare” time. Over the past couple of years, I’ve gotten to know a number of people involved in the WordPress community, and they’ve slowly dragged me into it as well. I’ve now been to four WordCamp events, and met even more folks that way.

WordPress is an interesting beast of a project, and I must confess having mixed feelings about it overall. Several years ago, I did the Courant News project because WordPress was not yet ready (IMHO) to power “proper” news websites. But since version 3.0, I think WordPress is better suited for the task, as evidenced by countless news sites now running on WordPress. I intend to write more about that in the future.

But my involvement with the news folks has been quickly winding down, so I’ve gotten involved in WordPress to scratch two of my own itches.

XML-RPC API

While working on Camayak last fall, I wrote a python library to integrate WordPress with other systems by leveraging WordPress’ XML-RPC API. We ended up going a different route for Camayak, and the library sat in my GitHub account for several months.

This spring, I discovered that several folks were using the library for real-world problems, such as the Chicago Tribune news apps team. This led me to finally finish several outstanding workitems and fix some discovered bugs. The library is now at version 1.5, and I intend to keep actively maintaining it.

This summer saw a Google Summer of Code project dedicated to expanding WordPress’ XML-RPC API, finally gaining feature parity with internal APIs and features such as custom post types, taxonomies, and post formats. The project just finished this past week, and will hopefully be merged into core by WordPress 3.4 (feature freeze for 3.3 is technically today, and the patches aren’t really ready).

I’ll be writing several posts on XML-RPC soon, as I have a ton of thoughts on the topic and where I see WordPress evolving on that front.

WordPress for Windows Phone

In May, Verizon finally launched their first Windows Phone 7 phone, the HTC Trophy. As a Microsoft intern, I had a voucher for a free WP7 phone, and was eager to take it for a spin. However, I soon discovered that the WordPress app for WP7 was pretty weak.

Given WP7′s low marketshare, Automattic’s mobile team understandably wasn’t spending much effort on the WP7 app. As all WordPress mobile apps are open-source, I decided to get involved and start submitting patches. We managed to trim the existing bug backlog and in July put out version 1.2, the first since December 2010.

I’ve since been given commit access, so that I can take a more active role in developing the app. I have a ton of ideas for where to take the app, just not enough time to execute on them for now. I’ll have a post in the coming weeks on some UI ideas for rich text editing and revamping the functionality to take advantage of the new XML-RPC methods mentioned above.

Conclusion

My goal is to post at least once per week, and eventually start microblogging and reducing my dependence on external services like Twitter and Delicious/Diigo.

As my primary work these days is as a full-time student, I’ll eventually be blogging about my studies. In the meantime, I have a backlog of post topics expanding on items mentioned in this post. Thanks to Andrew Nacin and the CoPress gang for inspiring me to get back into using my blog again.

Moore’s post contains a link to a Google document containing a list of the 577 sites. I downloaded this spreadsheet as a CSV for easy consumption and analysis, and set to work.

Duplicates/Sub-sites

First thing I noticed is that there are 25 entries whose domain is already mentioned previously in the list. For example, ‘http://www.picayune-times.com/’ is mentioned twice, in rows 34 and 43. There are also sub-sites listed, such as ‘http://www.tcpalm.com/’ (row 119) and ‘http://www.tcpalm.com/news/sebastian-sun/’ (row 121). If we remove all of these redundancies, we’re down to 552 sites.

CMSs

Then I started randomly going to sites in the list, and noticed that many of them look remarkably similar. It soon became apparent that there were a number of large chains sharing the same CMS and almost identical templates.

I came up with simple heuristics for determining which CMS powered a site (e.g., by looking for references to the CMS company’s domain in a CSS stylesheet or Javascript file link). I wrote a small Python program to go through all the URLs, check the HTML source of each page against my heuristics, and identify the CMS.

Results

The results (see appendix for full document):

• 170 sites using TownNews products (e.g., BLOX)
• 101 sites using MatchBin (many MediaNewsGroup papers)
• 213 sites using Zope.com/Zope4Media (many from the Community Newspaper Holdings chain)
• 21 sites using Ellington (Scripps chain)
• 2 sites using MediaNewsGroup (rest of MNG’s papers still using MatchBin, see above)
• 24 sites from the Rush Publishing chain (I couldn’t find a corporate site, but you can see a list of papers here)
• 17 sites from the Swift Communications chain
• 4 sites from the Freedom Communications chain
• 5 sites built using Microsoft FrontPage (seriously? in 2010?)
• 20 other sites with either custom or obscure CMSes

I’m not claiming that my script’s results are perfect, but they are at least indicative. I draw two conclusions here:

1. The majority of the 577 claimed sites are from a small set of CMSs. Notably absent are open-source CMSs. If you only count once for each CMS that has implemented hNews, you have the 8 listed above, plus perhaps a few more from the ‘other’ list.
2. The list does not contain the full client list for each of the chains represented, and thus there are far more sites that support hNews which just aren’t listed there.

Compliance

I had intended to algorithmically verify that all of these sites were actually hNews-compliant.

The microtron library has simple hNews extraction/validation which I was able to leverage, but I needed a way to find an individual article page on each site.

My first instinct was to use the RSS feed of each site to extract a recent article link. However, I discovered (and tweeted in despair) that almost none of the sites have a <link> tag in their homepage <head> to support RSS auto-discovery. A few had RSS feeds on their “News” section pages, but it become clear that automagically trying to find RSS feeds was doomed to fail. You would think that a site that implements a microformat standard would also implement the ubiquitous standard of RSS or Atom…

Instead, I manually verified compliance with examples of each of the CMSs discovered, and was glad to see that they all passed. I apologize for the scare quotes in one of my tweets, I did not mean to imply that Mr. Moore or anyone else was being disingenuous.

Final Thoughts

I am quite happy to see that hNews is being adopted more broadly, and I hope to see that trend continue. As a structured data fan, it makes me happy that so many articles out there are easily parse-able.

However, I think it’s a bit unfair for the hNews proponents to use individual sites as the primary metric for “reach” of the microformat. Instead, I would prefer that they look at the CMSs supported, and continue evangelism in an effort to convince the other newspaper chains out there to adopt hNews in their own systems.

Appendix

My annotated Google Spreadsheet with CMS breakdown and domain uniqueness.

My python script (‘hnews.csv’ is Moore’s original spreadsheet exported as a CSV, and ‘hnews_rev.csv’ was the basis for my own spreadsheet above).

Historically, the Content in CMS referred to an organization’s warehouse of documents and information, both for public and private consumption. Most ECMS suites handle tasks like document review workflows, business process management, archiving, access control, and other powerful tools for actually managing content.

For most personal and small business websites, such huge tools are overkill. Over the past several years, there has been an explosion of simpler “CMS” systems to address this market gap. There have also been tools targeted for specific niches, the most well-known of which is the blogging CMS WordPress.

Because the content management needs of these non-enterprise ventures are minimal, most of these CMSes focus the majority of their effort on “themes” and “plugins” and other ways of managing the public visual appearance of the website. In fact, the rich ecosystem of themes and plugins is one of the reasons that WordPress continues to thrive. People call these systems “CMS,” but really they are using the term as a new word meaning “web interface for managing the presentation layer and piping content into it.”

News organizations, finally beginning to realize the need for a more agile web presence, have begun using such CMSes for their websites. Where news organizations differ from most business and personal websites, and even most blogs, is that they are constantly publishing new content and do not linger on the past. Yet few “CMS”es really provide management and production tools that these newsrooms need.

Some organizations have begun to go “web-first,” meaning that they first post their content on their website and then later bring the content “down” to their print editions.

Web-first versus Print-first

Why is the production process tied to the delivery process? Although “web-first” is often lauded as the correct step forward for many organizations, is hitching your wagon to your web CMS really any better than when it was hitched to your print CMS? Either way, you are limiting your production processes to those supported by the delivery system.

When the next big delivery platform comes along, is your organization going to pick up shop and change to make that the “first” delivery target? Are you going to go through the painful process of migrating your legacy content, rewriting your themes, retraining your staff, and rethinking your workflows? Why are we letting the delivery platform hold our content hostage?

News organizations should instead be “content-first,” and use tools that promote content above all else. We need first-class tools for managing the production process, and then for archiving, analyzing, referencing, and otherwise taking advantage of our past work.

Yes, your content will end up in multiple different delivery vectors. Each of those delivery platforms has its own infrastructure supporting actual presentation of the content (InDesign for print, WordPress themes/plugins for web, Objective-C for iOS). And each of those delivery platforms will have staff members who specialize in them and their capabilities (P&D staff for print, web designers and developers for websites, UX/UI designers and programmers for iOS). Let your content production staff (reporters, editors, photographers, etc.) have their own specific toolset optimized for content production.

Content Hub & Delivery Platforms

News organizations are in the information business. Without tools to leverage their information archives, news orgs can’t live up to their potential. Without tools to analyze and cross-pollinate content, what opportunities are being missed? Without tools for internal and external collaboration, how can news orgs share resources, reduce redundant coverage, and leverage public insight and input?

As news organizations with legacy Java ECMS systems from the 90’s begin to look towards the future, and as college newsrooms continue to adapt to the web and new business models, I think the time is right for a new generation of tools and services designed for streamlined, efficient, collaborative, and forward-thinking content production. We need to spend less time reinventing the presentation layer (“web CMS”) and instead focus on news content production systems (“NCPS”).

I intend to throw my hat in that ring, and I hope others will join me.

Multimedia

Being both an early riser and three hours ahead by virtue of time zones, I started my day working on some YDN tasks and left Courant for the afternoon. First order of business was finishing the new multimedia landing page. We’ve been without that page since launching on Courant at the beginning of the semester, and the version before was barely functionality as it was. Zeke Miller, the new Online Editor, was the driving force for the design while brainstorming and gathering feedback from the rest of the Online desk. I’m very pleased with the results, but have opened a thread over at the CoPress forums to get feedback from the wider college news community.

Search

Next on the docket was finishing the transition from Sphinx to Haystack for the Courant News search system. Sphinx was very nice, but Haystack opens new doors and fits better with the Courant News project philosophy; it allows for the use of multiple search engine backends, so that sites can use whatever is most appropriate for their situation.

For example, if a site cannot install a powerful Java system like Solr, then they can use the pure-Python Whoosh backend (which is also nice for development use). In practice, I ran into problems while trying to use the Xapian (fast C++ engine) backend and had to switch back to the resource-heavy Solr. Hopefully as Haystack and its Xapian backend mature the situation will improve.

By the end of the afternoon, I had finished implementing it in Courant and had completed the templates for the YDN search system. Please feel free to explore. I’m especially proud of the filtering options, which allowing for filtering of the query results by section, staff members, content type (article, photo, video, etc.), and date range. It makes it very easy to find specific content that you are looking for, or even to just get a broader sense of how popular a given topic is.

Solr’s relevancy engine is giving disproportionate weight to photos because they aren’t as long as articles, which is not very desirable. However, I’m still a Solr newbie and am trying to explore options for resolving that issue. Once I figure out how to do it with Solr, I’ll have to push the Haystack folks to expose any requisite functionality.

Headers App

Rob’s big project, and his first major code contribution in several months, was the implementation of an app we’ve been designing for the past few days. He’s going to write a more in-depth post, so I won’t step on his toes too much and go too deeply into it right now; essentially, it allows you to vary the display of your news site based on HTTP header values, most importantly User-Agent and Referer [sic].

The biggest example is to show a different set of templates to visitors with mobile User-Agent strings by showing them .mobile templates (using the FileExtension middleware and render() function in Courant News). You can also define a cascade of extensions, so that you only have to customize certain parts of your site if you want, and show the standard .html version elsewhere; this could be interesting if you wanted to take advantage of new HTML5 functionality in browsers (User-Agents) that support it, while not having to duplicate templates on other pages (or if you want to customize only certain pages on your mobile site for iPhone users).

It also lets you simply set extra template context variables, so that you can show special content to people coming from certain referrers (e.g., Facebook or Twitter). There are countless uses for such functionality, I’ll let you come up with your own.

You can browse the headers app in our code repository; it only requires a bit more work before we will declare it complete, hopefully that will happen in the next several days. When Rob writes his post, I’ll update with a link.

Conclusion

Today was a productive day, with 9 commits to the Courant News repository. Hopefully the pace will keep up, and once these last few features are finalized the path will be clear to begin work on Nando, which will be the real cornerstone of the Courant News platform.

For your techies, some background: We have been using Subversion (svn) for source control, but there’s been a push to move towards git. The biggest holdup had been our dependence on svn:externals to pull in some other projects without having to fork them, which is a pretty poor reason. As a result, I’ve been seeking ways to get rid of that last vestige of svn so that we could allow people to use git if they prefer that. We intend to continue using svn for Courant’s core codebase, but we appreciate the fact that many devs prefer to use git for their workflow, so we don’t want to preclude that.

As a result, I’ve been seeking a solution to simultaneously get rid of svn:externals while also easing the process of installing and using Courant. Some posts by the famous djangonaut James Bennett, along with looking at projects like Pinax, convinced me that such a solution existed in pip with virtualenv. Virtualenv allows you to create an isolated environment on your computer in which you can install python packages without it polluting the rest of your system, while pip allows you to install python packages from the Chesseshop or from a repository (supports svn, git, bzr, and hg currently).

Now for everyone: To use Courant, you first have to install python (2.5 recommended), svn, git, and whatever database engine you’d like to use (we strongly recommend PostgreSQL, but MySQL works too). From there, you just download one file from the Courant servers, courant-boot.py. Running courant-boot.py with some options will install Courant and all of its python dependencies, including Django and many others, inside a fresh virtualenv. It can also set up a clean copy of our sample project, or it can checkout your own site project from an external repository.

python courant-boot.py myvirtualenv -n website -p default

That one command will create a new virtualenv called “myvirtualenv,” install everything, and make a copy of the “default” sample project in a folder called “website.” For the YDN, I can replace “default” with the URL to our private SVN repository, and it will check out the Courant-powered version of our site and be all ready to go. I don’t think it could be much simpler. It should work perfectly well across all the major operating systems, and I have personally tested it on Ubuntu and Windows. I’ll get Paul to verify it works on OSX before release.

More detailed installation instructions, including a number of other options and instructions for how to set things up for developing on Courant itself, will be included in the project documentation. This was one of our last hurdles before public release, so once we finalize some remaining details and get team consensus, we will be opening up the project site. Stay tuned.

One common way, adopted by the Django built-in admin app, as well as  a number of common Django reusable apps like django-tagging and django-mptt, is that of a registry system. I’ve been joking with one of my Courant cohorts, Robert Baskin (@rsbaskin), on twitter about registries, and I thought it was time to let everyone else in on the discussion.

An Example

One way to manage functionality across various content types is to define the set of content types in your settings file. A recent example is django-shorturls, which lets you generate short URLs for your content by specifiying the content types to expose with a given prefix/abbreviation.

# in settings.py
SHORTEN_MODELS = {
    'A': 'myapp.animal',
    'V': 'myapp.vegetable',
    'M': 'myapp.mineral'
}

This works wonderfully when you have full control of the codebase, but if we were to include django-shorturls in Courant itself, it would require news orgs to modify our code* to add in their own custom content types or tweak how our standard configuration works.

You can solve this problem by creating a registry system. Each model can then register itself with the URL shortening app, passing along the prefix parameter of its choice. For example:

# in myapp/models.py
from django.db import models
from courant.core.shorturls import shorturls

class Planet(models.Model):
    name = models.CharField(max_length=100)
    ...
shorturls.register(Planet, 'P')

In the courant.core.shorturls app, it keeps track of all the models that register with it, and then can use this registry in its internal code. It also means that if you want to change the prefix for, say, the built-in Article model from ‘A’ to ‘S’ (for story), you could do this:

# in myapp/models.py or any other place that will get automatically run by Django, such as an __init__.py
from courant.core.news.models import Article
from courant.core.shorturls import shorturls

shorturls.unregister(Article)
shorturls.register(Article, 'S')

In this manner you can tweak the default configuration, while also hooking your own new content types into our functionality.

The Proliferation Problem

The short URL app example above is a nice, clear example of the types of situations where registries make sense. But as I mentioned in the beginning of this post, there are a number of other apps that use registries, both within Courant and in the general Django community. There’s the Django admin, django-tagging, django-mptt (hierarchical relationships), and django-comment-utils on the outside. We also have the get tag which I described previously and a registry for the search system (upcoming post once a few bugs are fixed). Our Article model currently looks like this (much simplified):

# courant/core/news/models.py
from django.db import models
from courant.core.discussions.moderation import moderator, CourantModerator
from courant.core.gettag import gettag
from courant.core.search import search

class Article(models.Model):
    heading = ...
    ...
moderator.register(Article, CourantModerator)
gettag.register(Article, name_field='heading')
search.register(Article,
                fields=('heading', 'subheading', 'summary', 'body'),
                filter_fields=('section', 'display_type','status'),
                date_field='published_at',
                use_delta=True)

We haven’t actually written the short URLs app I described above, but likely will, in which case you can tack on yet another registration call there. As you can see, this starts to rapidly build up for the more commonly used models, although that’s admittedly a rather small percentage of all of our models.

For those who follow Rob or I on twitter, you may have noticed us joking about meta-registries, which would be a registry to help manage all of these individual registries. How that would actually work is up for debate, and is really nothing more than an inside joke (though not so inside anymore).

Alternatives?

In light of this potential problem of runaway registry creation, we’ve been considering some other options. For some of the more complicated registries, like for search, we’d most likely be better served by going to a declarative syntax like models themselves. django-haystack has taken this approach, and I actually much prefer it in many respects, and just haven’t yet gotten around to building something similar to work with django-sphinx (our search tool of choice; explanation of that decision for that future post on search). It still requires a small registry of its own, but the registration call is handled in a separate file from the model itself, similar to how the admin system works.

Another option could be to tack additional options onto the models’ internal Meta classes and customize the python metaprogramming that Django does to build python objects from your model definitions. A registration process would still be occuring behind the scenes, but you wouldn’t be required to interact with it directly. For the above Article example, it might look like this (again, the surrounding parts of the model are much simplified):

# courant/core/news/models.py
from django.db import models
from courant.core.discussions.moderation import CourantModerator

class Article(models.Model):
    heading = ...
    ...

    class Meta:
        # standard Django meta options
        ordering = '-published_at'

        # custom Courant meta options
        short_url_prefix = 'A'
        get_tag = {'name_field': 'heading'}
        moderator = CourantModerator
    ...

This is maybe slightly cleaner because you don’t need all the imports at the top of the file, but I’m not sure it gains you much in the end. I personally rather like the explicitness of importing everything and manually registering it. Note that the get_tag meta option in this example uses a python dictionary because there are actually a number of other optional parameters that you can pass to it, and you wouldn’t want to have a meta option for every single possible parameter. My biggest hesitation with going this route is that it probably involves mucking with metaprogramming and doing some behind-the-scenes magic, which I don’t really think is a favorable cost/benefit tradeoff.

Conclusion

I hope that explains some of my twitter ramblings over the past couple weeks, and gives some additional insight into how Courant will enable customization and extension of the platform without forking the code. Please post comments if you have any questions or thoughts or opinions.

* Technically, this example resides in the settings file, which the news org has full control over anyways and thus would have no problem modifying. But I think it makes more sense to define the prefixes with the models they are related to, and not hidden away in a setting.