Feedback on WP-API 1.0

Recently the WP-API team released version 1.0 of their new JSON-based API for WordPress. I’ve been following their progress from a distance, but now that it is under consideration for inclusion into WordPress core I thought it was time to take a closer look.

As context for those unaware, I was a large contributor to the WordPress XML-RPC API in 3.4 and 3.5 when we added a bunch of new features to that API. I was a minor contributor to the official WordPress mobile apps, and have been building my own (as-yet-unreleased) WordPress mobile app over the past few months. I’ve written libraries for the XML-RPC API in Python, C#, and JavaScript, and have consulted on numerous projects that consumed the API.

As such, most of this feedback is from the perspective of a client library author and mobile app developer. I will not address the actual PHP implementation, as other core devs/contributors are more qualified for that perspective.

High-level thoughts

Discovery & HATEOS

Good
    • Multiple easy methods of checking for the existence of WP-API in a WordPress installation
    • Link headers and ‘meta’ links in response body for relevant related API calls, like sub-collections (comments on a post) or resultset paging
    • Uses standard HTTP methods as expected for actions on collections and entities
Needs work
    • Meta links seem arbitrary – ‘author’ appears as both an embedded object in the post object, as well as a link in the meta links collection; terms is embedded in post, but comments are a link
    • No means of discovering routes based on content type, so clients must hardcode URLs for the top-level routes
    • No auto-generated schema to allow for discovery of what content types are supported on a particular server (other than checking for particular hard-coded routes)

Transport

Good
    • Default serialization is JSON, which is universally supported and easy to work with
    • Pluggable authentication, with plugins already built for Basic Auth, OAuth 1.0a, and WordPress cookie
Needs work
    • Need proof-of-concept plugin for at least one different serialization technology – YAML, Protobufs, XML, etc.
    • Need to figure out plan for client oAuth registration against many sites – Ryan has outline of options, needs to be solved before general purpose mobile clients can be developed
    • Need plan and documentation for how clients are supposed to detect available authentication schemes
    • “Context” abstraction does not give adequate flexibility on response structure – XML-RPC added “fields” parameter to allow for customization, may need something similar

Forward & Backward Compatibility

Good
    • WP-API team has acknowledged this will likely be an issue in the future
Needs work
    • Needs to be resolved before shipping in core, especially versioning of entity schemas – see below
    • Need to consider needs of both “core” API (posts, terms, comments, etc.) and APIs added by plugins (e.g., BuddyPress, Pods)

Documentation

Good
    • Quality documentation on the major topics that need to be covered
    • Schemas documented in both human-readable Markdown and machine-readable JSON
Needs work
    • Would be better if schema and route documentation were auto-generated rather than manually written – humans make mistakes, and plugin authors likely won’t have same level of documentation quality if it requires manual work

Musings on content types

Clients for the WP-API 1.0 release have to hard-code too much knowledge about routes. For example, the reference JS client has hard-coded path substrings rather than discovering them HATEOS-style from the root document. This means that sites cannot customize the API routes at all without breaking general-purpose clients; for example, if a site wanted to change their API to use different terminology like “document” instead of “post”, a client wouldn’t be able to interact with it even though it does actually understand the underlying schema.

Speaking of entity schemas, I think there have to be affordances for future changes in the API’s design and implementation. WordPress core has historically been relatively stable (though there’s long been talk of changes to taxonomies), so it’s perhaps not as critical for the core APIs. But if we expect plugins to extend the API with their own functionality, then there must be a mechanism for their authors to evolve the schemas over time.

This works in both directions. If I write code against a plugin’s 2014 schema, I don’t want that code to suddenly break in 2016 when the plugin’s schemas are dramatically changed. And if I write code in 2016, I should be able to talk to both the 2014 and 2016 schemas so that it works with WordPress installations that are slow to upgrade plugins. As a developer of a general-purpose client, I need the flexibility to handle those types of compatibility scenarios.

A potential reframing

To address some of these concerns, this is how I would consider framing the API:

Structural
    • Entities (e.g., post, comment, user, term, etc.) are defined by a schema that is identified as a vendor MIME-type. For example, ‘application/vnd.wordpress.post.v1.read+json’.
    • Collections are filter-able & page-able lists of a particular entity type.
    • Entities can contain sub-collections (e.g., post contains collection of comments).
    • Entities can expose “special” actions, that can be invoked via POST only (e.g., POST /posts/42/domagic).
    • The root document contains a map of MIME-type to collection URL.
Behavioral
    • Clients can use the ‘Accept’ HTTP header to specify the preferred MIME-types they would like to receive. If no Accept header is sent, the API can return the latest default type. This allows clients that care about forward and backward compatibility to negotiate appropriately with the server.
    • Clients can discover all URLs without any a priori knowledge other than the MIME-type schemas that they are programmed to understand, by using the index’s map to find a collection URL and then following links to the entities, sub-collections, special actions, etc.
    • Clients can determine if a server supports a particular feature (e.g., has the BuddyPress plugin installed) by checking for the presence of one of its MIME-types in the index document.
    • Each entity type can be versioned independently from other entity types.
    • Plugins could add handlers for new MIME-types to existing entities, and clients that are aware of those special types could use them or fall back on the standard type for that entity.
    • Auto-generated schema and/or documentation would allow for tools to do interesting things with the API, like generate alternate schema representations, sandbox tools (e.g., Swagger, iodocs), code for libraries, and anything else that clever developers can think of.

By having consistent structure and behavior, it becomes much easier for flexible and long-lived clients to be created. Core and plugins would have consistent behavior with minimal effort on the part of plugin authors.

I haven’t looked closely enough at the 1.0 implementation to know how easy it would be to enforce these structures and behaviors, that will have to be the topic of further investigation and experimentation.

Conclusion

If WP-API is integrated into WordPress core, it will be around for a long time (xmlrpc.php was added May 23rd, 2004). We owe it to the community to put in some work and experimentation now to avoid years of potential pain and workarounds.

For my part, I’ll be attending at least three WordCamps this summer (Philly, Seattle, NYC) where I will be working on client libraries and proof-of-concept apps that use WP-API 1.0 (and any future iterations). If you will be at any of these events and want to talk about or hack on the API, please let me know!

9 Responses to Feedback on WP-API 1.0
  1. Daniel Bachhuber

    Great feedback, Max. Really appreciate the time you put into it.

  2. Dan

    Thanks for writing this up! I haven’t had a chance to look at the API yet. Are you going to use the API in your new mobile app?

  3. maxcutler

    Yes, I intend to start incorporating it into my app this summer. v1 of the app has been built primarily on the XML-RPC API, but it has an API abstraction layer to make it easy to swap in other APIs (we use WP.com API for some things when it’s available). I’ll write more about the app in future posts, including why I’m doing my own instead of contributing to the official ones.

  4. Bryan Petty

    I’m curious to know what kind of customization would actually require registering entities with a different endpoint than the common, understood ones. Why might you need to move “/posts/” or “/users/” to something else? Are we saying that core entities are required to share namespace with plugin entities? This question does make me wonder if plugins should be required to register under “/plugin/*” in order to officially provide core breathing room to add new entities without collisions with plugins, and also in order to make it easier to see when clients are using plugin entities that may not be available on all WordPress sites.

    I’m also curious to know what REST APIs you’ve written clients for that actually use custom vendor MIME types for all entities for the purpose of discovery. I’m not familiar with any myself. None of the following REST APIs do it: WordPress.com, Amazon AWS APIs, GitHub (but does use a custom one for versioning / back-compat), PayPal, Twitter, Facebook Graph API. Who says this is appropriate, and where has it been a proven technique? What makes the MIME type so special that it’s hardcoded types are any better than hardcoded paths? Do you think that requesting the list of supported MIME types during discovery is any different than requesting the list of supported endpoints?

    Speaking of back-compat, Ryan has already stated that we can handle versioning through a new header, and that really isn’t even required to be added before core integration.

  5. maxcutler

    Thanks for the comment Bryan.

    At a high-level, I would say that it’s dangerous to use SaaS APIs as an example/role-model for a WordPress API. SaaS companies have a different set of requirements and compatibility concerns than distributed software like WordPress. Yes, there are WordPress SaaS companies like WP.com, but there is a huge long tail of WP.org installations. Most SaaS companies also do not have plugin systems that encourage modification of their API structure.

    Reasons why someone might want to customize the API routes: they want their site’s API to reflect their information domain, so they could change /posts/ to /documents/ like I mentioned above; they want their site’s API to reflect their audience’s primary language, so they might change /posts/ to /entradas/. API clients would still work perfectly fine with these API variants, but only if they aren’t looking for specific URL patterns.

    Vendor MIME types were just a suggestion for one way to handle this discovery, but the basic requirement is just to have some unique string that is distinct from the URL path. MIME-types have some nice advantages like easy content negotation via Accept/Content-Type HTTP headers, and it’s easy to encode variants into the MIME-type name (variants by version, serialization format, “context”, etc.).

    I know that Ryan has mentioned on make/core that we can always just add a header later, but I’m not convinced that that’s a desirable approach. If we release something in core, there will a large number of clients/consumers for it immediately, and those clients will not make any effort to handle compatibility. When we go to introduce a v2, we’d likely have to make it opt-in to avoid breaking the v1 clients. It also doesn’t account for plugins that may need to rev their exposed APIs faster than core does, so if we punt on this for core we’re also hurting plugin authors’ ability to evolve their own APIs. I’d prefer that we just put some thought into this now and encourage people to write well-behaved clients out of the gate.

  6. Bryan Petty

    This is an API we’re talking about. Just because one WP site is in Spanish doesn’t mean it renames all of WP’s hooks with Spanish names so it’s relevant in any plugins they write for it. What you’re saying still doesn’t convince me that having hardcoded MIME type names is any different than having hardcoded URL endpoints. They don’t call them “universal” resource locators for no reason, and these aren’t intended or expected to be optimized for search engines. It’s just an unnecessary abstraction that makes the API slightly more complicated for absolutely no benefit.

    Why do you suppose REST APIs are significantly more popular than XML-RPC now? It’s not because it’s more powerful and extensible – it’s not. XML-RPC definitely has far more features technically speaking. It’s because XML-RPC has so much abstraction that it’s difficult to write clients for.

    You also still don’t mention a single example of where this approach is already used.

    How many REST API development tools have you used? Most aren’t designed to test REST API routes with MIME-based route discovery.

  7. maxcutler

    “XML_RPC definitely has far more features … [and] has so much abstraction that it’s difficult to write clients for”

    I’m not sure what you’re referring to, the XML-RPC spec is one of the simplest spec’s I’ve ever seen; it’s just a way of serializating basic data structures and invoking methods remotely. From years of talking to developers about XML-RPC, the #1 reason that people don’t like it is that it’s XML. People have either been burnt by “enterprise” XML systems (see SOAP) or have a pavlovian/cargo-cult response from being told by their peers that XML sucks. Every programming language has libraries that make it easy to consume XML-RPC services; WordPress has made mistakes in crafting our XML-RPC APIs that sometimes make it harder to consume, but that’s not XML-RPC’s fault (and it seems I should write a series of posts about the mistakes we’ve made and what to learn from them).

    The reason for the “ful” in “RESTful” APIs, is that most of these popular SaaS APIs are really just JSON APIs that use HTTP verbs, but otherwise don’t really follow Fielding’s original description of REST. For example, the WordPress.com “REST” API is basically a JSON-based RPC API. People like JSON because it’s universal and easy to consume, and most people don’t actually mind RPC-style APIs.

    My understanding is that Ryan has been aiming for a HATEOS-style REST API, so my feedback in this post was about some of the gaps in the current implementation. If the community decides that that’s not what they want, then so be it.

    With regards to your URI point, I think we just have different expectations about how this API will be used. In an era of WordPress as an Application Platform, I think we want sites to be proud of their APIs and encourage many and varied uses. It’s inevitable that some sites will want to customize the API routes to better match their target users; you dismissed the Spanish example, but there are developer communities around the world (e.g., Brazil, China) where it’s the norm to have code and APIs in languages other than English, especially when most programming languages support Unicode names. Remember, this is not a single API hosted by a single service, this is a huge federated network of sites that may have varying versions of WordPress installed with untold combinations of plugins.

    With regards to an example, you brought up GitHub earlier. GitHub supports MIME-type based versioning for exactly the reasons I’ve been discussing, they even call it out in their documentation: https://developer.github.com/v3/media/#request-specific-version

    I’ve used plenty of REST API development tools, and each makes their own assumptions about the API structure. Without a doubt, the WordPress community will have to write tools and libraries to work best with whatever API design we settle on, which is why I wrote this post from the perspective of someone (me) who intends to write at least two or three of them.

    It seems like I need to write some more posts with examples, so I’ll start working on that over the next few days.

  8. maxcutler

    Another example of MIME-type usage in a REST API is Heroku:

    https://devcenter.heroku.com/articles/api-compatibility-policy
    https://github.com/interagent/http-api-design#version-with-accepts-header

    As a SaaS API, they have the luxury of versioning at the API level. What I’ve proposed is that WP-API might need to do that at the entity-level to support plugins and other scenarios where we want more fine-grained control.

Leave a Reply