WordPress 3.4 finally shipped a few weeks ago, and now attention turns to 3.5. Feature discussions will start at the July 11 IRC dev chat, but in the meantime there will be a sprint to revise the unit tests suite. This post is to serve as starting guidance for the contributors who will be helping in this sprint. For background, you can read the kickoff IRC chat logs.
First, a quick review of the current unit test suite. It current has its own SVN repository and Trac instance, and uses a custom test runner based on PHPUnit. There are a few data fixtures for testing with variously sized blogs, as well as utility methods for generating test data and managing the state of the database. While passable, it leaves much to be desired.
Nikolay’s Clean Slate
- runs every test case in a clean WordPress install
- uses the default PHPUnit runner, instead of a custom one
- doesn’t encourage or support the usage of shared/prebuilt fixtures
- uses SQL transactions to clean up automatically after each test
Many of us contributors felt strongly that this was a better direction for the test suite, and have been experimenting with branches of Nikolay’s code over the past months.
After the aforementioned group chat this week, it was decided that we would sprint to overhaul the test suite before 3.5 feature work begins in earnest. So let’s review the new architecture and how to port existing tests.
One of the big differences is the use of InnoDB database transactions around each test:
To run the suite, you execute the basic phpunit command-line test runner or use your IDE-of-choice’s PHPUnit feature (e.g., NetBeans, PHPStorm). The phpunit.xml file will contain all necessary configuration, and the bootstrapper will install WordPress and set up a clean database for tests to work off of.
Each test case extends WP_UnitTestCase, which handles wrapping the test in a database transaction and rolling back to a clean state after each test.
At the end, phpunit will output a report with the results. Eventually we can add configuration for code coverage and Selenium testing as well.
One of the goals is to remove dependence on pre-baked data fixtures. This means that each test case is responsible for populating the database with whatever data it needs.
For this purpose, we’ve begun creating data factory classes based on those used by the GlotPress unit test suite. In essence, there is one factory class per “data type” (e.g., post, comment, user, term. etc.) that is responsible for creating and updating objects of that type.
The factory defines reasonable default values, so the test case only has to specify values for fields that it actually cares about. The factories can use each other as well, so a post factory can use the comments factory to create a bunch of comments on a post.
The current implementation is still half-baked and will evolve as we need it to as tests get ported. Here’s an example usage adapted from one of the XML-RPC test cases:
$post_ids = $this->factory->post->create_many( 17, array( 'post_type' => 'products' ) );
How to Port Tests
So how do we actually port the hundreds of existing tests to the new methodology? Here are some guidelines:
- Change the test case class to inherit from the new base classes: WP_UnitTestCase or WP_XMLRPC_UnitTestCase.
- Replace any instances of $this->_insert_quick_* with calls to $this->factory. If relying on the presence of $this->author, $this->post_ids, or similar, capture the necessary IDs from return values of the factory calls.
- If using one of the data fixtures, replace it with calls to $this->factory.
- Remove any teardown work that touches the database. All database writes will be undone by the DB transaction rollback, so any teardown work is wasted effort. However, you may still need to unset certain globals if you changed them.
- If a test was using a utility method, think carefully about whether we should add that method to the new baseclass. In some cases, the functionality should be added to factories. In others, the functionality can be put in /lib/functions.php.
- Add an @group annotation to make it easier to run subsets of the suite.
If in doubt, ask in #wordpress-dev or make a ticket on the UT Trac.
As mentioned, the goal is to sprint on this work for the next 1.5-2 weeks. It is expected for the suite to be broken for much of this period, but we should aim to have all tests ported and passing by the start of 3.5 feature development. To accomplish this, we should take a divide-and-conquer approach.
I think the best way is to divide the tests by component/area or by test case file.We’ve created new milestones on the UT trac for this project. We’ll create tickets for each file of existing tests, and the assigned owner will be responsible for porting that file’s tests. So if you want to help, assign one of the tickets to yourself and jump in. If you don’t have commit access to the UT repository, you can submit it as a patch on the ticket or ping a core dev on the IRC channel.
Starting shortly, the UT Trac and SVN notifications will be sent to wp-trac and wp-svn to increase awareness of on-going work. The wp-unit-tests mailing list may be used for general discussion if the other channels are insufficient. As always, #wordpress-dev is a good home for discussion on this work.
I’ll be working on getting the basics of Nikolay’s repository over into the UT SVN repository. By the end of today (Saturday), enough should be setup for everyone else to get started on test porting.