types of data we care about in a manifest

This is a bit controversial (similar to “what OS do you run”), but I want to start outlining what I find as useful metadata to categorize tests with.

DATA:

Currently with Reftest, we have a sandbox that provides data the manifest files can use as conditional options.  The majority of the sandbox items used are:

• platform: MOZ_WIDGET_TOOLKIT (cocoa, windows, gtk2, qt)
• platform: xulrunner.OS, XPCOMABI (if “”, we are on ARM)
• environ:  haveTestPlugin, isDebugBuild, windowsDefaultTheme
• config:    getBoolPref, getIntPref, nativeThemePref (could be a getBoolPref)

This is the type of information that a large portion of our tests care about.  Most of these options are somehow coded into mochitests as well (through getPref calls, checking the OS, or in Makefile conditions).  I would like to figure out how to add this type of data:

• orange:   list of conditions this is random-orange on (but should pass)
• fails:        list of conditions this is expected to fail on
• e10s:       what service is used to cause this to fail under e10s
• fennec:   does this fail when run on fennec, which platforms, what versions
• remote:   does this fail when running tests remotely (required for android, originally for winmo)
• results:    number of failures (if orange or fails above) and number of todo
• runtime:  expected time to run this test (important on mobile)
• product:  product name and version
• future:     anything we find valuable in the future!

I can think of many ways to add this into the Reftest format or create a new format.  Looking at this data a but further, it really is not adding a lot of new information.  If we take the assumption that all tests are expected to pass in all configurations, any value assigned to a new piece of data would indicate that it fails under that given condition (or list of conditions).  As our supported platforms, configurations, and products grow, we will have a much greater need for this additional metadata.

INTEGRATION:

I would like to make all data pieces as tags vs raw conditions (Reftest does them like C conditions.)  This would allow much greater flexibility and adding data that doesn’t necessarily turn on/off a test.  For example, lets say a test is a random-orange for Firefox 1.9.1 (not 1.9.2), fails on Fennec Maemo 1.1 only, is orange on remote testing on Fennec android and currently is broken by e10s.  We could easily add those conditions to a list:

fails-if(OS==Maemo) fails-if(e10s==nsIPrefService) random-if(product=Firefox && xr.version==1.9.1) random-if(os=Android && remote==true) test_bitrot.html

So this is doable (please disregard any misused fails-if, random-if statements) and wouldn’t be too hard to add into a reftest.list style format for Mochitest (and even Reftest.)  Initially I thought it would be really cool to just run fails-if, random-if or skip-if statements with a small tweak to the command line.  This would give us the flexibility to turn on and off tests easier, but I realized that it would turn on/off all tests related to the condition.  I think a small adjustment in the format might allow for tags and we could tweak a run in the future with little work.  One example might be like:

fails(os=Maemo;e10s=nsIPrefService,nsIDOMGeoGeolocation) random(product=Firefox&xr.version=1.9.1;os=Android&remote=true) test_bitrot.html

This example is a minor change (which might not be needed), but helps simplify the syntax and keep in mind the idea of tags.  The backend details would need to be changed to support a ‘toggle’ of a tag in either scenario.  Maybe we just want to run e10s tests.  We can find all tests that have a e10s=nsiPrefService tag inside a fails tag block and just run those specific tests while maintaining all the other rules (skip on a certain OS or product).

There are still questions if the Reftest format is the right format for Mochitest.  It has a lot of weight since it works so well for so many of our tests.