Tag Archives: tools

converting xpcshell from listing directories to a manifest

Last year we ventured down the path of adding test manifests for xpcshell in bug 616999.  Finding a manifest format is not easy because there are plenty of objections to the format, syntax and relevance to the project at hand.  At the end of the day, we depend too much on our build system to filter tests and after that we have hardcoded data in tests or harnesses to run or ignore based on certain criteria.  So for xpcshell unittests, we have added a manifest so we can start to keep track of all these tests and not depend on iterating directories and sorting or reverse sorting head and tail files.

The first step is to get a manifest format for all existing tests.  This was landed today in bug 616999 and is currently on mozilla-central.  This requires that all test files in directories be in the manifest file and that the manifest file includes all files in the directory (verified at make time).  Basically if you do a build, it will error out if you forget to add a manifest or test file to the manifest.  Pretty straightforward.

The manifest we have chosen is the ini format from mozmill.  We found that there is no silver bullet for a perfect test manifest, which is why we chose an existing format that met the needs of xpcshell.  This is easy to hand edit (as opposed to json), is easy to parse from python and javascript.  As compared to reftests which have a custom manifest format, we needed to just have a list of test files and more specifically a way to associate a head and tail script file (not easy with reftest manifests).  The format might not work for everything, but it gives us a second format to work with depending on the problem we are solving.


Leave a comment

Filed under testdev

Some notes about adding new tests to talos

Over the last year and a half I have been editing the talos harness for various bug fixes, but just recently I have needed to dive in and add new tests and pagesets to talos for Firefox and Fennec.  Here are some of the things I didn’t realize or have inconveniently forget about what goes on behind the scenes.

  • tp4 is really 4 tests: tp4, tp4_nochrome, tp4_shutdown, tp4_shutdown_nochrome.  This is because in the .config file, we have “shutdown: true” which adds _shutdown to the test name and running with –noChrome adds the _nochrome to the test name.  Same with any test that us run with the shutdown=true and nochrome options.
  • when adding new tests, we need to add the test information to the graph server (staging and production).  This is done in the hg.mozilla.org/graphs repository by adding to data.sql.
  • when adding new pagesets (as I did for tp4 mobile), we need to provide a .zip of the pages and the pageloader manifest to release engineering as well as modifying the .config file in talos to point to the new manifest file.  see bug 648307
  • Also when adding new pages, we need to add sql for each page we load.  This is also in the graphs repository bug in pages_table.sql.
  • When editing the graph server, you need to file a bug with IT to update the live servers and attach a sql file (not a diff).   Some examples: bug 649774 and bug 650879
  • after you have the graph servers updated, staging run green, review done, then you can check in the patch for talos
  • For new tests, you also have to create a buildbot config patch to add the testname to the list of tests that are run for talos
  • the last step is to file a release engineering bug to update talos on the production servers.  This is done by creating a .zip of talos, posting it on a ftp site somewhere and providing a link to it in the bug.
  • one last thing is to make sure the bug to update talos has an owner and is looked at, otherwise it can sit for weeks with no action!!!

This is my experience from getting ts_paint, tpaint, and tp4m (mobile only) tests added to Talos over the last couple months.

Leave a comment

Filed under testdev

Orange Factor and the WOO-Tang Clan

I have silently put up a tool call Orange Factor early last month as part of the War On Orange (WOO) project.  Over the last few weeks I have been iterating on this and working with jgriffin, jhammel, mcote and ctalbert (some have referred to us as the WOO-tang clan) to make this more useful and accurate.  Let me outline a few features of the site to give you a general introduction.

To start off with I know it takes a long time to load, but it should load in <30 seconds.   All the data is collected from bugs that are blocking randomorange.  This is done by parsing the comments and linked tinderbox logs to determine the frequency and type of failure.  We display a graph that tracks the cumulative orange factor (failure/push) over time.  NOTE: we are going off the number of pushes, not the number of tests ran.


Orange Factor graph

Graph of the Orange Factor over time

Next there is  the Heatmap.  This is similar to what you see on tinderboxpushlog, except this is color code by the number of failures.

Overall HeatMap

Overall HeatMap

From the HeatMap, you can click on a specific value to see more details about that test run (in the time range).  For example, here is OSX MoOth:

OSX MOth Testrun

OSX MOth Testrun

Ok, this is really cool.  You can click on each day and filter down to the specific day, also at the top, you see the drop down select boxes.  This is super awesome because you can slice and dice up the data to view it just how you want.

Next I want to show you what the view looks like for a specific day.  On the left hand side of the webpage is a Calendar, you can click any day (I clicked Sept 11th) or click the day on a test run or orange factor graph (hover your mouse over the graph and a link will show up).

Daily Test Results for all tests by Platform

Daily Test Results for all tests by Platform

You should get the point that there are many ways to view the data.  Actually probably too much information!  So lately we have been working on some bug centric views.  To start off with, we have a topfails style report but this is based on bugs, not failures in log files.  To get here, click on the “Research and Top Bugs” link on the right hand side of the page.  Here is a  “weekly” view that is the top 5 bugs per week:

Top 5 Bugs every 7 days

Top 5 Bugs every 7 days

Hover over the color bars to see the bug number and research it in more details.  Here is what you see when viewing a specific bug (544601):

Individual Bug Graph over Time

Individual Bug Graph over Time

Orange Factor has much more to offer, just poke around and see how you can make it useful.  Feedback is welcome, and feel free to ask any questions in #ateam!


Filed under testdev

Talos, Remote Testing and Android

Last week I posted about mochikit.jar and what was done to enable testing on remote systems (specifically Android) for mochitest chrome style tests.  This post will discuss the work done to Talos for remote testing on Android.  I have been working with bear in release engineering a lot to flush out and bugs.  Now we are really close to turning this stuff on for the public facing tinderbox builds.

Talos + Remote Testing:

Last year, I had adding all the remote testing bits into Talos for windows mobile.  Luckily this time around I just had to clean up a few odds and ends (adding support for IPC).  Talos is setup to access a webserver and communicate with a SUTAgent (when you setup your .config file properly.)  This means you can have a static webserver on your desktop or the network and run talos from any SUTagent and a host machine.

Talos + Android:

This is a harder challenge to resolve than remote testing.  Android does not support redirecting to stdout which talos required.  For talos and all related tests (fennecmark, pageloader) we need to write to a log file from the test itself.

Run it for yourself:

Those are the core changes that needed to be made.  Here are some instructions for running it on your own:

hg clone http://hg.mozilla.org/build/talos

ln -s talos/ /var/www/talos #create a link on http://localhost/talos to the hg clone

python remotePerfConfigurator.py -v -e org.mozilla.fennec -t `hostname` -b mobile-browser –activeTests ts:tgfx –sampleConfig remote.config –browserWait 60 –noChrome –output test.config –remoteDevice –webServer

python run_tests.py test.config

* NOTE: is the address of my android device (SUTAgent), and is my webserver on my desktop

Leave a comment

Filed under testdev

mochikit.jar changes are in mozilla central

Last night we landed the final patches to make mochikit.jar a reality.  This started out as a bug where we would package all the mochikit harness + chrome tests into a single .jar file and then on a remote system copy that to the application directory and run the tests locally.  It ended up being much more than that, let me explain some of the changes that have taken place.

why change all of this?

In order to test remotely (on mobile devices such as windows mobile and android) where there are not options to run tools and python scripts, we need to put everything in the browser that it needs and launch the browser remotely.  The solution for tests that are not accessible over the network is to run them from local files.

what is mochikit.jar?

Mochikit.jar is an extension that is installed in the profile and contains all the core files that mochitest (plain, chrome, browser-chrome, a11y) needs to run in a browser.  This doesn’t contain any of the external tools such as ssltunnel and python scripts to set up a webserver.  When you do a build, you will see a new directory in $(objdir)/_tests/testing/mochitest called mochijar.  Feel free to poke around there.  As a standalone application all chrome://mochikit/content calls will use this extension, not a pointer to the local file system.  The original intention of mochkit.jar was to include tests data, but we found that to create an extension using jar.mn we needed a concrete list of files and that was not reasonable to do for our test files.  So we created tests.jar.

what is tests.jar?

tests.jar is the actual test data for browser-chrome, chrome, and a11y tests.  These are all tests that are not straightforward to access remotely over http, so we are running these locally out of a .jar file.  tests.jar is only created when you do a ‘make package-tests’ and ends up in the root of the mochitest directory as tests.jar.  If the harness finds this file, it copies it to the profile and generates a .manifest file for the .jar file, otherwise we generate a plain .manifest file to point to the filesytem.  Finally we dynamically registers tests.manifest from the profile.  Now all your tests will have a chrome://mochitests/content instead of chrome://mochikit/content.

What else changed?

A lot of tests had to change to work with this because we had hardcoded chrome://mochikit/content references in our test code and data.  It is fine to have that in there for references to the harness and core utilities, but to reference a local piece of test data, it was hardcoded and didn’t need to be.  A few tests required some more difficult work where we had to extract files temporarily to a temp folder in the profile and reference them with a file path.

what do I need to do when writing new tests?

please don’t cut and paste code then change it to reference a data, utility, or other uri that has chrome://mochikit/content/ in it.  If you need to access a file with the full URI or as a file path, here are some tips:

* a mochitest-chrome test that needs to reference a file in the same directory or subdir:
let chromeDir = getRootDirectory(window.location.href);

* a browser-chrome test that needs to reference a file in the same directory or subdir:
//NOTE: gTestPath is set because window.location.href is not always set on browser-chrome tests
let chromeDir = getRootDirectory(gTestPath);

* extracting files to temp and accessing them

  let rootDir = getRootDirectory(gTestPath);
  let jar = getJar(rootDir);
  if (jar) {
    let tmpdir = extractJarToTmp(jar);
    rootDir = "file://" + tmpdir.path + '/';
  loader.loadSubScript(rootDir + "privacypane_tests.js", this);

Leave a comment

Filed under general, testdev

More info on the localhost->mochi.test change

This is just another public service announcement about the upcoming change to the unit tests. Last week I wrong about ‘mochi.test is the new localhost’, and this week I am reiterating that with more details.

After some feedback on the bug for this, we now have these options available:
* mochi.test
* moztest

With these options for referencing data in a test, you should be able to cover all the different networking scenarios that need testing. Keep in mind that mochi.test is the primary host and if you are posting a message to the parent window, the default domain is mochi.test, not localhost.

There is one other scenario where you need to get the real ip address of the server and this can be done with code like this:
var ios = Cc[“@mozilla.org/network/io-service;1”].getService(Components.interfaces.nsIIOService);
var pps = Cc[“@mozilla.org/network/protocol-proxy-service;1”].getService();
var uri = ios.newURI(“http://example.com&#8221;, null, null);
var pi = pps.resolve(uri, 0);
var host = “http://&#8221; + pi.host + “:” + pi.port;

From all the tests, test_prompt_async is the only test that needs this type of solution.

When we start running tests on windows mobile using the remote webserver, there won’t be any other changes needed other than what is mentioned above. Look for a demo from Clint at an upcoming Monday meeting.

Leave a comment

Filed under qa, testdev

status of the winmo automated tests project

I have been posting about this project for a while, so I figured I should give an update. Currently patches are landing and we are starting to get the final set of patches ready for review.

  • Talos: This was the first part of this project and we have checked in 3 of the 4 patches to get Talos TS running. There is 1 patch remaining which I need to upload for review
  • Mochitest: There are 4 patches required for this to work:
    1. Fix tests to not use hardcoded localhost – early review stages
    2. Add CLI options to mochitest for remote webserver – I need to cleanup my patch for review, at the end game
    3. Add devicemanager.py to the source tree – review started, waiting on sutagent.exe to resolve a few minor bugs
    4. Add runtestsremote.py to the source tree – review process started, waiting on other patches

    Good news is all 4 patches are at the review stage

  • Reftest: This requires 4 patches (1 is devicemanager.py from mochitest)
    1. Modify reftest.jar to support http url for manifest and test files – up for review
    2. Refactor runreftests.py – up for review
    3. Add remotereftests.py to source tree – needs work before review, but WIP posted

    Keep in mind here we are still blocked on registering the reftest extension. I also have instructions for how to setup and run this.

  • Xpcshell: this requires 3 patches (1 is device manager) and is still in WIP stages. There are two pieces to this that we still need to resolve: copying over the xpcshell data to the device and setting up a webserver to serve pages. Here are the two patches to date:
    1. Refactor runxpcshelltests.py to support subclass for winmo – WIP patch posted, close to review stage
    2. Add remotexpcshelltests.py to source tree – WIP patch posted

    I have written some instructions on how to run xpcshell tests on winmo if you are interested.

Stay tuned for updates when we start getting these patches landed and resolving some of our device selection/setup process.


Filed under testdev