notes on a python webserver

Last week I created a python webserver as a patch for make talos-remote.  This ended up being frought with performance issues, so I have started looking into it.  I based it off of the profileserver.py that we have in mozilla-central, and while it worked I was finding my tp4 tests were timing out.

I come to find out we are using a synchronous webserver, so this is easy to fix with a ThreadingMixIn, just like the chromium perf.py script:

class MyThreadedWebServer(ThreadingMixIn, BaseHTTPServer.HTTPServer):
    pass

Now the test was finishing, but very very slowly (20+ minutes vs <3 minutes).  After doing a CTRL+C on the webserver, I saw a lot of requests hanging on log_message and gethostbyaddr() calls.  So I ended up overloading the log_message call and things worked.

class MozRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
    # I found on my local network that calls to this were timing out
    def address_string(self):
        return "a.b.c.d"

    # This produces a LOT of noise
    def log_message(self, format, *args):
        pass

Now tp4m runs as fast as using apache on my host machine.

converting xpcshell from listing directories to a manifest

Last year we ventured down the path of adding test manifests for xpcshell in bug 616999.  Finding a manifest format is not easy because there are plenty of objections to the format, syntax and relevance to the project at hand.  At the end of the day, we depend too much on our build system to filter tests and after that we have hardcoded data in tests or harnesses to run or ignore based on certain criteria.  So for xpcshell unittests, we have added a manifest so we can start to keep track of all these tests and not depend on iterating directories and sorting or reverse sorting head and tail files.

The first step is to get a manifest format for all existing tests.  This was landed today in bug 616999 and is currently on mozilla-central.  This requires that all test files in directories be in the manifest file and that the manifest file includes all files in the directory (verified at make time).  Basically if you do a build, it will error out if you forget to add a manifest or test file to the manifest.  Pretty straightforward.

The manifest we have chosen is the ini format from mozmill.  We found that there is no silver bullet for a perfect test manifest, which is why we chose an existing format that met the needs of xpcshell.  This is easy to hand edit (as opposed to json), is easy to parse from python and javascript.  As compared to reftests which have a custom manifest format, we needed to just have a list of test files and more specifically a way to associate a head and tail script file (not easy with reftest manifests).  The format might not work for everything, but it gives us a second format to work with depending on the problem we are solving.

Some notes about adding new tests to talos

Over the last year and a half I have been editing the talos harness for various bug fixes, but just recently I have needed to dive in and add new tests and pagesets to talos for Firefox and Fennec.  Here are some of the things I didn’t realize or have inconveniently forget about what goes on behind the scenes.

• tp4 is really 4 tests: tp4, tp4_nochrome, tp4_shutdown, tp4_shutdown_nochrome.  This is because in the .config file, we have “shutdown: true” which adds _shutdown to the test name and running with –noChrome adds the _nochrome to the test name.  Same with any test that us run with the shutdown=true and nochrome options.
• when adding new tests, we need to add the test information to the graph server (staging and production).  This is done in the hg.mozilla.org/graphs repository by adding to data.sql.
• when adding new pagesets (as I did for tp4 mobile), we need to provide a .zip of the pages and the pageloader manifest to release engineering as well as modifying the .config file in talos to point to the new manifest file.  see bug 648307
• Also when adding new pages, we need to add sql for each page we load.  This is also in the graphs repository bug in pages_table.sql.
• When editing the graph server, you need to file a bug with IT to update the live servers and attach a sql file (not a diff).   Some examples: bug 649774 and bug 650879
• after you have the graph servers updated, staging run green, review done, then you can check in the patch for talos
• For new tests, you also have to create a buildbot config patch to add the testname to the list of tests that are run for talos
• the last step is to file a release engineering bug to update talos on the production servers.  This is done by creating a .zip of talos, posting it on a ftp site somewhere and providing a link to it in the bug.
• one last thing is to make sure the bug to update talos has an owner and is looked at, otherwise it can sit for weeks with no action!!!

This is my experience from getting ts_paint, tpaint, and tp4m (mobile only) tests added to Talos over the last couple months.

Orange Factor and the WOO-Tang Clan

I have silently put up a tool call Orange Factor early last month as part of the War On Orange (WOO) project.  Over the last few weeks I have been iterating on this and working with jgriffin, jhammel, mcote and ctalbert (some have referred to us as the WOO-tang clan) to make this more useful and accurate.  Let me outline a few features of the site to give you a general introduction.

To start off with I know it takes a long time to load, but it should load in <30 seconds.   All the data is collected from bugs that are blocking randomorange.  This is done by parsing the comments and linked tinderbox logs to determine the frequency and type of failure.  We display a graph that tracks the cumulative orange factor (failure/push) over time.  NOTE: we are going off the number of pushes, not the number of tests ran.

 

Graph of the Orange Factor over time

Next there is  the Heatmap.  This is similar to what you see on tinderboxpushlog, except this is color code by the number of failures.

Overall HeatMap

From the HeatMap, you can click on a specific value to see more details about that test run (in the time range).  For example, here is OSX MoOth:

OSX MOth Testrun

Ok, this is really cool.  You can click on each day and filter down to the specific day, also at the top, you see the drop down select boxes.  This is super awesome because you can slice and dice up the data to view it just how you want.

Next I want to show you what the view looks like for a specific day.  On the left hand side of the webpage is a Calendar, you can click any day (I clicked Sept 11th) or click the day on a test run or orange factor graph (hover your mouse over the graph and a link will show up).

Daily Test Results for all tests by Platform

You should get the point that there are many ways to view the data.  Actually probably too much information!  So lately we have been working on some bug centric views.  To start off with, we have a topfails style report but this is based on bugs, not failures in log files.  To get here, click on the “Research and Top Bugs” link on the right hand side of the page.  Here is a  “weekly” view that is the top 5 bugs per week:

Top 5 Bugs every 7 days

Hover over the color bars to see the bug number and research it in more details.  Here is what you see when viewing a specific bug (544601):

Individual Bug Graph over Time

Orange Factor has much more to offer, just poke around and see how you can make it useful.  Feedback is welcome, and feel free to ask any questions in #ateam!

Talos, Remote Testing and Android

Last week I posted about mochikit.jar and what was done to enable testing on remote systems (specifically Android) for mochitest chrome style tests.  This post will discuss the work done to Talos for remote testing on Android.  I have been working with bear in release engineering a lot to flush out and bugs.  Now we are really close to turning this stuff on for the public facing tinderbox builds.

Talos + Remote Testing:

Last year, I had adding all the remote testing bits into Talos for windows mobile.  Luckily this time around I just had to clean up a few odds and ends (adding support for IPC).  Talos is setup to access a webserver and communicate with a SUTAgent (when you setup your .config file properly.)  This means you can have a static webserver on your desktop or the network and run talos from any SUTagent and a host machine.

Talos + Android:

This is a harder challenge to resolve than remote testing.  Android does not support redirecting to stdout which talos required.  For talos and all related tests (fennecmark, pageloader) we need to write to a log file from the test itself.

Run it for yourself:

Those are the core changes that needed to be made.  Here are some instructions for running it on your own:

hg clone http://hg.mozilla.org/build/talos

ln -s talos/ /var/www/talos #create a link on http://localhost/talos to the hg clone

python remotePerfConfigurator.py -v -e org.mozilla.fennec -t `hostname` -b mobile-browser –activeTests ts:tgfx –sampleConfig remote.config –browserWait 60 –noChrome –output test.config –remoteDevice 192.168.1.115 –webServer 192.168.1.102/talos

python run_tests.py test.config

* NOTE: 192.168.1.115 is the address of my android device (SUTAgent), and 192.168.1.102 is my webserver on my desktop

mochikit.jar changes are in mozilla central

Last night we landed the final patches to make mochikit.jar a reality.  This started out as a bug where we would package all the mochikit harness + chrome tests into a single .jar file and then on a remote system copy that to the application directory and run the tests locally.  It ended up being much more than that, let me explain some of the changes that have taken place.

why change all of this?

In order to test remotely (on mobile devices such as windows mobile and android) where there are not options to run tools and python scripts, we need to put everything in the browser that it needs and launch the browser remotely.  The solution for tests that are not accessible over the network is to run them from local files.

what is mochikit.jar?

Mochikit.jar is an extension that is installed in the profile and contains all the core files that mochitest (plain, chrome, browser-chrome, a11y) needs to run in a browser.  This doesn’t contain any of the external tools such as ssltunnel and python scripts to set up a webserver.  When you do a build, you will see a new directory in $(objdir)/_tests/testing/mochitest called mochijar.  Feel free to poke around there.  As a standalone application all chrome://mochikit/content calls will use this extension, not a pointer to the local file system.  The original intention of mochkit.jar was to include tests data, but we found that to create an extension using jar.mn we needed a concrete list of files and that was not reasonable to do for our test files.  So we created tests.jar.

what is tests.jar?

tests.jar is the actual test data for browser-chrome, chrome, and a11y tests.  These are all tests that are not straightforward to access remotely over http, so we are running these locally out of a .jar file.  tests.jar is only created when you do a ‘make package-tests’ and ends up in the root of the mochitest directory as tests.jar.  If the harness finds this file, it copies it to the profile and generates a .manifest file for the .jar file, otherwise we generate a plain .manifest file to point to the filesytem.  Finally we dynamically registers tests.manifest from the profile.  Now all your tests will have a chrome://mochitests/content instead of chrome://mochikit/content.

What else changed?

A lot of tests had to change to work with this because we had hardcoded chrome://mochikit/content references in our test code and data.  It is fine to have that in there for references to the harness and core utilities, but to reference a local piece of test data, it was hardcoded and didn’t need to be.  A few tests required some more difficult work where we had to extract files temporarily to a temp folder in the profile and reference them with a file path.

what do I need to do when writing new tests?

please don’t cut and paste code then change it to reference a data, utility, or other uri that has chrome://mochikit/content/ in it.  If you need to access a file with the full URI or as a file path, here are some tips:

* a mochitest-chrome test that needs to reference a file in the same directory or subdir:
let chromeDir = getRootDirectory(window.location.href);

* a browser-chrome test that needs to reference a file in the same directory or subdir:
//NOTE: gTestPath is set because window.location.href is not always set on browser-chrome tests
let chromeDir = getRootDirectory(gTestPath);

* extracting files to temp and accessing them

  let rootDir = getRootDirectory(gTestPath);
  let jar = getJar(rootDir);
  if (jar) {
    let tmpdir = extractJarToTmp(jar);
    rootDir = "file://" + tmpdir.path + '/';
  }
  loader.loadSubScript(rootDir + "privacypane_tests.js", this);

More info on the localhost->mochi.test change

This is just another public service announcement about the upcoming change to the unit tests. Last week I wrong about ‘mochi.test is the new localhost’, and this week I am reiterating that with more details.

After some feedback on the bug for this, we now have these options available:
* mochi.test
* 127.0.0.1
* moztest

With these options for referencing data in a test, you should be able to cover all the different networking scenarios that need testing. Keep in mind that mochi.test is the primary host and if you are posting a message to the parent window, the default domain is mochi.test, not localhost.

There is one other scenario where you need to get the real ip address of the server and this can be done with code like this:
var ios = Cc["@mozilla.org/network/io-service;1"].getService(Components.interfaces.nsIIOService);
var pps = Cc["@mozilla.org/network/protocol-proxy-service;1"].getService();
var uri = ios.newURI(“http://example.com&#8221;, null, null);
var pi = pps.resolve(uri, 0);
var host = “http://&#8221; + pi.host + “:” + pi.port;

From all the tests, test_prompt_async is the only test that needs this type of solution.

When we start running tests on windows mobile using the remote webserver, there won’t be any other changes needed other than what is mentioned above. Look for a demo from Clint at an upcoming Monday meeting.

status of the winmo automated tests project

I have been posting about this project for a while, so I figured I should give an update. Currently patches are landing and we are starting to get the final set of patches ready for review.

• Talos: This was the first part of this project and we have checked in 3 of the 4 patches to get Talos TS running. There is 1 patch remaining which I need to upload for review
• Mochitest: There are 4 patches required for this to work:
  1. Fix tests to not use hardcoded localhost – early review stages
  2. Add CLI options to mochitest for remote webserver – I need to cleanup my patch for review, at the end game
  3. Add devicemanager.py to the source tree – review started, waiting on sutagent.exe to resolve a few minor bugs
  4. Add runtestsremote.py to the source tree – review process started, waiting on other patches

  Good news is all 4 patches are at the review stage

• Reftest: This requires 4 patches (1 is devicemanager.py from mochitest)
1. Modify reftest.jar to support http url for manifest and test files – up for review
2. Refactor runreftests.py – up for review
3. Add remotereftests.py to source tree – needs work before review, but WIP posted

Keep in mind here we are still blocked on registering the reftest extension. I also have instructions for how to setup and run this.

• Xpcshell: this requires 3 patches (1 is device manager) and is still in WIP stages. There are two pieces to this that we still need to resolve: copying over the xpcshell data to the device and setting up a webserver to serve pages. Here are the two patches to date:
  1. Refactor runxpcshelltests.py to support subclass for winmo – WIP patch posted, close to review stage
  2. Add remotexpcshelltests.py to source tree – WIP patch posted

  I have written some instructions on how to run xpcshell tests on winmo if you are interested.

Stay tuned for updates when we start getting these patches landed and resolving some of our device selection/setup process.

I dream of green

After spending a month trying to get my first round of large changes into the test harness system, I learned a lot of little things along the way. First off, I will admit I am not a rock star developer who can whip out perfect code without testing it (although I did a hot patch to talos that worked) or look at code and see what is wrong at a glance. So knowing that, I have found that I can be successful by testing my code and following up on it to ensure it works and doesn’t cause problems.

First off, this patch literally rewrote 90% of the code in all of the test harnesses. Now most of that was just moving functions inside of classes, but you can imagine the chance of failure. I started writing this patch in October and after a few review cycles, it was ready to land (checkin for you non Mozillians.) A month later we finally get it in…why the delay?

Bitrot: This usually is not that big of a problem. All patches will experience people editing code in the same areas, but in this case with HG I was unable to fix the rejects by just editing the code that had conflicts. With my patch(es) they modified such large blocks of code that a 3 line patch caused me a whole day of headaches and I found it easier to back out those specific changes (hg revert -r xyz [filename]), apply my patch, and add those changes on top of it. Not the most ideal thing, but it works.

Reading the patch: I found that many times I would accidentally have a change in a patch that I didn’t realize was there. So I would submit it for review and get odd questions or it bounced back to me. I strongly encourage after you do you diff or qdiff to read over the patch like a reviewer would before submitting it. If it is an update to a patch, I diff it against the last version.

TryServer: This is a great tool that Mozilla has setup where you can submit a patch and it will do a build and testpass on all platforms. So while I might develop my code on Linux or Windows, I don’t have to build and run all the tests on my Mac as TryServer will do it for me. I didn’t realize this, but TryServer does builds and tests differently than Tinderbox (at least for mozilla-central). For example there are no debug builds (and tests), test packaging (run out of the objdir) or leaktests. So this isn’t perfect, but helps if you are not changing platform specific code.

Random Orange: This is a term at Mozilla to describe tests that are known to fail randomly causing the test run to turn orange instead of green. So while watching the Tinderbox Push Log, you start seeing failures in your tests and need to investigate them. It is easy to search in bugzilla for the test name that failed, but one case we didn’t find it in bugzilla and backed out the patch due to a debug leaktest (which was a known random failure) when in fact the patch would have been fine.

Tinderbox Logs: While investigating failures (either random or caused by my patch), we need to look at the logs generated by the build and tests to see what happened. There are brief and full logs available, but I found that the brief logs were not that useful in telling me how the build was done, or what command was executed. So loading up the full logs it is…all 30Mb+ of data into one Firefox window = crash. I found safari to be better at loading these large logs but most useful was doing a wget and then searching through the log (it is a .gz extension, but it is really .html) for ‘BuildStep ended’ to find the right area to focus on.

Although I never ran into these issues for the dozens of other patches that I have had land, it must have been pure luck. Now whenever I am ready for submitting a patch, I plan to:

1. Build on one OS in Opt and Debug
2. For both builds run leaktests.py, mochitest, mochitest-chrome, browser-chrome, mochitest-a11y, mochitest-ipcplugins, reftest, crashtest, jsreftest
3. Submit to tryserver and get all green (builds, talos, unittests)
4. Review patch for mistakes, comments, extra code
5. After review verify for bitrot, if bitrot- repeat

First look at the new remote testing

It has been almost 2 months since my last post and I have been heads down on bringing a new framework to life.

I discussed the 4 pieces involved in bringing automation to a new platform, and now we have what is shaping up to be a great approach for resolving the infrastructure and harness development.

What we have is a device specific agent which is written in a native language (C/C++) and does a small number of things (launch process, collect output, copy files to and from, query status such as processes, memory, cpu, disk, and lastly identify the device and os). This tool needs to act as a telnet server allowing us to telnet to it and execute a series of commands. Brad Lassey has developed such a tool for WinMo/CE which works partially on Win32 as well. I spent a couple days testing this interface and hammering out a python library to interact with a remote device. Now scripting file copy and process launching is easy. Clint has also used this tool to get mochitest and xpcshell running on Windows CE based devices!

That takes care of most of our infrastructure, now we need to get this stuff working on our test harnesses. My first task (only cleaned up one atm) is talos (see attached patch). This required massive changes to the Talos codebase, but no changes functionally for desktop based talos.

I found that in adding support to Talos for a remote device (as well as initial work for xpcshell and mochitest) that once we develop a DeviceAgent for a given platform there will be almost no additional work required to start running tests! I might be out of a job!!!

Next week I am going to work with Aki to get talos up and running (trial runs…don’t expect true miracles) on winmo and reporting to a staging graph server. Following that, we will start cleaning up our other harness code and getting mochitest, xpcshell, and reftests underway.

Stay tuned for progress updates and more details about a much needed updating to the way automation is run!