Tag Archives: code

Some notes about adding new tests to talos

Over the last year and a half I have been editing the talos harness for various bug fixes, but just recently I have needed to dive in and add new tests and pagesets to talos for Firefox and Fennec.  Here are some of the things I didn’t realize or have inconveniently forget about what goes on behind the scenes.

  • tp4 is really 4 tests: tp4, tp4_nochrome, tp4_shutdown, tp4_shutdown_nochrome.  This is because in the .config file, we have “shutdown: true” which adds _shutdown to the test name and running with –noChrome adds the _nochrome to the test name.  Same with any test that us run with the shutdown=true and nochrome options.
  • when adding new tests, we need to add the test information to the graph server (staging and production).  This is done in the hg.mozilla.org/graphs repository by adding to data.sql.
  • when adding new pagesets (as I did for tp4 mobile), we need to provide a .zip of the pages and the pageloader manifest to release engineering as well as modifying the .config file in talos to point to the new manifest file.  see bug 648307
  • Also when adding new pages, we need to add sql for each page we load.  This is also in the graphs repository bug in pages_table.sql.
  • When editing the graph server, you need to file a bug with IT to update the live servers and attach a sql file (not a diff).   Some examples: bug 649774 and bug 650879
  • after you have the graph servers updated, staging run green, review done, then you can check in the patch for talos
  • For new tests, you also have to create a buildbot config patch to add the testname to the list of tests that are run for talos
  • the last step is to file a release engineering bug to update talos on the production servers.  This is done by creating a .zip of talos, posting it on a ftp site somewhere and providing a link to it in the bug.
  • one last thing is to make sure the bug to update talos has an owner and is looked at, otherwise it can sit for weeks with no action!!!

This is my experience from getting ts_paint, tpaint, and tp4m (mobile only) tests added to Talos over the last couple months.

Leave a comment

Filed under testdev

tests that require privileged access

I have been working on a project to get mochitests running on a build of Fennec + electrolysis.  In general, you can follow along in bug 567417.

One of the large TODO items in getting the tests to run is actually fixing the tests which use UniversalXPConnect.  So my approach was to grep through a mochitest tests/ directory for @mozilla and parse it out.  With a few corner cases, this resulted in a full list of services we utilize from our tests (here is a sorted list by frequency 76 total services.)  Cool, but that didn’t seem useful enough.  Then I took my work that I have done for filtering (the json file) and cross referenced that with my original list of tests that use UniversalXPConnect.

Now I have a list of 59 services which all should pass in Fennec (a mozilla-central build from 2 weeks ago on n900) along with the first filename of the test which utilizes that services!

What else would be useful?

Leave a comment

Filed under testdev

accessing privileged content from a normal webpage, request for example

One project which I am working on is getting mochitests to run in fennec + electrolysis.  Why this is a project is we don’t allow web pages to access privileged information in the browser anymore.  Mochitests in the simplest form use file logging and event generation.

The communication channel that is available between the ‘chrome’ and ‘content’ process is the messageManager protocol.  There are even great examples of using this in the unit tests for ipc.  Unfortunately I have not been able to load a normal web page and allow for my extension which used messageManager calls to interact.

I think what would be nice to see is a real end to end example of an extension that would demonstrate functionality on any given webpage.  This would be helpful to all addon authors, not just me:)  If I figure this out, I will update with a new blog post.

1 Comment

Filed under testdev

filtering mochitests for remote testing

One major problem we encounter in running all our unittests on Fennec is the large volume of failures and how to manage them.  Currently we have turned off mochitests on the Maemo tinderbox because nobody looked at the results (we still run reftest/crashtest/xpcshell!)

In many of my previous posts, I outlined methods for running tests remotely and that has proven to be very useful.  In order to test this code and continue developing it (without windows mobile or a working android build yet), I have developed a simple python test-agent that can run on a linux box (including n900.)  If you are curious, check it out and watch tests run remotely…it is pretty cool.

So the real problem I need to solve is how to not run a list of tests on a mobile device.  Solving this could get us to green faster and reduce the mochitest runtime in half!

In 2008 my solution was Maemkit.  Maemkit is just a small wrapper around the python test runner scripts that does some file (renaming) and directory (splitting into smaller chunks) manipulation to allow for more reliable test runs.  This has worked great and still works.  Enter remote testing and we need to hack up Maemkit a lot to accommodate for everything.  In addition a lot of the work maemkit does is already in the test runners.

Today I have moved the filtering to be a bit more configurable and less obscure.  This is really just a prototype and toolset to solve a problem for me locally, but the idea is something worth sharing.  What I have done is built up a json file (most of this was done automatically with this parsing script) which outlines the test and has some ‘tags’ that I can filter on:

     "fennec-tags" : {"orange": "", "remote": "", "timeout": ""},
     "name" : "/tests/toolkit/content/tests/widgets/test_tooltip.xul"
     "fennec-results" : {"fail": 0, "todo": 0, "pass": 51},
     "name" : "/tests/MochiKit_Unit_Tests/test_MochiKit-Async.html",
     "note" : []

You can see I now can run or skip tests that are tagged ‘orange’ or ‘timeout’. Better yet, I can skip tests with fennec-results that match fail > 0 if I want everything to be green.

So I took this a bit further since I wanted to turn these on or off depending on if I wanted to run tests or to investigate bugs, and I allowed for a filter language to be parsed in mochitest/tests/SimpleTest/setup.js.  I then modified my runtestsremote.py (subclass of runtests.py) so when launching mochitest from the command line I could control this like:

#run all tests that match the 'orange' tag
python runtestsremote.py --filter='run:fennec-tags(orange)'

#skip all tests that have the timeout tag
python runtestsremote.py --filter='skip:fennec-tags(timeout)'

#skip all tests that have failures > 0
python runtestsremote.py --filter='skip:fennec-results(fail>0)'

You should now see the power of this filtering and that with some more detailed thinking we could have a powerful engine to run what we want.  Of course this can run on regular mochitest (if you take the code in this patch from runtestsremote.py and add it to runtests.py.in) and run all the orange tests in a loop or something like that.

As a note, I previously mentioned a parsing script.   With some cleanup you could automatically create this json filter file based on tinderbox runs and fill in the tags to identify scenarios like orange (failing sometimes), timeouts, focus problems, etc…

Happy filtering!

Leave a comment

Filed under general, testdev

reftests for windows mobile

In my never ending quest to conquer the unittests, I have made another dent by getting reftests running (in a very basic form) on windows mobile!

It turns out this was much easier than I thought to do, but getting my dozen or so changes to automation.py and runreftest.py took many hours of debugging and trial and error. The whole process started out just to get a test to run, and the journey ended up touching just about every line of code.

I started off commenting everything out and bringing each chunk of code one at a time. This was a process that would take a full day, but knowing there was so many calls to subprocess and os, I knew I was up against a tough problem.

So began the slashing of os.path.* calls and writing stubs for anything that references subprocess or its returned process handle. What I found was I had no good way to debug, so I set out to verify Fennec could run a reftest by hand with a static profile that I munged together and I got results! Now can I get it working from runreftests.py? Not so lucky.

Problem #1 – trouble creating profile and crashing on rmdir:
Creating a profile. This seems simple and it appears to create a profile as expected in the code. For some reason the code kept crashing while trying to rmdir on the profile. Ugh, lets stick with a static profile and get these tests running. This was actually resolved when problem #2 was figured out.

Problem #2 – fennec restarts each time it is launched:
We call automation.runApp twice and for some reason it never terminates fennec after the first call so the second call fails. Ok, I can remove the first instance since it just registers the reftest.jar extension (remember we are using a static profile, so we can cheat). To make matters worse, this didn’t solve my problem. I found out that we were restarting Fennec every time we launched. This was a nasty problem that was easily resolved with adding a –environ:”NO_EM_RESTART=1″. Now I can have both the extension registration and the test.

Problem #3 – python script can’t open another python script process:
In order to run reftests on a small device we need to run them in smaller sets rather than all 3800+ at once. The big bottleneck is the memory required to store the tests and results until we are done. So I used my maemkit toolkit to call runreftest.py over and over again with a different .manifest file! No luck 😦 I cannot seem to get python to create a new process (using osce) when that process is another python instance. Ok, I can work around this and just call runreftest.py (main) from inside my maemkit. Problem solved!

Problem #4 – osce.blockingtee crashes python script on the third call in a row:
I noticed this while getting xpcshell up and running, and never had a solution. Unfortunately this doesn’t allow me to run the full test suite. The problem is blockingtee will write the output of the process to a given file. The alternative is to use osce.systema() instead. This works for the majority of the tests (a few appear to hang) and will be good enough to prove that I can get this done end to end.

So other than actually collecting the output, I have a working demo. To simplify my changes, I did this (note, the code is attached to bug 493792):

runreftest.py –
remove arg dependencies:

#  if len(args) != 1:
#    print >>sys.stderr, "No reftest.list specified."
#    sys.exit(1)

hardcode cli arguments:

#  reftestlist = getFullPath(args[0])
  reftestlist = r'\tests\reftest\tests\layout\reftests\image\reftest.list'
  options.app = r'\tests\fennec\fennec.exe'
  options.xrePath = r'\tests\fennec\xulrunner'

and use tempfile for mkdtemp as it would crash all the time:

#    profileDir = mkdtemp()
    profileDir = tempfile.mkdtemp()

Ok, that isn’t too bad. Now for automation.py, I needed to change a lot of things. Mostly what I did was change things based on platform:

platform = "wince"
if platform == "wince":
  import osce
  import subprocess

and fix some of the hardcoded variables that are useless from the wince build:

#for wince: this defaults to c:\
DIST_BIN = r'\tests\fennec'
DEFAULT_APP = DIST_BIN + r'\fennec.exe'
CERTS_SRC_DIR = r'\tests\certs'

There is a Process class which manages the subprocess.Popen, so I do this:

if platform == "wince":
 class Process():
  def kill(self):
 class Process(subprocess.Popen):

for the last set of changes, in runApp, I disable SSL Tunnel (will have to deal with this for mochitest):

  if platform == "wince":
    runSSLTunnel = False

and since we are not in cygwin, we need the real backslash:

   if platform == "wince":
      profileDirectory = profileDir + "\\"
      profileDirectory = profileDir + "/"

For launching the actual application, I have to use osce.systema instead of Popen:

  if platform == "wince":
    proc = osce.systema(cmd, args)
    log.info("INFO | (automation.py) | Application pid: None")
    proc = Process([cmd] + args, env = environment(env), stdout = outputPipe, stderr = subprocess.STDOUT)
    log.info("INFO | (automation.py) | Application pid: %d", proc.pid)

I force outputPipe = None which skips a lot of the process interaction code:

  if debuggerInfo and debuggerInfo["interactive"]:
    outputPipe = None
  elif platform == "wince":
    outputPipe = None
    outputPipe = subprocess.PIPE

and finally I force the status = 0 so we can avoid more tinkering with the process handle:

  if platform == "wince":
    status = 0
    status = proc.wait()

That is really all it takes. I think a better approach for the long run is to create a subprocess.py which gives us a Popen and spoofs the other commands (wait, PIPE, stdout, stderr). Also debugging the blockingtee crash after three calls and integrating that into subprocess.py.

Leave a comment

Filed under testdev

tpan – first draft

I got this wrapped up into patch and last night got it working on a deployed n810!

Here is what it took:

  • creating a shell page that took queryString parameters and controlled fennecmark
  • modifying fennecmark to take control commands
  • modifying fennecmark to report data correctly
  • modifying fennecmark to use a local webpage instead of a page on the internet
  • modifying the talos mobile.config to support fennecmark

the shell page and control commands go hand in hand. Here I created a very simple .html page which utilized the eventdata to communicate between privileged and no privileged modes:

    var test = "";
    function parseParams() {
      var s = document.location.search.substring(1);
      var params = s.split('&');
      for (var i = 0; i < params.length; ++i) {
        var fields = params[i].split('=');
        switch (fields[0]) {
        case 'test':
          test = fields[1];
    if (test == "Zoom" || test == "PanDown") {
      var element = document.createElement("myExtensionDataElement");
      element.setAttribute("attribute1", test);

      var evt = document.createEvent("Events");
      evt.initEvent("myExtensionEvent", true, false);

This is a very simple and basic page, but it does the trick. On the fennecmark side, I created a listener which upon receiving an event, would get the testname to run and kick off fennecmark inside of overlay.js:

var myExtension = {
  myListener: function(evt) {
    var test = evt.target.getAttribute("attribute1");
    if (test == "Zoom") { BenchFE.tests = [LagDuringLoad, Zoom]; };
    if (test == "PanDown") { BenchFE.tests = [LagDuringLoad, PanDown]; };

    setTimeout(function() {BenchFE.nextTest(); }, 3000);

The next modification to fennecmark is to fix the report.js script and conform to the talos reporting standards:

    if (pretty_array(this.panlag) == null) {
      tpan = "__start_report" + median_array(this.zoominlag) + "__end_report";
    else {
      tpan = "__start_report" + this.pantime + "__end_report";

One quirky thing here is since I am only running pan or zoom, I check if pan is null and print zoom, otherwise just print pan. I suspect as I near this code to a checkin state I will make this more flexible.

Lastly to finish this off, I need to point fennecmark at a page that is local, not on the internet. My initial stab at doing everything had me developing with the standalone pageset which doesn’t life on the production talos boxes. After some back and forth with Aki, I learned what I needed to do and modified pageload.js to do this:

browser.loadURI("http://localhost/page_load_test/pages/www.wikipedia.org/www.wikipedia.org/index.html", null, null, false);

Ok, now I have a toolset to work with. What do we need to do for talos to install and recognize it. I found out that in the .config file there is a section that deals with extensions:

# Extensions to install in test (use "extensions: {}" for none)
# Need quotes around guid because of curly braces
# extensions : 
#     "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
#     foo@sample.com : c:\path\to\other\unzipped\xpi
#extensions : { bench@taras.glek : /home/mozilla/Desktop/talos/tpan }
extensions : {}

#any directories whose contents need to be installed in the browser before running the tests
# this assumes that the directories themselves already exist in the firefox path
  chrome : page_load_test/chrome
  components : page_load_test/components
  chrome : tpan/chrome

Here, I needed to add a new directory for chrome: tpan/chrome. In order to make this work, I needed to create a .jar file out of fennecmark instead of an unzipped extension in the profile (similar to dom inspector). This was a frustrating process until I found Ted’s wizard. After running the wizard, copying my code, tweaking config_build.sh to keep the .jar and running the build.sh script, I had what I needed.

The last step is to add the raw config to mobile.config so fennecmark will run:
tests :

- name: tpan
  shutdown: false
  url: tpan/fennecmark.html?test=PanDown
  resolution: 1
  cycles : 1
  timeout : 60
  win_counters: []
  unix_counters : []
- name: tzoom
  shutdown: false
  url: tpan/fennecmark.html?test=Zoom
  resolution: 1
  cycles : 1
  timeout : 60
  win_counters: []
  unix_counters : []

This leaves me at a good point where we can run fennecmark. Next steps are to solidify reporting, decide where to check in the fennecmark code (not just the .jar) and finally make any adjustments needed to get the installation, running and reporting well documented and stable.

Leave a comment

Filed under testdev

automating fennec bookmark manager

This is more of a technical note and probably a boring read for anybody not interested in test development for fennec.

In an effort to add real automation for Fennec we have had issues making progress using browser-chrome to develop tests. Our latest bits of work come from bookmarks where happyhans is adding additional tests to what zad started (let me note that zad really paved the way for these tests on Fennec).

One thing we were trying to figure out is how to click on the edit button while managing bookmarks. It would be nice if we could just reuse test code from Firefox, but that is not an option. So using DOM Inspector and a lot of other help on IRC, browsing source code and just hacking, I have found the method to click a bookmark.


    var bookkmarkitems = chromeWindow.document.getElementById("bookmark-items");
    var bookmarkitem = window.document.getAnonymousElementByAttribute(bookmarkitems, "class", "bookmark-item");
    var editButton = window.document.getAnonymousElementByAttribute(bookmarkitem, "anonid", "edit-button");

A lot of this code :happyhans did. I found out the last two lines where we have a list of bookmarks and just need to find the “Edit” button to click.

What we see here is getting a list of bookmarkitems with getElementById limits our scope and returns a XULElement. Inside this Element, we have a list of “bookmark-item” classes which contain the bookmarks and the buttons to interact with them.

Since there is no ID on the bookmark-item, I query it with the getAnonymousElementByAttribute but here I start with my bookmark-items and look for a class named “bookmark-item”. This sound straightforward, but finding examples of other code to do this is not really possible.

Lastly, I took the bookmark-item (which defaults to the first item) and search for the “edit-button” as it has a “anonid”. I found out all of this information using the DOM Inspector (once mfinkle helped me use it properly).

So that is the basics. This code needs some real love to make it work with a specific title or URI instead of defaulting to the first item in the list. Knowing what I do now gives me some confidence in being able to figure out how to solve other problems in the bookmark manager.

Leave a comment

Filed under testdev