Talos, Remote Testing and Android

Last week I posted about mochikit.jar and what was done to enable testing on remote systems (specifically Android) for mochitest chrome style tests.  This post will discuss the work done to Talos for remote testing on Android.  I have been working with bear in release engineering a lot to flush out and bugs.  Now we are really close to turning this stuff on for the public facing tinderbox builds.

Talos + Remote Testing:

Last year, I had adding all the remote testing bits into Talos for windows mobile.  Luckily this time around I just had to clean up a few odds and ends (adding support for IPC).  Talos is setup to access a webserver and communicate with a SUTAgent (when you setup your .config file properly.)  This means you can have a static webserver on your desktop or the network and run talos from any SUTagent and a host machine.

Talos + Android:

This is a harder challenge to resolve than remote testing.  Android does not support redirecting to stdout which talos required.  For talos and all related tests (fennecmark, pageloader) we need to write to a log file from the test itself.

Run it for yourself:

Those are the core changes that needed to be made.  Here are some instructions for running it on your own:

hg clone http://hg.mozilla.org/build/talos

ln -s talos/ /var/www/talos #create a link on http://localhost/talos to the hg clone

python remotePerfConfigurator.py -v -e org.mozilla.fennec -t `hostname` -b mobile-browser –activeTests ts:tgfx –sampleConfig remote.config –browserWait 60 –noChrome –output test.config –remoteDevice 192.168.1.115 –webServer 192.168.1.102/talos

python run_tests.py test.config

* NOTE: 192.168.1.115 is the address of my android device (SUTAgent), and 192.168.1.102 is my webserver on my desktop

tpan fennecmark – update

A quick update on what it took to get this showing up on the graph server.

As Aki tried to get this going, he kept running into issues. I looked into it and found out that he was using perfConfigurator.py to generate the config:

python PerfConfigurator.py -v -e /media/mmc1/fennec/fennec -t `hostname` -b mobile-browser --activeTests tzoom --sampleConfig mobile.config --browserWait 60 --noChrome --oldresultsServer graphs-stage.mozilla.org --oldresultsLink /server/bulk.cgi --output local.config


So I do this, run the test and it spits out a number. The problem is that it was not sending data up to the graph server as Aki pointed out in the console output (notice there are no links on the RETURN: graph links):

transmitting test: tzoom: 
		Stopped Sat, 13 Jun 2009 09:58:37
RETURN: graph links
RETURN:

RETURN:
Details:
|
Completed sending results: 
		Stopped Sat, 13 Jun 2009 09:58:37

After poking around I had no luck, so I asked Alice (who is the talos/graph-server expert). She had me try two things:
1) increase the # of cycles > 1 (I did 5 for the trial)
2) edit run_tests.py to include ‘tpan’ and ‘tzoom’ in the list of tests to look for (filed bug 497922)

That did the trick. I got output to the graph server and we are all set. All that remains is to check in the raw code for fennecmark.jar. This will be done in aki’s talos-maemo repository.

tpan – first draft

I got this wrapped up into patch and last night got it working on a deployed n810!

Here is what it took:

• creating a shell page that took queryString parameters and controlled fennecmark
• modifying fennecmark to take control commands
• modifying fennecmark to report data correctly
• modifying fennecmark to use a local webpage instead of a page on the internet
• modifying the talos mobile.config to support fennecmark

the shell page and control commands go hand in hand. Here I created a very simple .html page which utilized the eventdata to communicate between privileged and no privileged modes:

    var test = "";
    function parseParams() {
      var s = document.location.search.substring(1);
      var params = s.split('&');
      for (var i = 0; i < params.length; ++i) {
        var fields = params[i].split('=');
        switch (fields[0]) {
        case 'test':
          test = fields[1];
          break;
        }
      }
    }
    parseParams();
    if (test == "Zoom" || test == "PanDown") {
      var element = document.createElement("myExtensionDataElement");
      element.setAttribute("attribute1", test);
      document.documentElement.appendChild(element);

      var evt = document.createEvent("Events");
      evt.initEvent("myExtensionEvent", true, false);
      element.dispatchEvent(evt);
    }

This is a very simple and basic page, but it does the trick. On the fennecmark side, I created a listener which upon receiving an event, would get the testname to run and kick off fennecmark inside of overlay.js:

var myExtension = {
  myListener: function(evt) {
    var test = evt.target.getAttribute("attribute1");
    if (test == "Zoom") { BenchFE.tests = [LagDuringLoad, Zoom]; };
    if (test == "PanDown") { BenchFE.tests = [LagDuringLoad, PanDown]; };

    setTimeout(function() {BenchFE.nextTest(); }, 3000);
  }
}

The next modification to fennecmark is to fix the report.js script and conform to the talos reporting standards:

    if (pretty_array(this.panlag) == null) {
      tpan = "__start_report" + median_array(this.zoominlag) + "__end_report";
    }
    else {
      tpan = "__start_report" + this.pantime + "__end_report";
    }

One quirky thing here is since I am only running pan or zoom, I check if pan is null and print zoom, otherwise just print pan. I suspect as I near this code to a checkin state I will make this more flexible.

Lastly to finish this off, I need to point fennecmark at a page that is local, not on the internet. My initial stab at doing everything had me developing with the standalone pageset which doesn’t life on the production talos boxes. After some back and forth with Aki, I learned what I needed to do and modified pageload.js to do this:

browser.loadURI("http://localhost/page_load_test/pages/www.wikipedia.org/www.wikipedia.org/index.html", null, null, false);

Ok, now I have a toolset to work with. What do we need to do for talos to install and recognize it. I found out that in the .config file there is a section that deals with extensions:

# Extensions to install in test (use "extensions: {}" for none)
# Need quotes around guid because of curly braces
# extensions : 
#     "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
#     foo@sample.com : c:\path\to\other\unzipped\xpi
#extensions : { bench@taras.glek : /home/mozilla/Desktop/talos/tpan }
extensions : {}

#any directories whose contents need to be installed in the browser before running the tests
# this assumes that the directories themselves already exist in the firefox path
dirs:
  chrome : page_load_test/chrome
  components : page_load_test/components
  chrome : tpan/chrome

Here, I needed to add a new directory for chrome: tpan/chrome. In order to make this work, I needed to create a .jar file out of fennecmark instead of an unzipped extension in the profile (similar to dom inspector). This was a frustrating process until I found Ted’s wizard. After running the wizard, copying my code, tweaking config_build.sh to keep the .jar and running the build.sh script, I had what I needed.

The last step is to add the raw config to mobile.config so fennecmark will run:
tests :

- name: tpan
  shutdown: false
  url: tpan/fennecmark.html?test=PanDown
  resolution: 1
  cycles : 1
  timeout : 60
  win_counters: []
  unix_counters : []
- name: tzoom
  shutdown: false
  url: tpan/fennecmark.html?test=Zoom
  resolution: 1
  cycles : 1
  timeout : 60
  win_counters: []
  unix_counters : []

This leaves me at a good point where we can run fennecmark. Next steps are to solidify reporting, decide where to check in the fennecmark code (not just the .jar) and finally make any adjustments needed to get the installation, running and reporting well documented and stable.