Tag Archives: howto

Talos, Remote Testing and Android

Last week I posted about mochikit.jar and what was done to enable testing on remote systems (specifically Android) for mochitest chrome style tests.  This post will discuss the work done to Talos for remote testing on Android.  I have been working with bear in release engineering a lot to flush out and bugs.  Now we are really close to turning this stuff on for the public facing tinderbox builds.

Talos + Remote Testing:

Last year, I had adding all the remote testing bits into Talos for windows mobile.  Luckily this time around I just had to clean up a few odds and ends (adding support for IPC).  Talos is setup to access a webserver and communicate with a SUTAgent (when you setup your .config file properly.)  This means you can have a static webserver on your desktop or the network and run talos from any SUTagent and a host machine.

Talos + Android:

This is a harder challenge to resolve than remote testing.  Android does not support redirecting to stdout which talos required.  For talos and all related tests (fennecmark, pageloader) we need to write to a log file from the test itself.

Run it for yourself:

Those are the core changes that needed to be made.  Here are some instructions for running it on your own:

hg clone http://hg.mozilla.org/build/talos

ln -s talos/ /var/www/talos #create a link on http://localhost/talos to the hg clone

python remotePerfConfigurator.py -v -e org.mozilla.fennec -t `hostname` -b mobile-browser –activeTests ts:tgfx –sampleConfig remote.config –browserWait 60 –noChrome –output test.config –remoteDevice 192.168.1.115 –webServer 192.168.1.102/talos

python run_tests.py test.config

* NOTE: 192.168.1.115 is the address of my android device (SUTAgent), and 192.168.1.102 is my webserver on my desktop

Leave a comment

Filed under testdev

mochikit.jar changes are in mozilla central

Last night we landed the final patches to make mochikit.jar a reality.  This started out as a bug where we would package all the mochikit harness + chrome tests into a single .jar file and then on a remote system copy that to the application directory and run the tests locally.  It ended up being much more than that, let me explain some of the changes that have taken place.

why change all of this?

In order to test remotely (on mobile devices such as windows mobile and android) where there are not options to run tools and python scripts, we need to put everything in the browser that it needs and launch the browser remotely.  The solution for tests that are not accessible over the network is to run them from local files.

what is mochikit.jar?

Mochikit.jar is an extension that is installed in the profile and contains all the core files that mochitest (plain, chrome, browser-chrome, a11y) needs to run in a browser.  This doesn’t contain any of the external tools such as ssltunnel and python scripts to set up a webserver.  When you do a build, you will see a new directory in $(objdir)/_tests/testing/mochitest called mochijar.  Feel free to poke around there.  As a standalone application all chrome://mochikit/content calls will use this extension, not a pointer to the local file system.  The original intention of mochkit.jar was to include tests data, but we found that to create an extension using jar.mn we needed a concrete list of files and that was not reasonable to do for our test files.  So we created tests.jar.

what is tests.jar?

tests.jar is the actual test data for browser-chrome, chrome, and a11y tests.  These are all tests that are not straightforward to access remotely over http, so we are running these locally out of a .jar file.  tests.jar is only created when you do a ‘make package-tests’ and ends up in the root of the mochitest directory as tests.jar.  If the harness finds this file, it copies it to the profile and generates a .manifest file for the .jar file, otherwise we generate a plain .manifest file to point to the filesytem.  Finally we dynamically registers tests.manifest from the profile.  Now all your tests will have a chrome://mochitests/content instead of chrome://mochikit/content.

What else changed?

A lot of tests had to change to work with this because we had hardcoded chrome://mochikit/content references in our test code and data.  It is fine to have that in there for references to the harness and core utilities, but to reference a local piece of test data, it was hardcoded and didn’t need to be.  A few tests required some more difficult work where we had to extract files temporarily to a temp folder in the profile and reference them with a file path.

what do I need to do when writing new tests?

please don’t cut and paste code then change it to reference a data, utility, or other uri that has chrome://mochikit/content/ in it.  If you need to access a file with the full URI or as a file path, here are some tips:

* a mochitest-chrome test that needs to reference a file in the same directory or subdir:
let chromeDir = getRootDirectory(window.location.href);

* a browser-chrome test that needs to reference a file in the same directory or subdir:
//NOTE: gTestPath is set because window.location.href is not always set on browser-chrome tests
let chromeDir = getRootDirectory(gTestPath);

* extracting files to temp and accessing them

  let rootDir = getRootDirectory(gTestPath);
  let jar = getJar(rootDir);
  if (jar) {
    let tmpdir = extractJarToTmp(jar);
    rootDir = "file://" + tmpdir.path + '/';
  }
  loader.loadSubScript(rootDir + "privacypane_tests.js", this);

Leave a comment

Filed under general, testdev

I dream of green

After spending a month trying to get my first round of large changes into the test harness system, I learned a lot of little things along the way. First off, I will admit I am not a rock star developer who can whip out perfect code without testing it (although I did a hot patch to talos that worked) or look at code and see what is wrong at a glance. So knowing that, I have found that I can be successful by testing my code and following up on it to ensure it works and doesn’t cause problems.

First off, this patch literally rewrote 90% of the code in all of the test harnesses. Now most of that was just moving functions inside of classes, but you can imagine the chance of failure. I started writing this patch in October and after a few review cycles, it was ready to land (checkin for you non Mozillians.) A month later we finally get it in…why the delay?

Bitrot: This usually is not that big of a problem. All patches will experience people editing code in the same areas, but in this case with HG I was unable to fix the rejects by just editing the code that had conflicts. With my patch(es) they modified such large blocks of code that a 3 line patch caused me a whole day of headaches and I found it easier to back out those specific changes (hg revert -r xyz [filename]), apply my patch, and add those changes on top of it. Not the most ideal thing, but it works.

Reading the patch: I found that many times I would accidentally have a change in a patch that I didn’t realize was there. So I would submit it for review and get odd questions or it bounced back to me. I strongly encourage after you do you diff or qdiff to read over the patch like a reviewer would before submitting it. If it is an update to a patch, I diff it against the last version.

TryServer: This is a great tool that Mozilla has setup where you can submit a patch and it will do a build and testpass on all platforms. So while I might develop my code on Linux or Windows, I don’t have to build and run all the tests on my Mac as TryServer will do it for me. I didn’t realize this, but TryServer does builds and tests differently than Tinderbox (at least for mozilla-central). For example there are no debug builds (and tests), test packaging (run out of the objdir) or leaktests. So this isn’t perfect, but helps if you are not changing platform specific code.

Random Orange: This is a term at Mozilla to describe tests that are known to fail randomly causing the test run to turn orange instead of green. So while watching the Tinderbox Push Log, you start seeing failures in your tests and need to investigate them. It is easy to search in bugzilla for the test name that failed, but one case we didn’t find it in bugzilla and backed out the patch due to a debug leaktest (which was a known random failure) when in fact the patch would have been fine.

Tinderbox Logs: While investigating failures (either random or caused by my patch), we need to look at the logs generated by the build and tests to see what happened. There are brief and full logs available, but I found that the brief logs were not that useful in telling me how the build was done, or what command was executed. So loading up the full logs it is…all 30Mb+ of data into one Firefox window = crash. I found safari to be better at loading these large logs but most useful was doing a wget and then searching through the log (it is a .gz extension, but it is really .html) for ‘BuildStep ended’ to find the right area to focus on.

Although I never ran into these issues for the dozens of other patches that I have had land, it must have been pure luck. Now whenever I am ready for submitting a patch, I plan to:

  1. Build on one OS in Opt and Debug
  2. For both builds run leaktests.py, mochitest, mochitest-chrome, browser-chrome, mochitest-a11y, mochitest-ipcplugins, reftest, crashtest, jsreftest
  3. Submit to tryserver and get all green (builds, talos, unittests)
  4. Review patch for mistakes, comments, extra code
  5. After review verify for bitrot, if bitrot- repeat

Leave a comment

Filed under testdev

Fennec proved more useful than Opera

Yesterday while driving to Mountain View from Oakland we were trying to find some dinner in the general Mountain View area. I pulled out my omnia II, launched Opera, and found a Thai restaurant in Palo Alto! Unfortunately with Opera I was unable to get a google map to load…FAIL.

I had a build of Fennec that I was using for mochitest development on the filesystem, so I launched it and tried to do the same search. Fully expecting a crash or two, I was surprised when I got search results in the same perceived amount of time as I did on Opera. The best thing was I could get a google map and zoom in/out to get the details I needed to figure out how to get to the restaurant.

End result: Fennec saved the day and proved itself as a useful browser. Time to clean it up and release Alpha4!

7 Comments

Filed under general, reviews

running httpd.js on a remote host

This exercise came about as an option for running unittests on windows mobile. We ran into an issue where the http server was running but not returning web pages. Vlad had mentioned running the web server on a host machine to be respectful of the memory on the device. In the past other members of the QA team have tried this with no luck. With the help of :waldo, I was able to get tests running while accessing a remote webserver.

For a quick background, all unittests rely on httpd.js running via xpcshell to serve web pages. For example, xpcshell update tests (test_0030_general.js) utilize httpd.js to download a .mar file and mochitest uses httpd.js for every test it runs. It is a little tricky to get this to run on a remote server as there are a few things to edit:

Now you need to fire up xpcshell to run as a server:

#set any env variables such as
#LD_LIBRARY_PATH=$firefox_dir/xulrunner on linux
cd $firefox_dir/xulrunner
./xpcshell -f ../../mochitest/httpd.js -f ../../mochitest/server.js

On the remote machine launch your test and be happy!

To really utilize this for the Fennec unittests on Windows Mobile, we will have to make many changes to the tests in addition to the ones mentioned above. Making it fit into automation would be very difficult as ip addresses can change and we might need to set it dynamically. Alternatively, running 1 host to many clients does have some attraction.

5 Comments

Filed under testdev

reftests for windows mobile

In my never ending quest to conquer the unittests, I have made another dent by getting reftests running (in a very basic form) on windows mobile!

It turns out this was much easier than I thought to do, but getting my dozen or so changes to automation.py and runreftest.py took many hours of debugging and trial and error. The whole process started out just to get a test to run, and the journey ended up touching just about every line of code.

I started off commenting everything out and bringing each chunk of code one at a time. This was a process that would take a full day, but knowing there was so many calls to subprocess and os, I knew I was up against a tough problem.

So began the slashing of os.path.* calls and writing stubs for anything that references subprocess or its returned process handle. What I found was I had no good way to debug, so I set out to verify Fennec could run a reftest by hand with a static profile that I munged together and I got results! Now can I get it working from runreftests.py? Not so lucky.

Problem #1 – trouble creating profile and crashing on rmdir:
Creating a profile. This seems simple and it appears to create a profile as expected in the code. For some reason the code kept crashing while trying to rmdir on the profile. Ugh, lets stick with a static profile and get these tests running. This was actually resolved when problem #2 was figured out.

Problem #2 – fennec restarts each time it is launched:
We call automation.runApp twice and for some reason it never terminates fennec after the first call so the second call fails. Ok, I can remove the first instance since it just registers the reftest.jar extension (remember we are using a static profile, so we can cheat). To make matters worse, this didn’t solve my problem. I found out that we were restarting Fennec every time we launched. This was a nasty problem that was easily resolved with adding a –environ:”NO_EM_RESTART=1″. Now I can have both the extension registration and the test.

Problem #3 – python script can’t open another python script process:
In order to run reftests on a small device we need to run them in smaller sets rather than all 3800+ at once. The big bottleneck is the memory required to store the tests and results until we are done. So I used my maemkit toolkit to call runreftest.py over and over again with a different .manifest file! No luck :( I cannot seem to get python to create a new process (using osce) when that process is another python instance. Ok, I can work around this and just call runreftest.py (main) from inside my maemkit. Problem solved!

Problem #4 – osce.blockingtee crashes python script on the third call in a row:
I noticed this while getting xpcshell up and running, and never had a solution. Unfortunately this doesn’t allow me to run the full test suite. The problem is blockingtee will write the output of the process to a given file. The alternative is to use osce.systema() instead. This works for the majority of the tests (a few appear to hang) and will be good enough to prove that I can get this done end to end.

So other than actually collecting the output, I have a working demo. To simplify my changes, I did this (note, the code is attached to bug 493792):

runreftest.py -
remove arg dependencies:

#  if len(args) != 1:
#    print >>sys.stderr, "No reftest.list specified."
#    sys.exit(1)
 

hardcode cli arguments:

#  reftestlist = getFullPath(args[0])
  reftestlist = r'\tests\reftest\tests\layout\reftests\image\reftest.list'
  options.app = r'\tests\fennec\fennec.exe'
  options.xrePath = r'\tests\fennec\xulrunner'

and use tempfile for mkdtemp as it would crash all the time:

#    profileDir = mkdtemp()
    profileDir = tempfile.mkdtemp()

Ok, that isn’t too bad. Now for automation.py, I needed to change a lot of things. Mostly what I did was change things based on platform:

platform = "wince"
if platform == "wince":
  import osce
else:
  import subprocess

and fix some of the hardcoded variables that are useless from the wince build:

#for wince: this defaults to c:\
DIST_BIN = r'\tests\fennec'
DEFAULT_APP = DIST_BIN + r'\fennec.exe'
CERTS_SRC_DIR = r'\tests\certs'

There is a Process class which manages the subprocess.Popen, so I do this:

if platform == "wince":
 class Process():
  def kill(self):
    pass
else:
 class Process(subprocess.Popen):
 ...

for the last set of changes, in runApp, I disable SSL Tunnel (will have to deal with this for mochitest):

  if platform == "wince":
    runSSLTunnel = False

and since we are not in cygwin, we need the real backslash:

   if platform == "wince":
      profileDirectory = profileDir + "\\"
      args.append(r'--environ:"NO_EM_RESTART=1"')
    else:
      profileDirectory = profileDir + "/"

For launching the actual application, I have to use osce.systema instead of Popen:

  if platform == "wince":
    proc = osce.systema(cmd, args)
    log.info("INFO | (automation.py) | Application pid: None")
  else:
    proc = Process([cmd] + args, env = environment(env), stdout = outputPipe, stderr = subprocess.STDOUT)
    log.info("INFO | (automation.py) | Application pid: %d", proc.pid)

I force outputPipe = None which skips a lot of the process interaction code:

  if debuggerInfo and debuggerInfo["interactive"]:
    outputPipe = None
  elif platform == "wince":
    outputPipe = None
  else:
    outputPipe = subprocess.PIPE

and finally I force the status = 0 so we can avoid more tinkering with the process handle:

  if platform == "wince":
    status = 0
  else:
    status = proc.wait()

That is really all it takes. I think a better approach for the long run is to create a subprocess.py which gives us a Popen and spoofs the other commands (wait, PIPE, stdout, stderr). Also debugging the blockingtee crash after three calls and integrating that into subprocess.py.

Leave a comment

Filed under testdev

using dom inspector with fennec

Today I dove in headfirst on automating bookmarks from browser-chrome. I normally read the code to figure out what is going on, but instead installed DOM Inspector and my life became much easier.

I found a post on mfinkle’s blog and started there. This didn’t work so well (we have incremented Fennec version since it was last modified) so I installed it on firefox and then copied the extension to fennec. This can be done by looking in your extensions directory inside your profile ~/.mozilla/firefox//extensions/inspector@mozilla.org/* and then copying all of that to ~/.mozilla/fennec//extensions/inspector@mozilla.org/.

This is all assuming you have installed Fennec and it uses your default profile directory. In my case, I have many versions of Fennec, so I have to go into my Fennec/extensions directory and copy the inspector@mozilla.org to it.

Ok, once installed, I fired up fennec, and hit CTRL+SHIFT+I and up popped the inspector. Being a n00b, I didn’t figure out the file->”inspect chrome document” for about 5 minutes, then it started making sense and I was able to drill down and figure out the layout inside of Fennec.

3 Comments

Filed under testdev