Monthly Archives: April 2010

filtering mochitests for remote testing

One major problem we encounter in running all our unittests on Fennec is the large volume of failures and how to manage them.  Currently we have turned off mochitests on the Maemo tinderbox because nobody looked at the results (we still run reftest/crashtest/xpcshell!)

In many of my previous posts, I outlined methods for running tests remotely and that has proven to be very useful.  In order to test this code and continue developing it (without windows mobile or a working android build yet), I have developed a simple python test-agent that can run on a linux box (including n900.)  If you are curious, check it out and watch tests run remotely…it is pretty cool.

So the real problem I need to solve is how to not run a list of tests on a mobile device.  Solving this could get us to green faster and reduce the mochitest runtime in half!

In 2008 my solution was Maemkit.  Maemkit is just a small wrapper around the python test runner scripts that does some file (renaming) and directory (splitting into smaller chunks) manipulation to allow for more reliable test runs.  This has worked great and still works.  Enter remote testing and we need to hack up Maemkit a lot to accommodate for everything.  In addition a lot of the work maemkit does is already in the test runners.

Today I have moved the filtering to be a bit more configurable and less obscure.  This is really just a prototype and toolset to solve a problem for me locally, but the idea is something worth sharing.  What I have done is built up a json file (most of this was done automatically with this parsing script) which outlines the test and has some ‘tags’ that I can filter on:

   {
     "fennec-tags" : {"orange": "", "remote": "", "timeout": ""},
     "name" : "/tests/toolkit/content/tests/widgets/test_tooltip.xul"
   },
   {
     "fennec-results" : {"fail": 0, "todo": 0, "pass": 51},
     "name" : "/tests/MochiKit_Unit_Tests/test_MochiKit-Async.html",
     "note" : []
   }

You can see I now can run or skip tests that are tagged ‘orange’ or ‘timeout’. Better yet, I can skip tests with fennec-results that match fail > 0 if I want everything to be green.

So I took this a bit further since I wanted to turn these on or off depending on if I wanted to run tests or to investigate bugs, and I allowed for a filter language to be parsed in mochitest/tests/SimpleTest/setup.js.  I then modified my runtestsremote.py (subclass of runtests.py) so when launching mochitest from the command line I could control this like:

#run all tests that match the 'orange' tag
python runtestsremote.py --filter='run:fennec-tags(orange)'

#skip all tests that have the timeout tag
python runtestsremote.py --filter='skip:fennec-tags(timeout)'

#skip all tests that have failures > 0
python runtestsremote.py --filter='skip:fennec-results(fail>0)'

You should now see the power of this filtering and that with some more detailed thinking we could have a powerful engine to run what we want.  Of course this can run on regular mochitest (if you take the code in this patch from runtestsremote.py and add it to runtests.py.in) and run all the orange tests in a loop or something like that.

As a note, I previously mentioned a parsing script.   With some cleanup you could automatically create this json filter file based on tinderbox runs and fill in the tags to identify scenarios like orange (failing sometimes), timeouts, focus problems, etc…

Happy filtering!

Leave a comment

Filed under general, testdev

My impressions of Android

About 4 weeks ago I purchased a Motorola Droid as my personal device and I figured I would share some of the pros and cons. Keep in mind I have been playing with a lot of the latest and greatest devices in order to test Fennec, so I have some comparisons I can make.

Overall: a big win.

Store/Market: There are a lot of apps easily available for Android and the interface is easy to use. I found that getting apps for Windows Mobile was difficult (I was really trying to do that for a test case) and gave up. On Maemo (n900) there are a few apps in Ovi which are easy to install and use. Overall, I find Ovi to be sluggish and lacking apps that I searched for.

Calling: This has been very easy. I would say all three platforms are easy to call with. n900 gets a big win for SIP calling!

Typing: The Droid has a hard keyboard which really sucks. I love the n900 hard keyboard. Much to my surprise the virtual keyboard is easy to use. I have never been able to use a virtual keyboard due to the high error rate (bad eye sight for me). I found that putting the phone in landscape mode gives me much larger keys and I have 90% accuracy. One other bonus here is the dictionary uses words from my content (email, contacts) so I don’t have to type long names out. Note to Google and UI designers: if you take up the whole screen for a form fill UI, please let me know what element I am filling in!!!

Maps/Navigation: I find this to be top quality. The maps are pretty standard (you can get google maps for winmo phones), but the integration to make this work like a handheld GPS is awesome. Turn by turn directions and it is easy to use even while driving. Another ++ here is you can click on an address in the web browser and it takes you to the map (again for directions if you want)

Voice Search: probably the coolest feature yet. I have found this to be 98% accurate (at least for my voice and accent). When I see something, I just push the mic button on my search bar, say it (even with a lot of background noise) and I have results.

Email/Social networking: I like the built in gmail support, that is easy to use. The facebook app sucks and I prefer to use a web browser. Sharing pictures is easy, but rotating them needs some love.

Camera: Not as cool as the n900, but better than the Omnia 2. Pretty usable, but watch out for the ~3 second delay from taking the picture to seeing it. The gallery is easy to find your pictures and apps can access them with ease.

Hardware: It doesn’t weigh much (not as light as the Omnia 2) and isn’t too clunky despite my first impressions. I like the hard buttons on the touch screen, but I accidentally hit them all the time and end up doing crap I didn’t intend. Also it seems that shaking the device does random things, so don’t go running with it!

I am looking forward to running Firefox as my default browser. I fully expect the click to call and click to map functionality integration!

2 Comments

Filed under reviews