1) Mozbase.  Our efforts in our spare time is to refactor our test harnesses within Mozilla to share common code where possible, we call it mozbase.  I recommend doing a git clone of mozbase and getting it installed on your system: git clone git@github.com:mozilla/mozbase.git

2) Talos.  Next is to pick up a test harness.  We have been focusing on talos.  Mostly because you don’t have to pull the entire mozilla-central tree, do hour long builds, and really because the talos code base is in need of some serious updating.  To get talos, you need to clone it: hg clone http://hg.mozilla.org/build/talos

3) Configure Talos.  Talos is run in 2 steps right now.  A configuration step and a execution step.  The configuration step requires a path to firefox.exe as well as an active test (I use ts to keep it simple) is pretty easy:  “python PerfConfigurator.py –develop -a ts -e <path>/firefox.exe –output mytest.yml”.

4) Run Talos.  this step is easy.  Make sure you don’t have another instance of firefox.exe running on the computer and then run: “python run_tests.py -d -n mytest.yml”.

5) Take a look at some of these bugs that we have which are related to mozbase and/or talos:  http://bit.ly/tZHs3G

While this isn’t exhaustive or a perfect guide for how to work on the perfect bug in an hour or less, these 5 steps should get you setup to work on basic Mozilla code and start fixing bugs!  Pop into #ateam on irc.mozilla.org and ask some questions.

Now back to the other PI(e) that I always talk about!

So as the months went on from last December and up through August, we really made little progress.  A few tests were fixed, some disabled, some checks in place to make the boards stay online, but really no consistent set of test results.

There were a couple things that fixed our problems:

1) a rock star intern (:jchen) who found and fixed some workarounds with the OS so fennec wouldn’t crash all the time (issues with networking and libc).

2) a weekly meeting started by :blassey to go over all the bugs, status, issues, future work, and other items.

Both of these items are signs that the mobile development team was serious about testing and wanting to see Android unittests become a part of Mozilla.  While this seems trivial, it was next to impossible to keep tests running smoothly without support from the entire team.

Enjoy the reliable unit tests on Android!

A few weeks ago, I got nominated as a friend of a tree for being able to help out with mobile development get past issues with talos. Being a relatively new hire, I too am aware about how difficult it can be to get a testing environment set up, especially if you haven’t worked with mobile devices at all. Luckily, the first project I worked on this term is especially useful for those who can’t be bothered with setting up talos, or getting a Tegra and setting up the proper config for it.

Tegra Pool is an internal-only site for those who need to debug the issues in an automated testing run (such as on TBPL).  It can be found here: TegraPool. It will show you a table of all the devices available, and a couple forms to checkin and checkout.

If you are local and have the entire mobile testing suite set up, then it’s easy. Just put in your LDAP credentials and click “Check Out”. The IP of the Tegra you get will pop up, and not only will you be able to telnet and use the SUTAgent, but the AndroidDeviceBridge(ADB) will use TCP/IP, allowing you to connect with “adb connect <ip>”.

Remoties have a bit more of a problem, as most tests will require the Tegra to contact an “external server”, but we don’t want it to be making requests off-network. Also, many users don’t want to run the tests on their own computer, because they might need to run other things, or might not want to set up the entire testing environment. Luckily TegraPool solves these problems too.

If you are remote, or don’t have the time to set up the entire testing set, you can select the “I want server…” checkbox. To set up everything for you, you will have to point it to a test folder to get the app and zip files. This is best done with going to the build on Try in TinderBoxPushLog and clicking, “go to build directory”, when B is selected, and selecting the try-android-xul directory (or equivalent). Alternatively, ftp…mozilla-central-android  (or other folders in the nightly directory) is usually a good folder to use. This will set up a temporary account for you on the TegraPool server (based on your LDAP username).

Once you have checked out a Tegra and received a temporary account, you can now SSH into the machine (The password is a standard “giveMEtegra”). If you look in the home folder, you can see a lot of scripts that will just run. runMochiRemote.sh will run every single mochitest, runTalosRemote.sh will run the quick tpan test, and runRefRemote.sh will run the ref tests. If you connect to the device with ADB before running this, you should be able to pinpoint where issues are occurring.
This directory is a quick product, and should not limit what you can do. You can sftp new fennec.apk files, or modify the .sh files to run the necessary tests (i.e. other talos tests or specific mochitests).

Hopefully, this should let anybody who wants to debug mobile issues have a fast and easy option. Right now there are only 2 Tegra boards and 2 Panda boards running android, but if there is enough usage, more devices will be added. Happy debugging!

-Trevor (tfair on IRC)

In bug 704509, I have a patch to get reftests working with a Java front end.  This is really just using the XUL backend, but making it work with the limited support we have for addons and XUL, here are some differences:

• I am using a bootstrapped extension
• the reftest code needs to specify the window and document we are using
• I am not using a commandline handler, all options are set as preferences
• There is a nasty hack to attach our reftest <browser> to the default <window>

I need to make this work with our current reftest harness for Firefox.  So most of these changes will need to be cleaned up to work in a way acceptable to everybody and minimize the special case hacking for android.

So there are plenty of efforts underway to refactor talos to make it easier to expand.  This is fine and dandy, but for a developer wanting to help out or reproduce a bug it is next to impossible.  We have standalone talos, but that still requires some effort and hacking.

If you are interested in running talos, or if you have some pet peeve that you have encountered while running talos please file a bug, comment on existing bugs, or let us know in #ateam on irc.

For more data on specifics to our test failure distribution, please check out this spreadsheet and look at the different sheets.  We have been working for the last few weeks trying to reproduce these failures and the only concrete reproducible bug we could come up with was bug 691073.

We will continue to fix a few oranges that we see on the other tests as well as reduce the number of red/purple.

So what are these acceptable failures and what does 5% really mean. Failures can happen when we have tests which fail randomly, usually poorly written tests or tests which have been written a long time ago and hacked to work in todays environment. This doesn’t mean any test that fails is a problem, it could be a previous test that changes a Firefox preference on accident. For Android testing, this currently means the browser failed to launch and load the test webpage properly or it crashed in the middle of the test. Other failures are the device losing connectivity, our host machine having hiccups, the network going down, sdcard failures, and many other problems. With our current state of testing this mostly falls into the category of losing connectivity to the device. For infrastructure problems they are indicated as Red or Purple and for test related problems they are Orange.

I took at a look at the last 10 runs on mozilla-central (where we build Firefox nightlies from) and built this little graph:

Firefox Android Failures

Here you can see that our tests are causing 6.67% of the failures and 12.33% of the time we can expect a failure on Android.

We have another branch called mozilla-inbound (we merge this into mozilla-central regularly) where most of the latest changes get checked in.  I did the same thing here:

mozilla-inbound Android Failures

Here you can see that our tests are causing 7.77% of the failures and 9.89% of the time we can expect a failure on Android.

This is only a small sample of the tests, but it should give you a good idea of where we are.

I come to find out we are using a synchronous webserver, so this is easy to fix with a ThreadingMixIn, just like the chromium perf.py script:

class MyThreadedWebServer(ThreadingMixIn, BaseHTTPServer.HTTPServer):
    pass

Now the test was finishing, but very very slowly (20+ minutes vs <3 minutes).  After doing a CTRL+C on the webserver, I saw a lot of requests hanging on log_message and gethostbyaddr() calls.  So I ended up overloading the log_message call and things worked.

class MozRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
    # I found on my local network that calls to this were timing out
    def address_string(self):
        return "a.b.c.d"

    # This produces a LOT of noise
    def log_message(self, format, *args):
        pass

Now tp4m runs as fast as using apache on my host machine.

Currently this is only active on Mozilla-Central as we will run the numbers side by side to ensure we get a solid new baseline number.  In addition we upgraded the version of flash we are using and this seems to cause a small increase in the numbers as well.

We will run these side by side for a week and then we will turn off the non paint versions.  This will go branch by branch until we have no more side by side tests running.  If you look at the talos names, the original tests are marked as old_{testname} (i.e. old_tp, or old_chrome), and on the graph server the new tests are called {testname}_paint (i.e. tp_paint, tdhtml_paint, etc…)

One reason I found these interesting is that in a previous life I had attended a few years of Improv acting classes and did a short stint of real onstage Improv acting.  In looping back to these professional development sessions, they reminded me of the core concepts we learned in Improv 101.  So if you felt that you missed out, sign up for an Improv class.  Maybe if there are professional development sessions at a future event they could just have an Improv acting class.

Related to the professional development courses, I found that most of these were sparsely attended.  Of those that did attend the courses received great reviews/ratings.  To be fair, the technical tracks that I attended had about the same attendance records of the professional development tracks.  Maybe we are not creating sessions that are of interest to our audience?  I know for the technical tracks we just propose something and it magically becomes a session.  I don’t recall getting any input in what sessions would be available to me.  Maybe in the future we can do a better job of getting input from the community (a.k.a audience)!