What is so special about 267194?

Today I landed the final patch for bug 761125.  This is a huge milestone for Firefox Android automation as we had been running 23,531 checks for each push by running known good directories of tests but now we will be running 290,725 checks by running ALL test cases except the 282 known test cases which fail, timeout, hang, crash.

I fully expect a lot of new random failures to crop up, this is just the nature of automated testing.

For the PI geeks out there, 267194 can be found at the 2051st digit of PI

Android unittests now have LOLcat

One common critcism of the android unittests is there is no logcat information.  I could go on for a long time about the decisions made from day 1 up to today, but the main thing is we have found adb to be very unreliable and we connect via telnet to a SUT agent on the tegra.  No usb cable or active adb session for all the automation we do.  This means collecting logcat information is not as easy as issuing a ‘adb logcat > testrun.log’. 

Today I landed bug 754873 which collects logcat information and puts it in the log file generated by tinderbox.  This does not capture the entire logcat session, but it does filter out the random noise that shows up in our logcat coming from tegras (dalvikvm and network wifi messages).  We will display up to 500 messages if they exist.  As logcat is a rotating queue of logs, it will be rare that we find a full 500 lines of useful information in the logs.  What this will solve is when we have a crash in Java or some other crash that is not detected by the crashreporter we will be able to see it in our tinderbox logs.

random fact: did you know you can do ‘adb lolcat’ and it will produce the same output as ‘adb logcat’

Reducing the Noise in Talos

Over the last year there has been a lot of research into reducing the noise in our talos performance numbers.  For example looking at tscroll, we have a fluctuation in the reported numbers of almost 400 (out of 14000).  Jan Larres took a look at this problem in his masters thesis, and found a variety of factors that did and didn’t contribute to the noise.  We actually have Bug 706912 filed to implement some of his suggestions on how to calculated the posted number.  Last fall, Stephen Lewchuk look at the raw data that was collected and found some inconsistencies in the way we were aggregating the data.  In short, we have a lot of ground to cover if we want to reduce our numbers.

Over the last couple months, we have been working on a project call Signal From Noise.  This is an attempt to fix the way we collect some numbers and redo the way we aggregate numbers for reporting.  We have done a lot of experimenting with the primary focus on tp5.  The way we run tp5 is to load each of the 100 pages once, then repeat 10 times.  For each page, we would drop the highest value and take the median value of the remaining 9 numbers.  This results in an array of 100 data points which get reported to the graph server.  We take those 100 data points and average them out to generate the single number for tp5.  It is easy to imagine that the small samples and median/average combination will produce a lot of noise.

Going forward, we are looking to change from column major to row major and collect 30 samples instead of 10.  This means we focus on one page and load it 30 times, then move to the next page and repeat until all 100 pages have been loaded.  The downfall is the runtime as we move from an average of 17 minutes to an average of 39 minutes for the entire tp5 run.  Collecting 30 samples will give us a much more meaningful number, but we also found that the first 5-10 iterations contain the most noise.  So initially we are looking to throw away the first 10 numbers instead of what we originally did by throwing away the highest number. When looking at the raw numbers (not the aggregated number), here are some graphs to highlight the difference:

 

This is only the first step in many changes needed.  After rolling this out, we need to evaluate the other test suites as well and ensure we are running adequate cycles to get a valid sample size.  We are also working on allowing the database to accept the raw values instead of the single median value per page.  Likewise are are looking to stop doing a average([median(page)]).  All of this will allow us to find regressions easier per page instead of having it washed over with the other numbers.

Love Mozilla, Love Python, Want to Help, What Next?

I have been asked a few times over the last couple months how to help out at Mozilla, specifically with python.  I know there are dozens of teams within Mozilla that have various python related projects.  I am on the automation and tools team at Mozilla (known as the A*Team) and we do a lot of python related work.  It seems that we are asked to add new and crazy stuff to harnesses or write new and interesting tools (usually a blend of python and javascript).

1) Mozbase.  Our efforts in our spare time is to refactor our test harnesses within Mozilla to share common code where possible, we call it mozbase.  I recommend doing a git clone of mozbase and getting it installed on your system: git clone git@github.com:mozilla/mozbase.git

2) Talos.  Next is to pick up a test harness.  We have been focusing on talos.  Mostly because you don’t have to pull the entire mozilla-central tree, do hour long builds, and really because the talos code base is in need of some serious updating.  To get talos, you need to clone it: hg clone http://hg.mozilla.org/build/talos

3) Configure Talos.  Talos is run in 2 steps right now.  A configuration step and a execution step.  The configuration step requires a path to firefox.exe as well as an active test (I use ts to keep it simple) is pretty easy:  “python PerfConfigurator.py –develop -a ts -e <path>/firefox.exe –output mytest.yml”.

4) Run Talos.  this step is easy.  Make sure you don’t have another instance of firefox.exe running on the computer and then run: “python run_tests.py -d -n mytest.yml”.

5) Take a look at some of these bugs that we have which are related to mozbase and/or talos:  http://bit.ly/tZHs3G

While this isn’t exhaustive or a perfect guide for how to work on the perfect bug in an hour or less, these 5 steps should get you setup to work on basic Mozilla code and start fixing bugs!  Pop into #ateam on irc.mozilla.org and ask some questions.

Now back to the other PI(e) that I always talk about!

How the mobile automation for Android became reliable in the last few months

Making the Mozilla automation infrastructure run reliably for each checkin on mobile devices has been my primary focus for the last few years.  Last year at this time we were just trying to get Android automation up and running, all tools and harnesses had been written and ready to run.  The core buildbot code for running the tests was in place.  The problem was that we just had so many failures of the devices (NVidia Tegra development boards) and the tests.

So as the months went on from last December and up through August, we really made little progress.  A few tests were fixed, some disabled, some checks in place to make the boards stay online, but really no consistent set of test results.

There were a couple things that fixed our problems:

1) a rock star intern (:jchen) who found and fixed some workarounds with the OS so fennec wouldn’t crash all the time (issues with networking and libc).

2) a weekly meeting started by :blassey to go over all the bugs, status, issues, future work, and other items.

Both of these items are signs that the mobile development team was serious about testing and wanting to see Android unittests become a part of Mozilla.  While this seems trivial, it was next to impossible to keep tests running smoothly without support from the entire team.

Enjoy the reliable unit tests on Android!

TegraPool – Bathing suit not required

I have had the honor of working with Trevor on a few projects during his internship at Mozilla.  One of earlier projects he worked on was TegraPool, a utility to check out a tegra and run tests on it as we do on tinderbox.  Trevor doesn’t have a blog setup or a feed to planet, so here is what Trevor has to say:

A few weeks ago, I got nominated as a friend of a tree for being able to help out with mobile development get past issues with talos. Being a relatively new hire, I too am aware about how difficult it can be to get a testing environment set up, especially if you haven’t worked with mobile devices at all. Luckily, the first project I worked on this term is especially useful for those who can’t be bothered with setting up talos, or getting a Tegra and setting up the proper config for it.

Tegra Pool is an internal-only site for those who need to debug the issues in an automated testing run (such as on TBPL).  It can be found here: TegraPool. It will show you a table of all the devices available, and a couple forms to checkin and checkout.

If you are local and have the entire mobile testing suite set up, then it’s easy. Just put in your LDAP credentials and click “Check Out”. The IP of the Tegra you get will pop up, and not only will you be able to telnet and use the SUTAgent, but the AndroidDeviceBridge(ADB) will use TCP/IP, allowing you to connect with “adb connect <ip>”.

Remoties have a bit more of a problem, as most tests will require the Tegra to contact an “external server”, but we don’t want it to be making requests off-network. Also, many users don’t want to run the tests on their own computer, because they might need to run other things, or might not want to set up the entire testing environment. Luckily TegraPool solves these problems too.

If you are remote, or don’t have the time to set up the entire testing set, you can select the “I want server…” checkbox. To set up everything for you, you will have to point it to a test folder to get the app and zip files. This is best done with going to the build on Try in TinderBoxPushLog and clicking, “go to build directory”, when B is selected, and selecting the try-android-xul directory (or equivalent). Alternatively, ftp…mozilla-central-android  (or other folders in the nightly directory) is usually a good folder to use. This will set up a temporary account for you on the TegraPool server (based on your LDAP username).

Once you have checked out a Tegra and received a temporary account, you can now SSH into the machine (The password is a standard “giveMEtegra”). If you look in the home folder, you can see a lot of scripts that will just run. runMochiRemote.sh will run every single mochitest, runTalosRemote.sh will run the quick tpan test, and runRefRemote.sh will run the ref tests. If you connect to the device with ADB before running this, you should be able to pinpoint where issues are occurring.
This directory is a quick product, and should not limit what you can do. You can sftp new fennec.apk files, or modify the .sh files to run the necessary tests (i.e. other talos tests or specific mochitests).

Hopefully, this should let anybody who wants to debug mobile issues have a fast and easy option. Right now there are only 2 Tegra boards and 2 Panda boards running android, but if there is enough usage, more devices will be added. Happy debugging!

-Trevor (tfair on IRC)

Work In Progress – making Talos easier to run

This quarter I became the proud owner of Talos (well at least for a quarter or two).  Over the last few years talos has not had much churn, but this year (2011 proper) we have seen addons, responsiveness, xperf, mozafterpaint and experiments with eideticker.  With all of this talos has grown and more people are working on writing patches for it.

So there are plenty of efforts underway to refactor talos to make it easier to expand.  This is fine and dandy, but for a developer wanting to help out or reproduce a bug it is next to impossible.  We have standalone talos, but that still requires some effort and hacking.

If you are interested in running talos, or if you have some pet peeve that you have encountered while running talos please file a bug, comment on existing bugs, or let us know in #ateam on irc.

Android Orange rate is 4.46%

Ok, the title might be misleading, but as of the last few days we are <5% orange for android unittests on mozilla-central.  The reason this was done is we have hidden J1 and R2 from the results.  We are tracking these in the weekly mobile automation meetings, and will continue to do so until those tests are live again.

For more data on specifics to our test failure distribution, please check out this spreadsheet and look at the different sheets.  We have been working for the last few weeks trying to reproduce these failures and the only concrete reproducible bug we could come up with was bug 691073.

We will continue to fix a few oranges that we see on the other tests as well as reduce the number of red/purple.

talos pageloader now supports timestamps using MozAfterPaint

Today we rolled out changes to talos such that tests that use the pageloader (chrome, nochrome, tp) will have the option to report the page load times after we receive a MozAfterPaint event instead of a Load event.

Currently this is only active on Mozilla-Central as we will run the numbers side by side to ensure we get a solid new baseline number.  In addition we upgraded the version of flash we are using and this seems to cause a small increase in the numbers as well.

We will run these side by side for a week and then we will turn off the non paint versions.  This will go branch by branch until we have no more side by side tests running.  If you look at the talos names, the original tests are marked as old_{testname} (i.e. old_tp, or old_chrome), and on the graph server the new tests are called {testname}_paint (i.e. tp_paint, tdhtml_paint, etc…)

 

 

5 minute challenge: fresh samsung tablet to mochitest results

I did an experiment today, I took my new shiny Samsung Galaxy tablet (which I received late yesterday) and tried to run mochitests on it with a stop watch running.  To preamble this, I had an existing objdir.  Here are a few of the things I had to do:

• fix my /etc/udev/rules.d/51-android.rules
• restart my adb server
• change my tablet settings to install from unknown sources and allow usb debugging
• run mochitests with a TEST_PATH=gfx (the gfx tests are what I always use as a sanity check since it is so fast)
• smile!

The challenge is out, can you beat my 5 minutes?