Category Archives: qa

Automation found its first regression for Fennec!

Yesterday I noticed our Mochitest-1 test was orange (since Nov 17th…still need to train others to watch the results), so I spent a few minutes looking into it and it turned out to be a regression (which was promptly fixed by the mobile guys!)  I know automation has uncovered errors in the past, but this is the first time we have had automation that was truly green for a week (we just turned it on) and then turned orange as the result of a unfriendly patch.

 

1 Comment

Filed under qa, testdev

More info on the localhost->mochi.test change

This is just another public service announcement about the upcoming change to the unit tests. Last week I wrong about ‘mochi.test is the new localhost’, and this week I am reiterating that with more details.

After some feedback on the bug for this, we now have these options available:
* mochi.test
* 127.0.0.1
* moztest

With these options for referencing data in a test, you should be able to cover all the different networking scenarios that need testing. Keep in mind that mochi.test is the primary host and if you are posting a message to the parent window, the default domain is mochi.test, not localhost.

There is one other scenario where you need to get the real ip address of the server and this can be done with code like this:
var ios = Cc["@mozilla.org/network/io-service;1"].getService(Components.interfaces.nsIIOService);
var pps = Cc["@mozilla.org/network/protocol-proxy-service;1"].getService();
var uri = ios.newURI(“http://example.com”, null, null);
var pi = pps.resolve(uri, 0);
var host = “http://” + pi.host + “:” + pi.port;

From all the tests, test_prompt_async is the only test that needs this type of solution.

When we start running tests on windows mobile using the remote webserver, there won’t be any other changes needed other than what is mentioned above. Look for a demo from Clint at an upcoming Monday meeting.

Leave a comment

Filed under qa, testdev

how we are tracking unittest failures for fennec

Currently Fennec has thousands of failures when running the full set of unittests. As it stands when tinderbox runs these, we just set the “pass” criteria as total failures <= acceptable # of failures. As you can imagine, this has a lot of room for improvement.

Enter the LogCompare tool which happyhans and I have been working on with help from mikeal for the couchdb backend. What we do is take the tinderbox log file, parse it and upload it to a database! This way we get a list of all the tests that were run and if they passed or failed. Now we can compare test by test what is fixed, a known failure or a new failure. What is even better is that we are running Mochitests in parallel on 4 different machines and LogCompare can tell us if the tests on machine1 pass or fail without necessarily waiting for the other tests to complete. Another bonus is we can track a specific test over time to look for random orange data.

The concept is simple, here are some of the details and caveats:

  • We track tests by test filename, not by directory or test suite
  • A single filename can have many tests (mochitest), so there is no clean way to track each specific test.
  • If a test fails, future tests (sometimes in the same file, folder, or suite) are skipped.
  • Parsing the log file is a nasty task with many corner cases
  • To match test names up correctly, we need to strip out full paths and just view the relative path/filename.
  • Need to handle when new tests are added or existing ones removed
  • Need to baseline from Firefox for full list of tests and counts

The goal here is to keep it simple while bringing the total failure count of the unittests on Fennec to Zero!

2 Comments

Filed under qa, testdev

Where is the Fennec QA team?

This title can imply a lot of things, yet this post will outline where the Fennec QA team is on developing tests, plans, and processes for our Fennec releases.

As many of you have seen, we released Fennec Maemo Beta2 and Windows Mobile Alpha2 last Friday. We also held the first ever Fennec testday logging a massive 43 bugs. This brings up some questions about why would we release after just finding 43 bugs, what is our release criteria, and what is the QA team doing.

Here is my take on the Firefox QA process (which the Mozilla community is accustomed to and looks to Fennec for a similar process):

  • Build master test plan for release (alpha, beta, final)
    • This includes a list of new features, update features and bugs fixed
    • Define the target audience (hardware), performance goals, and list of must pass tests
    • Outline points of intersection with other products, upgrades, l10n, support, marketing and schedule
  • Start writing test cases and feature specific test plans
  • Test private builds for feature and integration with product and ecosystem
  • Test nightly builds
  • As bugs trend towards zero (naturally or through triage) finalize test plan for features, dates, criteria
  • Get release candidate build, start running full tests
  • When no blockers and tests are done start running update, l10n, and documentation tests
  • QA gives sign off, everybody is happy

For Fennec, we are not ready for that type of a cycle. Dev is cranking out serious changes to code. We are building from mozilla-central which is a roaming beast. Unittests are not running for all platforms. Everybody is asking where they can download Fennec for iPhone! A lot of chaos to apply quality measures to. Lets say (hypothetically) we want to check in a performance fix to gain 25% faster panning, but causes some issues in the url bar and focus. This might get done overnight (not planned) and there are only a few notes in a bug outlining this. Worse yet, this could happen after we are already testing a release candidate since it resolves so many side effects of bad performance and only introduces a few new problems.

Here is a quick outline of the current QA cycle for the Fennec releases to date:

  • List of bugs is built up (~40) for the upcoming release
  • a few weeks later, we are at 30 bugs (a lot of new issues)
  • We triage down to top 5, build release candidate and start testing
  • QA runs through litmus test suite, also some manual browsing
  • If no big issues are found, lets ship tonight

I know this is a lot less formal than Firefox and rightly so. We are looking with our Alpha releases to get feedback from the motivated community members who are willing to tinker and hack and give us good feedback that we act upon. For the Beta releases, we need to be much more stable and change our audience to people who are not so patient but willing to accept a lot of bugs. This means little to no crashes, faster performance and good compatibility with the web.

Going forward, we will step up the QA involvement and evolve into a more formal QA process. Keep in mind we will be doing alpha, beta, and formal releases all at the same time just on different platforms (we can’t lose our current methods completely).

Here is what we have in place right now:

These are all key components to providing quality test coverage. If you look at putting some solid release criteria around the things we have in place for an official release we will look more like this:

  • Create test plan with release criteria
  • Test individual changes and new features
  • Develop new test cases and flag cases required for release
  • Get consensus from team on release criteria
  • Request milestone build every month that we can make a quick pass on to see how close we are
  • Utilize Testdays to keep the product stable at each milestone
  • Bug count trends to zero, RC build is generated, test pass starts
  • Tests pass, move onto final release prep (note: not just call it good)
  • Test install on criteria hardware, l10n, and release note testing

This new approach falls in line with the Firefox QA methodology quite well. It really adds more overhead to the process, but give appropriate time for each area as well as a general consensus with the whole team of what to expect and how to make the decisions.

It is time to mature the Fennec release process and make it a real piece of software that is reliable and well respected in the mobile community.

2 Comments

Filed under qa