Tackling the large backlog of failed unittests for Fennec

Most of my posts are related to getting the unittests to run on Fennec, there is not much communication about how we are tracking and getting the tests to be green (zero failures). Simple explanation, up until now there was no plan.

Last December, I went through every failure and documented what I thought was the problem. I created a little web tool to see the differences and track my bugs. Of course this is a static tool and was a real pain to update with new bugs and tests.

Now it is August and many new failures are occuring and the old failures are not fixed. I am going to outline an approach to get us to ZERO failures by the end of the year. In order to be successful, we need to reduce the variables as much as possible. This means that we will run Fennec on desktop linux builds in tinderbox per checkin instead of on Maemo! This sets us up for getting green Tinderboxes in this environment vs a device (I suspect we will be 90%+ passing when run on a device).

Actions to take:

  1. Start with XPCShell tests first (then do Crashtest, Reftest, Mochitest, Chrome one at a time) and for each failure do the next steps
  2. Reproduce failure (twice)
  3. Reduce testcase (if possible)
  4. File bug/update existing bug, add bug # to a master tracking bug
  5. When done with a specific test harness (XPCShell in this case), meet with the devs to prioritize bugs and get everybody on the same page

This sounds simple but could take a long time. The benefit of tackling the smaller test harnesses first is that we can see progress (list of bugs, green) faster and start keeping those harnesses green.

What this does not do:

  1. Help us track new failures
  2. Get green tinderboxes on Maemo and WinMo
  3. Resolve remote web server related issues
  4. Fix issues when running tests one at a time

Stay tuned for an update when we get our first batch of bugs filed for XPCShell.

Advertisements

1 Comment

Filed under testdev

One response to “Tackling the large backlog of failed unittests for Fennec

  1. Sounds like a good place to start, since desktop Linux is a lot easier to debug on.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s