This title can imply a lot of things, yet this post will outline where the Fennec QA team is on developing tests, plans, and processes for our Fennec releases.
As many of you have seen, we released Fennec Maemo Beta2 and Windows Mobile Alpha2 last Friday. We also held the first ever Fennec testday logging a massive 43 bugs. This brings up some questions about why would we release after just finding 43 bugs, what is our release criteria, and what is the QA team doing.
Here is my take on the Firefox QA process (which the Mozilla community is accustomed to and looks to Fennec for a similar process):
- Build master test plan for release (alpha, beta, final)
- This includes a list of new features, update features and bugs fixed
- Define the target audience (hardware), performance goals, and list of must pass tests
- Outline points of intersection with other products, upgrades, l10n, support, marketing and schedule
- Start writing test cases and feature specific test plans
- Test private builds for feature and integration with product and ecosystem
- Test nightly builds
- As bugs trend towards zero (naturally or through triage) finalize test plan for features, dates, criteria
- Get release candidate build, start running full tests
- When no blockers and tests are done start running update, l10n, and documentation tests
- QA gives sign off, everybody is happy
For Fennec, we are not ready for that type of a cycle. Dev is cranking out serious changes to code. We are building from mozilla-central which is a roaming beast. Unittests are not running for all platforms. Everybody is asking where they can download Fennec for iPhone! A lot of chaos to apply quality measures to. Lets say (hypothetically) we want to check in a performance fix to gain 25% faster panning, but causes some issues in the url bar and focus. This might get done overnight (not planned) and there are only a few notes in a bug outlining this. Worse yet, this could happen after we are already testing a release candidate since it resolves so many side effects of bad performance and only introduces a few new problems.
Here is a quick outline of the current QA cycle for the Fennec releases to date:
- List of bugs is built up (~40) for the upcoming release
- a few weeks later, we are at 30 bugs (a lot of new issues)
- We triage down to top 5, build release candidate and start testing
- QA runs through litmus test suite, also some manual browsing
- If no big issues are found, lets ship tonight
I know this is a lot less formal than Firefox and rightly so. We are looking with our Alpha releases to get feedback from the motivated community members who are willing to tinker and hack and give us good feedback that we act upon. For the Beta releases, we need to be much more stable and change our audience to people who are not so patient but willing to accept a lot of bugs. This means little to no crashes, faster performance and good compatibility with the web.
Going forward, we will step up the QA involvement and evolve into a more formal QA process. Keep in mind we will be doing alpha, beta, and formal releases all at the same time just on different platforms (we can’t lose our current methods completely).
Here is what we have in place right now:
- Litmus test cases
- Firefox unittests running on Maemo
- Maemo talos runs (including fennecmark for panning and zoom performance)
- Initial presence in phone specific forums: htc touch pro and samsung omnia
- Account with Device Anywhere for compatibility testing
These are all key components to providing quality test coverage. If you look at putting some solid release criteria around the things we have in place for an official release we will look more like this:
- Create test plan with release criteria
- Test individual changes and new features
- Develop new test cases and flag cases required for release
- Get consensus from team on release criteria
- Request milestone build every month that we can make a quick pass on to see how close we are
- Utilize Testdays to keep the product stable at each milestone
- Bug count trends to zero, RC build is generated, test pass starts
- Tests pass, move onto final release prep (note: not just call it good)
- Test install on criteria hardware, l10n, and release note testing
This new approach falls in line with the Firefox QA methodology quite well. It really adds more overhead to the process, but give appropriate time for each area as well as a general consensus with the whole team of what to expect and how to make the decisions.
It is time to mature the Fennec release process and make it a real piece of software that is reliable and well respected in the mobile community.