As we get closer to having unittests running on windows mobile, I am starting to wonder how long it will take to complete a test run. Just like maemo with the n810’s we will be running the tests in parallel, but can we see full build + test + results in < 8 hours for a nightly?
For desktop builds, we have an enormous amount of building and testing that we do. This means we would be running tests for about 512 checkins/month (just less than 1/hour) and double that if we wanted to include the try server.
What I propose doing is running a small set of unittests on each checkin, such that we get simple coverage on each build and it runs fast enough so we have results in the same time window as our other test data comes in for desktop tests. Right now when you check into m-c or 1.9.2 tests are only run on desktop builds, this would run tests on windows mobile as well.
The big question is what tests to run. Here are a couple ideas:
- Run a small smoketest for each test suite (reftest, mochitest, xpcshell) which covers different areas, but not as in depth. Save the full test for the nightlies
- Iterate through a series of chunks (say 30 chunks for mochitest; desktop builds split it 5 ways) and for each checkin just run a single chunk. After the course of 2 days we will have done a full cycle of little chunks.
Personally I like the second option best, but really I am trying to think out of the box for ways to reduce regression windows while getting feedback on a per checkin basis. What do you think?