joel’s rage – dom tests on windows mobile

Previously I had mentioned getting windows mobile mochitests up and running. Since then they have been running mostly full time. I calculated out that it would take about 50 hours to run the full test suite assuming there were no errors that caused the browser to not exit automatically. Unfortunately I ran into a block of 234 tests files which over 175 of them fail to execute (no log generated and browser doesn’t terminate within 5 minutes).

These are all dom-level1-core/test_hc_* tests. In trying to debug the problem (by editing a testcase), I found that I could pinpoint an issue, but then it wouldn’t pass twice in a row. Further tinkering showed that about 1 out of 4 times I ran a test (with/without changing it) it would pass. Looking at my log files that I generated during my initial automation pass on the dom-level1-core directory I see the same statistics (1 out of 4 passing) for the fully automated run. Time for a debug build to figure out what is going on.

With a debug build installed, every time I ran a test manually it would pass. The same test that would not pass two times in a row passed 6 times in a row. So much for a debug build helping me out. Next I thought that this might be related to running on a remote web server and one file at a time. I ran a series of tests on desktop Linux Fennec and they all ran end to end without hanging.

This is where I get stuck. My infrastructure is not the problem as the tests obviously run. The problem is they only run *reliably* on debug builds. This all points to some kind of timing issue with WinMo Fennec. Any tips for how to figure out what the problem is?

Debugging and python don’t mix

Earlier this week I spent some time getting xpcshell to run on windows mobile and was excited to use my debugging tips from my last post. After dozens of attempts, I found that running python to launch processes (using osce.py) would not work when launched from Visual Studio debugger. So my debugging is limited to python output and sleep statements so I have time to read it.

Luckily launching other processes like fennec.exe or xpcshell.exe work great from the Visual Studio environment and I was able to get some progress on the reftests.