In my never ending quest to conquer the unittests, I have made another dent by getting reftests running (in a very basic form) on windows mobile!
It turns out this was much easier than I thought to do, but getting my dozen or so changes to automation.py and runreftest.py took many hours of debugging and trial and error. The whole process started out just to get a test to run, and the journey ended up touching just about every line of code.
I started off commenting everything out and bringing each chunk of code one at a time. This was a process that would take a full day, but knowing there was so many calls to subprocess and os, I knew I was up against a tough problem.
So began the slashing of os.path.* calls and writing stubs for anything that references subprocess or its returned process handle. What I found was I had no good way to debug, so I set out to verify Fennec could run a reftest by hand with a static profile that I munged together and I got results! Now can I get it working from runreftests.py? Not so lucky.
Problem #1 – trouble creating profile and crashing on rmdir:
Creating a profile. This seems simple and it appears to create a profile as expected in the code. For some reason the code kept crashing while trying to rmdir on the profile. Ugh, lets stick with a static profile and get these tests running. This was actually resolved when problem #2 was figured out.
Problem #2 – fennec restarts each time it is launched:
We call automation.runApp twice and for some reason it never terminates fennec after the first call so the second call fails. Ok, I can remove the first instance since it just registers the reftest.jar extension (remember we are using a static profile, so we can cheat). To make matters worse, this didn’t solve my problem. I found out that we were restarting Fennec every time we launched. This was a nasty problem that was easily resolved with adding a –environ:”NO_EM_RESTART=1″. Now I can have both the extension registration and the test.
Problem #3 – python script can’t open another python script process:
In order to run reftests on a small device we need to run them in smaller sets rather than all 3800+ at once. The big bottleneck is the memory required to store the tests and results until we are done. So I used my maemkit toolkit to call runreftest.py over and over again with a different .manifest file! No luck 😦 I cannot seem to get python to create a new process (using osce) when that process is another python instance. Ok, I can work around this and just call runreftest.py (main) from inside my maemkit. Problem solved!
Problem #4 – osce.blockingtee crashes python script on the third call in a row:
I noticed this while getting xpcshell up and running, and never had a solution. Unfortunately this doesn’t allow me to run the full test suite. The problem is blockingtee will write the output of the process to a given file. The alternative is to use osce.systema() instead. This works for the majority of the tests (a few appear to hang) and will be good enough to prove that I can get this done end to end.
So other than actually collecting the output, I have a working demo. To simplify my changes, I did this (note, the code is attached to bug 493792):
runreftest.py –
remove arg dependencies:
# if len(args) != 1:
# print >>sys.stderr, "No reftest.list specified."
# sys.exit(1)
hardcode cli arguments:
# reftestlist = getFullPath(args[0])
reftestlist = r'\tests\reftest\tests\layout\reftests\image\reftest.list'
options.app = r'\tests\fennec\fennec.exe'
options.xrePath = r'\tests\fennec\xulrunner'
and use tempfile for mkdtemp as it would crash all the time:
# profileDir = mkdtemp()
profileDir = tempfile.mkdtemp()
Ok, that isn’t too bad. Now for automation.py, I needed to change a lot of things. Mostly what I did was change things based on platform:
platform = "wince"
if platform == "wince":
import osce
else:
import subprocess
and fix some of the hardcoded variables that are useless from the wince build:
#for wince: this defaults to c:\
DIST_BIN = r'\tests\fennec'
DEFAULT_APP = DIST_BIN + r'\fennec.exe'
CERTS_SRC_DIR = r'\tests\certs'
There is a Process class which manages the subprocess.Popen, so I do this:
if platform == "wince":
class Process():
def kill(self):
pass
else:
class Process(subprocess.Popen):
...
for the last set of changes, in runApp, I disable SSL Tunnel (will have to deal with this for mochitest):
if platform == "wince":
runSSLTunnel = False
and since we are not in cygwin, we need the real backslash:
if platform == "wince":
profileDirectory = profileDir + "\\"
args.append(r'--environ:"NO_EM_RESTART=1"')
else:
profileDirectory = profileDir + "/"
For launching the actual application, I have to use osce.systema instead of Popen:
if platform == "wince":
proc = osce.systema(cmd, args)
log.info("INFO | (automation.py) | Application pid: None")
else:
proc = Process([cmd] + args, env = environment(env), stdout = outputPipe, stderr = subprocess.STDOUT)
log.info("INFO | (automation.py) | Application pid: %d", proc.pid)
I force outputPipe = None which skips a lot of the process interaction code:
if debuggerInfo and debuggerInfo["interactive"]:
outputPipe = None
elif platform == "wince":
outputPipe = None
else:
outputPipe = subprocess.PIPE
and finally I force the status = 0 so we can avoid more tinkering with the process handle:
if platform == "wince":
status = 0
else:
status = proc.wait()
That is really all it takes. I think a better approach for the long run is to create a subprocess.py which gives us a Popen and spoofs the other commands (wait, PIPE, stdout, stderr). Also debugging the blockingtee crash after three calls and integrating that into subprocess.py.
3.1415926535897932384626433... 