reftests for windows mobile

In my never ending quest to conquer the unittests, I have made another dent by getting reftests running (in a very basic form) on windows mobile!

It turns out this was much easier than I thought to do, but getting my dozen or so changes to and took many hours of debugging and trial and error. The whole process started out just to get a test to run, and the journey ended up touching just about every line of code.

I started off commenting everything out and bringing each chunk of code one at a time. This was a process that would take a full day, but knowing there was so many calls to subprocess and os, I knew I was up against a tough problem.

So began the slashing of os.path.* calls and writing stubs for anything that references subprocess or its returned process handle. What I found was I had no good way to debug, so I set out to verify Fennec could run a reftest by hand with a static profile that I munged together and I got results! Now can I get it working from Not so lucky.

Problem #1 – trouble creating profile and crashing on rmdir:
Creating a profile. This seems simple and it appears to create a profile as expected in the code. For some reason the code kept crashing while trying to rmdir on the profile. Ugh, lets stick with a static profile and get these tests running. This was actually resolved when problem #2 was figured out.

Problem #2 – fennec restarts each time it is launched:
We call automation.runApp twice and for some reason it never terminates fennec after the first call so the second call fails. Ok, I can remove the first instance since it just registers the reftest.jar extension (remember we are using a static profile, so we can cheat). To make matters worse, this didn’t solve my problem. I found out that we were restarting Fennec every time we launched. This was a nasty problem that was easily resolved with adding a –environ:”NO_EM_RESTART=1″. Now I can have both the extension registration and the test.

Problem #3 – python script can’t open another python script process:
In order to run reftests on a small device we need to run them in smaller sets rather than all 3800+ at once. The big bottleneck is the memory required to store the tests and results until we are done. So I used my maemkit toolkit to call over and over again with a different .manifest file! No luck 😦 I cannot seem to get python to create a new process (using osce) when that process is another python instance. Ok, I can work around this and just call (main) from inside my maemkit. Problem solved!

Problem #4 – osce.blockingtee crashes python script on the third call in a row:
I noticed this while getting xpcshell up and running, and never had a solution. Unfortunately this doesn’t allow me to run the full test suite. The problem is blockingtee will write the output of the process to a given file. The alternative is to use osce.systema() instead. This works for the majority of the tests (a few appear to hang) and will be good enough to prove that I can get this done end to end.

So other than actually collecting the output, I have a working demo. To simplify my changes, I did this (note, the code is attached to bug 493792): –
remove arg dependencies:

#  if len(args) != 1:
#    print >>sys.stderr, "No reftest.list specified."
#    sys.exit(1)

hardcode cli arguments:

#  reftestlist = getFullPath(args[0])
  reftestlist = r'\tests\reftest\tests\layout\reftests\image\reftest.list' = r'\tests\fennec\fennec.exe'
  options.xrePath = r'\tests\fennec\xulrunner'

and use tempfile for mkdtemp as it would crash all the time:

#    profileDir = mkdtemp()
    profileDir = tempfile.mkdtemp()

Ok, that isn’t too bad. Now for, I needed to change a lot of things. Mostly what I did was change things based on platform:

platform = "wince"
if platform == "wince":
  import osce
  import subprocess

and fix some of the hardcoded variables that are useless from the wince build:

#for wince: this defaults to c:\
DIST_BIN = r'\tests\fennec'
DEFAULT_APP = DIST_BIN + r'\fennec.exe'
CERTS_SRC_DIR = r'\tests\certs'

There is a Process class which manages the subprocess.Popen, so I do this:

if platform == "wince":
 class Process():
  def kill(self):
 class Process(subprocess.Popen):

for the last set of changes, in runApp, I disable SSL Tunnel (will have to deal with this for mochitest):

  if platform == "wince":
    runSSLTunnel = False

and since we are not in cygwin, we need the real backslash:

   if platform == "wince":
      profileDirectory = profileDir + "\\"
      profileDirectory = profileDir + "/"

For launching the actual application, I have to use osce.systema instead of Popen:

  if platform == "wince":
    proc = osce.systema(cmd, args)"INFO | ( | Application pid: None")
    proc = Process([cmd] + args, env = environment(env), stdout = outputPipe, stderr = subprocess.STDOUT)"INFO | ( | Application pid: %d",

I force outputPipe = None which skips a lot of the process interaction code:

  if debuggerInfo and debuggerInfo["interactive"]:
    outputPipe = None
  elif platform == "wince":
    outputPipe = None
    outputPipe = subprocess.PIPE

and finally I force the status = 0 so we can avoid more tinkering with the process handle:

  if platform == "wince":
    status = 0
    status = proc.wait()

That is really all it takes. I think a better approach for the long run is to create a which gives us a Popen and spoofs the other commands (wait, PIPE, stdout, stderr). Also debugging the blockingtee crash after three calls and integrating that into


Leave a comment

Filed under testdev

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s