Tag Archives: notes

notes on a python webserver

Last week I created a python webserver as a patch for make talos-remote.  This ended up being frought with performance issues, so I have started looking into it.  I based it off of the profileserver.py that we have in mozilla-central, and while it worked I was finding my tp4 tests were timing out.

I come to find out we are using a synchronous webserver, so this is easy to fix with a ThreadingMixIn, just like the chromium perf.py script:

class MyThreadedWebServer(ThreadingMixIn, BaseHTTPServer.HTTPServer):

Now the test was finishing, but very very slowly (20+ minutes vs ❤ minutes).  After doing a CTRL+C on the webserver, I saw a lot of requests hanging on log_message and gethostbyaddr() calls.  So I ended up overloading the log_message call and things worked.

class MozRequestHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
    # I found on my local network that calls to this were timing out
    def address_string(self):
        return "a.b.c.d"

    # This produces a LOT of noise
    def log_message(self, format, *args):

Now tp4m runs as fast as using apache on my host machine.


Filed under testdev

Professional Development, Improv and your audience

I had the opportunity to attend some really exciting professional development sessions at the All Hands.  Personally I found these very interesting, but I heard a lot of grumbling about how these are not adding a lot of value or of interest.

One reason I found these interesting is that in a previous life I had attended a few years of Improv acting classes and did a short stint of real onstage Improv acting.  In looping back to these professional development sessions, they reminded me of the core concepts we learned in Improv 101.  So if you felt that you missed out, sign up for an Improv class.  Maybe if there are professional development sessions at a future event they could just have an Improv acting class.

Related to the professional development courses, I found that most of these were sparsely attended.  Of those that did attend the courses received great reviews/ratings.  To be fair, the technical tracks that I attended had about the same attendance records of the professional development tracks.  Maybe we are not creating sessions that are of interest to our audience?  I know for the technical tracks we just propose something and it magically becomes a session.  I don’t recall getting any input in what sessions would be available to me.  Maybe in the future we can do a better job of getting input from the community (a.k.a audience)!

1 Comment

Filed under general, reviews

I dream of green

After spending a month trying to get my first round of large changes into the test harness system, I learned a lot of little things along the way. First off, I will admit I am not a rock star developer who can whip out perfect code without testing it (although I did a hot patch to talos that worked) or look at code and see what is wrong at a glance. So knowing that, I have found that I can be successful by testing my code and following up on it to ensure it works and doesn’t cause problems.

First off, this patch literally rewrote 90% of the code in all of the test harnesses. Now most of that was just moving functions inside of classes, but you can imagine the chance of failure. I started writing this patch in October and after a few review cycles, it was ready to land (checkin for you non Mozillians.) A month later we finally get it in…why the delay?

Bitrot: This usually is not that big of a problem. All patches will experience people editing code in the same areas, but in this case with HG I was unable to fix the rejects by just editing the code that had conflicts. With my patch(es) they modified such large blocks of code that a 3 line patch caused me a whole day of headaches and I found it easier to back out those specific changes (hg revert -r xyz [filename]), apply my patch, and add those changes on top of it. Not the most ideal thing, but it works.

Reading the patch: I found that many times I would accidentally have a change in a patch that I didn’t realize was there. So I would submit it for review and get odd questions or it bounced back to me. I strongly encourage after you do you diff or qdiff to read over the patch like a reviewer would before submitting it. If it is an update to a patch, I diff it against the last version.

TryServer: This is a great tool that Mozilla has setup where you can submit a patch and it will do a build and testpass on all platforms. So while I might develop my code on Linux or Windows, I don’t have to build and run all the tests on my Mac as TryServer will do it for me. I didn’t realize this, but TryServer does builds and tests differently than Tinderbox (at least for mozilla-central). For example there are no debug builds (and tests), test packaging (run out of the objdir) or leaktests. So this isn’t perfect, but helps if you are not changing platform specific code.

Random Orange: This is a term at Mozilla to describe tests that are known to fail randomly causing the test run to turn orange instead of green. So while watching the Tinderbox Push Log, you start seeing failures in your tests and need to investigate them. It is easy to search in bugzilla for the test name that failed, but one case we didn’t find it in bugzilla and backed out the patch due to a debug leaktest (which was a known random failure) when in fact the patch would have been fine.

Tinderbox Logs: While investigating failures (either random or caused by my patch), we need to look at the logs generated by the build and tests to see what happened. There are brief and full logs available, but I found that the brief logs were not that useful in telling me how the build was done, or what command was executed. So loading up the full logs it is…all 30Mb+ of data into one Firefox window = crash. I found safari to be better at loading these large logs but most useful was doing a wget and then searching through the log (it is a .gz extension, but it is really .html) for ‘BuildStep ended’ to find the right area to focus on.

Although I never ran into these issues for the dozens of other patches that I have had land, it must have been pure luck. Now whenever I am ready for submitting a patch, I plan to:

  1. Build on one OS in Opt and Debug
  2. For both builds run leaktests.py, mochitest, mochitest-chrome, browser-chrome, mochitest-a11y, mochitest-ipcplugins, reftest, crashtest, jsreftest
  3. Submit to tryserver and get all green (builds, talos, unittests)
  4. Review patch for mistakes, comments, extra code
  5. After review verify for bitrot, if bitrot- repeat

Leave a comment

Filed under testdev

running httpd.js on a remote host

This exercise came about as an option for running unittests on windows mobile. We ran into an issue where the http server was running but not returning web pages. Vlad had mentioned running the web server on a host machine to be respectful of the memory on the device. In the past other members of the QA team have tried this with no luck. With the help of :waldo, I was able to get tests running while accessing a remote webserver.

For a quick background, all unittests rely on httpd.js running via xpcshell to serve web pages. For example, xpcshell update tests (test_0030_general.js) utilize httpd.js to download a .mar file and mochitest uses httpd.js for every test it runs. It is a little tricky to get this to run on a remote server as there are a few things to edit:

Now you need to fire up xpcshell to run as a server:

#set any env variables such as
#LD_LIBRARY_PATH=$firefox_dir/xulrunner on linux
cd $firefox_dir/xulrunner
./xpcshell -f ../../mochitest/httpd.js -f ../../mochitest/server.js

On the remote machine launch your test and be happy!

To really utilize this for the Fennec unittests on Windows Mobile, we will have to make many changes to the tests in addition to the ones mentioned above. Making it fit into automation would be very difficult as ip addresses can change and we might need to set it dynamically. Alternatively, running 1 host to many clients does have some attraction.


Filed under testdev

how we are running xpcshell unittests on windows mobile

Here is another scrap of information on how our progress is coming on getting unittests to run on Windows Mobile. A lot has changed on our approach and tools which has allowed us to make measurable progress towards seeing these run end to end. Here are some of the things we have done.

Last month I discussed launching a unittest in wince and taking that further blassey has compiled python25 with the windows mobile compilers and added the pipelib code to allow for stdout to a file. This has been a huge step forward and resolves a lot of our problems

I found out that runxpcshelltests.py had gone under a major overhaul and my previous work was not compatible. While working with the new code, I ran into a problem where defining a variable with the -e parameter to xpcshell was not working. To work around this I have changed:
xpcshell.exe -e ‘const _HEAD_FILES = [“/head.js”, “/head.js”];’

xpcshell.exe -e ‘var[_HEAD_FILES]=[[“/head.js”,”/head.js”]];’

This new method is ugly but works. We suspect this is related to how subprocess.Popen handles the commandline to execute (appears to require 2 args: app, argv). Regardless, with a series of additional hacks to runxpcshelltests.py, we can launch xpcshell.exe and get results in a log file.

Lastly, there is another issue we have where the lack of support for cwd on windows mobile is causing some of our test_update/unit/* tests to fail. There is a workaround in bug 458950 that we have where we can support cwd when it is passed on the command line:
xpcshell.exe –environ:CWD=path

Of course we need to have this code for xpcshell.exe (as noted in bug 503137), not just fennec.exe or xulrunner.exe.

With all the changes above, we can launch our python script (including cli args) via the visual studio debugger or a command line version on our device that is tethered via USB cable and activesync.

Unfortunately this is not complete yet. We have to clean up the python code and make it a patch. That is hinged on finding better fixes for the -e parameters. Also, while running on my HTC Touch Pro, the device hangs a lot for various reasons (requiring reseating the battery). Stabilizing this could require a tool change, as well as a different way to run the tests.

Leave a comment

Filed under testdev

reftests for windows mobile

In my never ending quest to conquer the unittests, I have made another dent by getting reftests running (in a very basic form) on windows mobile!

It turns out this was much easier than I thought to do, but getting my dozen or so changes to automation.py and runreftest.py took many hours of debugging and trial and error. The whole process started out just to get a test to run, and the journey ended up touching just about every line of code.

I started off commenting everything out and bringing each chunk of code one at a time. This was a process that would take a full day, but knowing there was so many calls to subprocess and os, I knew I was up against a tough problem.

So began the slashing of os.path.* calls and writing stubs for anything that references subprocess or its returned process handle. What I found was I had no good way to debug, so I set out to verify Fennec could run a reftest by hand with a static profile that I munged together and I got results! Now can I get it working from runreftests.py? Not so lucky.

Problem #1 – trouble creating profile and crashing on rmdir:
Creating a profile. This seems simple and it appears to create a profile as expected in the code. For some reason the code kept crashing while trying to rmdir on the profile. Ugh, lets stick with a static profile and get these tests running. This was actually resolved when problem #2 was figured out.

Problem #2 – fennec restarts each time it is launched:
We call automation.runApp twice and for some reason it never terminates fennec after the first call so the second call fails. Ok, I can remove the first instance since it just registers the reftest.jar extension (remember we are using a static profile, so we can cheat). To make matters worse, this didn’t solve my problem. I found out that we were restarting Fennec every time we launched. This was a nasty problem that was easily resolved with adding a –environ:”NO_EM_RESTART=1″. Now I can have both the extension registration and the test.

Problem #3 – python script can’t open another python script process:
In order to run reftests on a small device we need to run them in smaller sets rather than all 3800+ at once. The big bottleneck is the memory required to store the tests and results until we are done. So I used my maemkit toolkit to call runreftest.py over and over again with a different .manifest file! No luck 😦 I cannot seem to get python to create a new process (using osce) when that process is another python instance. Ok, I can work around this and just call runreftest.py (main) from inside my maemkit. Problem solved!

Problem #4 – osce.blockingtee crashes python script on the third call in a row:
I noticed this while getting xpcshell up and running, and never had a solution. Unfortunately this doesn’t allow me to run the full test suite. The problem is blockingtee will write the output of the process to a given file. The alternative is to use osce.systema() instead. This works for the majority of the tests (a few appear to hang) and will be good enough to prove that I can get this done end to end.

So other than actually collecting the output, I have a working demo. To simplify my changes, I did this (note, the code is attached to bug 493792):

runreftest.py –
remove arg dependencies:

#  if len(args) != 1:
#    print >>sys.stderr, "No reftest.list specified."
#    sys.exit(1)

hardcode cli arguments:

#  reftestlist = getFullPath(args[0])
  reftestlist = r'\tests\reftest\tests\layout\reftests\image\reftest.list'
  options.app = r'\tests\fennec\fennec.exe'
  options.xrePath = r'\tests\fennec\xulrunner'

and use tempfile for mkdtemp as it would crash all the time:

#    profileDir = mkdtemp()
    profileDir = tempfile.mkdtemp()

Ok, that isn’t too bad. Now for automation.py, I needed to change a lot of things. Mostly what I did was change things based on platform:

platform = "wince"
if platform == "wince":
  import osce
  import subprocess

and fix some of the hardcoded variables that are useless from the wince build:

#for wince: this defaults to c:\
DIST_BIN = r'\tests\fennec'
DEFAULT_APP = DIST_BIN + r'\fennec.exe'
CERTS_SRC_DIR = r'\tests\certs'

There is a Process class which manages the subprocess.Popen, so I do this:

if platform == "wince":
 class Process():
  def kill(self):
 class Process(subprocess.Popen):

for the last set of changes, in runApp, I disable SSL Tunnel (will have to deal with this for mochitest):

  if platform == "wince":
    runSSLTunnel = False

and since we are not in cygwin, we need the real backslash:

   if platform == "wince":
      profileDirectory = profileDir + "\\"
      profileDirectory = profileDir + "/"

For launching the actual application, I have to use osce.systema instead of Popen:

  if platform == "wince":
    proc = osce.systema(cmd, args)
    log.info("INFO | (automation.py) | Application pid: None")
    proc = Process([cmd] + args, env = environment(env), stdout = outputPipe, stderr = subprocess.STDOUT)
    log.info("INFO | (automation.py) | Application pid: %d", proc.pid)

I force outputPipe = None which skips a lot of the process interaction code:

  if debuggerInfo and debuggerInfo["interactive"]:
    outputPipe = None
  elif platform == "wince":
    outputPipe = None
    outputPipe = subprocess.PIPE

and finally I force the status = 0 so we can avoid more tinkering with the process handle:

  if platform == "wince":
    status = 0
    status = proc.wait()

That is really all it takes. I think a better approach for the long run is to create a subprocess.py which gives us a Popen and spoofs the other commands (wait, PIPE, stdout, stderr). Also debugging the blockingtee crash after three calls and integrating that into subprocess.py.

Leave a comment

Filed under testdev

automating fennec bookmark manager

This is more of a technical note and probably a boring read for anybody not interested in test development for fennec.

In an effort to add real automation for Fennec we have had issues making progress using browser-chrome to develop tests. Our latest bits of work come from bookmarks where happyhans is adding additional tests to what zad started (let me note that zad really paved the way for these tests on Fennec).

One thing we were trying to figure out is how to click on the edit button while managing bookmarks. It would be nice if we could just reuse test code from Firefox, but that is not an option. So using DOM Inspector and a lot of other help on IRC, browsing source code and just hacking, I have found the method to click a bookmark.


    var bookkmarkitems = chromeWindow.document.getElementById("bookmark-items");
    var bookmarkitem = window.document.getAnonymousElementByAttribute(bookmarkitems, "class", "bookmark-item");
    var editButton = window.document.getAnonymousElementByAttribute(bookmarkitem, "anonid", "edit-button");

A lot of this code :happyhans did. I found out the last two lines where we have a list of bookmarks and just need to find the “Edit” button to click.

What we see here is getting a list of bookmarkitems with getElementById limits our scope and returns a XULElement. Inside this Element, we have a list of “bookmark-item” classes which contain the bookmarks and the buttons to interact with them.

Since there is no ID on the bookmark-item, I query it with the getAnonymousElementByAttribute but here I start with my bookmark-items and look for a class named “bookmark-item”. This sound straightforward, but finding examples of other code to do this is not really possible.

Lastly, I took the bookmark-item (which defaults to the first item) and search for the “edit-button” as it has a “anonid”. I found out all of this information using the DOM Inspector (once mfinkle helped me use it properly).

So that is the basics. This code needs some real love to make it work with a specific title or URI instead of defaulting to the first item in the list. Knowing what I do now gives me some confidence in being able to figure out how to solve other problems in the bookmark manager.

Leave a comment

Filed under testdev