Tag Archives: windows mobile

patches checked in, tests can run on windows mobile!

My previous posts on the status of winmo automation outlined a series of patches to land. I am proud to say all of those have been reviewed (thanks to everybody), have landed (thanks ctalbert for checking these in) and with the help of this buildbot shim script are running very well!

So some highlights of what works:

  • Mochitest: runs great on the phone, and we use the –total-chunks and –this-chunk options so we don’t run out of memory. Right now I am testing it with –total-chunks=20, but suspect we can go a bit lower. As another note, the overhead to restart the phone, install the build, load the browser, and start the tests is 7.5 minutes on my HTC touch pro
  • Reftest/Crashtest/JSreftest: all of these run great. We need to run smaller manifest files though as after about 800 files on my device the tests come to a halt and execute maybe 1 test every few minutes. Luckily these run with manifest files so we can easily create a few manifests and have a working solution
  • Xpcshell: this is pretty straightforward. I don’t see any problems running this end to end as the harness by design only runs one case at a time. As a note, this is the only test harness that copies over the test files to the phone.
  • shim script: this turns on the webserver on your local IP so we can access it from the phone, as well as a bunch of other setup, monitoring and cleanup tasks. It would be nice to move the webserver functionality into the remote harness scripts in the near future so developers can easily run from a build tree
  • sutagent: this is actually the backbone of these tests. This tool runs on the phone and has come a long way over the last few months. This agent is a product of blassey and bmoss. The next steps here are to get the code checked into one of our source trees.

There are a few things we want to clean up, but overall we are at a great milestone on this project and ready to start rolling this out.

Leave a comment

Filed under testdev

updated status of the winmo automation project

A couple weeks ago, I posted an initial status of the winmo automation project. Here is an update of where we are.

Only a few patches have landed since the last update, but a lot of reviews have happened. Great progress has been made to resolve some of the unknown issues with running xpcshell and reftest remotely and everything is truly up for review. I expect next week at this time to have a much shorter list of patches.

1 Comment

Filed under Uncategorized

More info on the localhost->mochi.test change

This is just another public service announcement about the upcoming change to the unit tests. Last week I wrong about ‘mochi.test is the new localhost’, and this week I am reiterating that with more details.

After some feedback on the bug for this, we now have these options available:
* mochi.test
* 127.0.0.1
* moztest

With these options for referencing data in a test, you should be able to cover all the different networking scenarios that need testing. Keep in mind that mochi.test is the primary host and if you are posting a message to the parent window, the default domain is mochi.test, not localhost.

There is one other scenario where you need to get the real ip address of the server and this can be done with code like this:
var ios = Cc["@mozilla.org/network/io-service;1"].getService(Components.interfaces.nsIIOService);
var pps = Cc["@mozilla.org/network/protocol-proxy-service;1"].getService();
var uri = ios.newURI(“http://example.com”, null, null);
var pi = pps.resolve(uri, 0);
var host = “http://” + pi.host + “:” + pi.port;

From all the tests, test_prompt_async is the only test that needs this type of solution.

When we start running tests on windows mobile using the remote webserver, there won’t be any other changes needed other than what is mentioned above. Look for a demo from Clint at an upcoming Monday meeting.

Leave a comment

Filed under qa, testdev

status of the winmo automated tests project

I have been posting about this project for a while, so I figured I should give an update. Currently patches are landing and we are starting to get the final set of patches ready for review.

  • Talos: This was the first part of this project and we have checked in 3 of the 4 patches to get Talos TS running. There is 1 patch remaining which I need to upload for review
  • Mochitest: There are 4 patches required for this to work:
    1. Fix tests to not use hardcoded localhost – early review stages
    2. Add CLI options to mochitest for remote webserver – I need to cleanup my patch for review, at the end game
    3. Add devicemanager.py to the source tree – review started, waiting on sutagent.exe to resolve a few minor bugs
    4. Add runtestsremote.py to the source tree – review process started, waiting on other patches

    Good news is all 4 patches are at the review stage

  • Reftest: This requires 4 patches (1 is devicemanager.py from mochitest)
    1. Modify reftest.jar to support http url for manifest and test files – up for review
    2. Refactor runreftests.py – up for review
    3. Add remotereftests.py to source tree – needs work before review, but WIP posted

    Keep in mind here we are still blocked on registering the reftest extension. I also have instructions for how to setup and run this.

  • Xpcshell: this requires 3 patches (1 is device manager) and is still in WIP stages. There are two pieces to this that we still need to resolve: copying over the xpcshell data to the device and setting up a webserver to serve pages. Here are the two patches to date:
    1. Refactor runxpcshelltests.py to support subclass for winmo – WIP patch posted, close to review stage
    2. Add remotexpcshelltests.py to source tree – WIP patch posted

    I have written some instructions on how to run xpcshell tests on winmo if you are interested.

Stay tuned for updates when we start getting these patches landed and resolving some of our device selection/setup process.

4 Comments

Filed under testdev

first round of new test harness code has landed

Back in October, I started working on code to run the unittests on Windows CE + Mobile. This is an ongoing project, but I am starting to get the ball rolling in the right direction finally.

Today I checked in my first (actually a set of 2) patch (and it didn’t get backed out this time) which converts the bulk of the python test harness code to be OO instead of a raw scripts.

This is sort of a halfway point in the code that needs to get checked into mozilla-central in order for us to be testing automation on a windows mobile phone. Big thanks to Ted for reviewing all my patches and to Clint for helping me test and do the actual checkin.

NOTE: I originally wrote this Jan 7th, and it finally made in it today:)

1 Comment

Filed under testdev

Fennec proved more useful than Opera

Yesterday while driving to Mountain View from Oakland we were trying to find some dinner in the general Mountain View area. I pulled out my omnia II, launched Opera, and found a Thai restaurant in Palo Alto! Unfortunately with Opera I was unable to get a google map to load…FAIL.

I had a build of Fennec that I was using for mochitest development on the filesystem, so I launched it and tried to do the same search. Fully expecting a crash or two, I was surprised when I got search results in the same perceived amount of time as I did on Opera. The best thing was I could get a google map and zoom in/out to get the details I needed to figure out how to get to the restaurant.

End result: Fennec saved the day and proved itself as a useful browser. Time to clean it up and release Alpha4!

7 Comments

Filed under general, reviews

First look at the new remote testing

It has been almost 2 months since my last post and I have been heads down on bringing a new framework to life.

I discussed the 4 pieces involved in bringing automation to a new platform, and now we have what is shaping up to be a great approach for resolving the infrastructure and harness development.

What we have is a device specific agent which is written in a native language (C/C++) and does a small number of things (launch process, collect output, copy files to and from, query status such as processes, memory, cpu, disk, and lastly identify the device and os). This tool needs to act as a telnet server allowing us to telnet to it and execute a series of commands. Brad Lassey has developed such a tool for WinMo/CE which works partially on Win32 as well. I spent a couple days testing this interface and hammering out a python library to interact with a remote device. Now scripting file copy and process launching is easy. Clint has also used this tool to get mochitest and xpcshell running on Windows CE based devices!

That takes care of most of our infrastructure, now we need to get this stuff working on our test harnesses. My first task (only cleaned up one atm) is talos (see attached patch). This required massive changes to the Talos codebase, but no changes functionally for desktop based talos.

I found that in adding support to Talos for a remote device (as well as initial work for xpcshell and mochitest) that once we develop a DeviceAgent for a given platform there will be almost no additional work required to start running tests! I might be out of a job!!!

Next week I am going to work with Aki to get talos up and running (trial runs…don’t expect true miracles) on winmo and reporting to a staging graph server. Following that, we will start cleaning up our other harness code and getting mochitest, xpcshell, and reftests underway.

Stay tuned for progress updates and more details about a much needed updating to the way automation is run!

5 Comments

Filed under testdev