considering new approach to mochitest on mobile

Windows mobile is rocking the boat for us and we are getting more and more creative. My last challenge is getting the mochitests up and running and this has proved very difficult.

Last week I setup the xpcshell httpd.js web server remotely and resolved my pending xpchell test. This week I moved onto mochitest. I did a sanity check by running a desktop version of fennec mochitest run with the remote httpd.js web server and it worked (with a lot of new failures/timeouts due to proxy and other issues I didn’t investigate).

Since my experience with xpcshell and reftest on Windows Mobile has resulted in more crashes and smaller chunks, I am considering running each test file in the mochitest harness by itself. This requires some retrofitting of the mochikit integration as there is no logging available for a single test file. It will also require more time to run as the overhead of starting the browser is expensive.

Any thoughts on this? Should I just find a more powerful phone to run these tests on? Is this something that we could find other uses for within Mozilla?

Advertisements

5 Comments

Filed under testdev

5 responses to “considering new approach to mochitest on mobile

  1. I’m not surprised?
    I’d like some more info on *how much* longer it takes. We can also try 2 files at a time, 3, etc. til we find the limit. But I think running + results over a long time trumps crashing quickly every time.

  2. elvis314

    How much longer depends on:
    * do we reboot between runs
    * can we get the startup time <10 seconds (currently ~30)

    There will also be increased time in accessing tests on a remote server. The logistics of doing 2/3/4/… files at a time really points towards a manifest system. splitting directories up in to that many small chunks could be very time consuming as well. I would rather do a small number of tests/run if possible.

  3. cmtalbert

    Well, I for one, would love to have the ability to run each file on mochitest individually and do it for an entire firefox build tree a couple of times over.

    I’d hope that by doing that I might find out what files are depending on state of other files, so yes, I think it would be useful from a “helping make the tests more reliable” standpoint.

    I’d raise a shaky hand and say I’m for it, but I’m a little leery of it because that is going to take a LONG LONG time to finish and (it seems) quite a bit of work.

    But like Aki said, long awaited results without crashing are worth more than quick repetitive crashes.

    And just because it ought to be said, these are miserable choices that you are having to make. 😦

  4. It’s crashing due to running out of memory, right? Have you tried hacking Mochitest to send a memory-pressure event after every test?

  5. elvis314

    Jesse, it is unclear. I have seen it crash many times without running out of memory. It seems like the majority of the issues are related to memory. Another issue with it all is we have found that a reboot is necessary very frequently on these devices.

    It looks like windows ce sends a wm_hibernate event when hitting memory pressure: http://msdn.microsoft.com/en-us/library/microsoft.windowsce.forms.mobiledevice.hibernate.aspx. We handle that in our code. I wonder if there is something I can do to manage it better.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s