As I mentioned earlier in this year, I have had the pleasure of working with Vaibhav. Now that time has passed he continues to contribute to Mozilla, and he will be participating this year in Google Summer of Code with Mozilla. I am excited.
Tag Archives: automation
On Friday we rolled out a big change to split up our browser-chrome tests. It started out as a great idea to split the devtools out into their own suite, then after testing, we ended up chunking the remaining browser chrome tests into 3 chunks.
No more 200 minute wait times, in fact we probably are running too many chunks. A lot of heavy lifting took place, a lot of it in releng from Armen and Ben, and much work from Gavin and RyanVM who pushed hard and proposed great ideas to see this through.
What is next?
There are a few more test cases to fix and to get all these changes on Aurora. We have more work we want to do (lower priority) on running the tests differently to help isolate issues where one test affects another test.
In the next few weeks I want to put together a list of projects and bugs that we can work on to make our tests more useful and reliable. Stay tuned!
The last 2 weeks I have gone head first into a world of resolving some issues with our mochitest browser-chrome tests with RyanVM, Armen, and the help of Gavin and many developers who are fixing problems left and right.
There are 3 projects I have been focusing on:
1) Moving our Linux debug browser chrome tests off our old fedora slaves in a datacenter and running them on ec2 slave instances, in bug 987892.
These are live and green on all Firefox 29, 30, and 31 trees! More work is needed for Firefox-28 and ESR-24 which should be wrapped up this week. Next week we can stop running all linux unittests on fedora slaves.
2) Splitting all the developer tools tests out of the browser-chrome suite into their own suite in bug 984930.
browser-chrome tests have been a thorn in the side of the sheriff team for many months. More and more the rapidly growing features and tests of developer tools have been causing the entire browser-chrome suite to fail, in cases of debug to run for hours. Splitting this out gives us a small shield of isolation. In fact, we have this running well on Cedar, we are pushing hard to have this rolled out to our production and development branches by the end of this week!
3) Splitting the remaining browser chrome tests into 3 chunks, in bug 819963.
Just like the developer tools, we have been running browser-chrome in 3 chunks on Cedar. With just 7 tests disabled, we are very green and consistently green.
While there are a lot of other changes going on under the hood, what will be seen by next week on your favorite branch of Firefox will be:
- ‘dt’ jobs for opt, and ‘dt1′, ‘dt2′, ‘dt3′ jobs for debug
- ‘bc’ job will turn into ‘bc1′, ‘bc2′, ‘bc3′
- much faster turnaround times on bc tests (62 minutes is the slowest right now, the rest are averaging ~20 minutes/job)
- less random orange cluttering up results
Of all the tests that are run on tbpl, mochitests are the last ones to receive manifests. As of this morning, we have landed all the changes that we can to have all our tests defined in mochitest.ini files and have removed the entries in b2g*.json, by putting entries in the appropriate mochitest.ini files.
Ahal, has done a good job of outlining what this means for b2g in his post. As mentioned there, this work was done by a dedicated community member :vaibhav1994 as he continues to write patches, investigate failures, and repeat until success.
For those interested in the next steps, we are looking forward to removing our build time filtering and start filtering tests at runtime. This work is being done by billm in bug 938019. Once that is landed we can start querying which tests are enabled/disabled per platform and track that over time!
As this is the short time window of Google Summer of Code applications, I have seen a lot of requests for mochitest related bugs to work on. Normally, we look for new bugs on the bugs ahoy! tool. Most of these have been picked through, so I spent some time going through a bunch of mochitest/automation related bugs. Many of the bugs I found were outdated, duplicates of other things, or didn’t apply to the tools today.
Here is my short list of bugs to get more familiar with automation while fixing bugs which solve real problems for us:
- bug 958897 – ssltunnel lives if mochitest killed
- Bug 841808 – mozfile.rmtree should handle windows directory in use better
Bug 892283 – consider using shutil.rmtree and/or distutils remove_tree for mozfile
- Bug 908945 – Fix automation.py’s exit code handling
- Bug 912243 – Mochitest shouldnt chdir in __init__
- Bug 939755 – With httpd.js we sometimes don’t get the most recent version of the file
I have added the appropriate tags to those bugs to make them good first bugs. Please take time to look over the bug and ask questions in the bug to get a full understanding of what needs to be done and how to test it.
I have had the pleasure to work with Vaibhav for the last 6 weeks as he has joined the Mozilla community as a contributor on the A-Team. As I have watched him grow in his skills and confidence, I thought it would be useful to introduce him and share a bit more about him.
What is your background?
I currently reside in Pilani, a town in Rajasthan, India. The thing that I like the most about where I live is the campus life. I am surrounded by some awesome and brilliant people, and everyday I learn something new and interesting.
I am a third year student pursuing Electronics and Instrumentation at BITS Pilani. I have always loved and been involved in coding and hacking stuff. My favourite subjects so far have been Computer Programming and Data Structures and Algorithms. These courses have made my basics strong and I find the classes very interesting.
I like to code and hack stuff, and I am an open source enthusiast. Also, I enjoy solving algorithmic problems in my free time. I like following new startups and the technology they are working on, running, playing table tennis, and I am a cricket follower.
How did you get involved with Mozilla?
I have been using Mozilla Firefox for many years now. I have recently started contributing to the Mozilla community when a friend of mine encouraged me to do so. I had no idea how the open source community worked and I had the notion that people generally do not have time to answer silly questions and doubts of newcomers. But guess what? I was totally wrong. The contributors in Mozilla are really very helpful and are ready to answer every trivial question that a newbie faces.
Where do you see yourself in 5 years?
I see myself as a software developer solving big problems, building great products and traveling different places!
What advice would you give someone?
Do what you believe in and have the courage to follow your heart and instincts. They somehow know what you truly want to become.
In January :vaibhav1994 popped into IRC and wanted to contribute to a bug that I was a mentor on. This was talos bug 931337, from that first bug Vaibhav has fixed many bugs including work on finalizing our work to support manifests in mochitest. He wrote a great little script to generate patches for bugs 971132 and 970725.
Say hi in IRC and keep an eye out for bugs related to automation where he is uploading patches and fixing many outstanding issues!
A year without blogging and I am back. I figured there was some cool stuff to share, here is one tidbit.
In the last year I have picked up looking at talos results and filing regression bugs for results. This has been useful. What currently happens is when results are submitted to g.m.o (graph server) we detect a regression and send out an email to the original patch author (if we can determine it) and post to mozilla.dev.tree-management. I have been using dev.tree-management as a starting point for my hunting regressions. When things are busy it can eat up a couple hours in a day. Luckily many developers are responsible in taking action when they receive the emails.
Given that at least half of the regressions are not acted upon by the original developer, it is important to read the newsgroup. One of the things which makes it frustrating is that for a single regression we can get multiple alerts (regular builds vs pgo builds and as the patch merges between branches/projects).
To make my life easier, I have taken all the alerts on dev.tree-management and put them in a database (local right now). The final goal is a webUI that lets me easily annotate these alerts similar to tbpl for random test failures. One thing I wanted to do was help identify duplicate alerts. Today in my attempt I had a clear picture of what the lifecycle of a regression looks like:
mysql> select date,branch,percent,keyrevision from alerts where test=’Paint’ and platform=’WINNT 6.2 x64′ order by date ASC;
| date | branch | percent | keyrevision |
| 2014-02-14 19:41:38 | Mozilla-Inbound-Non-PGO | 10.1% | c7802c9d6eec |
| 2014-02-15 01:03:54 | Fx-Team-Non-PGO | 9.53% | 7a3adc5aac28 |
| 2014-02-15 21:43:48 | Mozilla-Inbound | 10.6% | c7802c9d6eec |
| 2014-02-16 03:46:12 | Firefox-Non-PGO | 8.88% | 5d7caa093f4f |
| 2014-02-16 03:46:13 | B2g-Inbound-Non-PGO | 9.44% | 071885f79841 |
| 2014-02-16 14:22:38 | Fx-Team | 10.4% | 7a3adc5aac28 |
| 2014-02-17 04:42:57 | B2g-Inbound | 10.7% | 071885f79841 |
| 2014-02-18 11:43:54 | Firefox | 9.76% | eac89fb04bb9 |
8 rows in set (0.00 sec)
This is really cool to see how 1 change can generate alerts for 4 days.
Stay tuned for more information on this and other topics!