Talos is the framework used for desktop Firefox to measure performance for every patch that gets checked in. Running tests for every checkin on every platform is great, but who looks at the results?
As I mentioned in a previous blog post, I have been looking at the alerts which are posted to dev.tree-management, and taking action on them if necessary. I will save discussing my alert manager tool for another day. One great thing about our alert system is that we send an email to the original patch author if we can determine who it is. What is great is many developers already take note of this and take actions on their own. I see many patches backed out or discussed with no one but the developer initiating the action.
So why do we need a Talos alert sheriff? For the main reason that not even half of the regressions are acted upon. There are valid reasons for this (wrong patch identified, noisy data, doesn’t seem related to the patch) and of course many regressions are ignored due to lack of time. When I started filing bugs 6 months ago, I incorrectly assumed all of them would be fixed or resolved as wontfix for a valid reason. This happens for most of the bugs, but many regressions get forgotten about.
When we did the uplift of Firefox 30 from mozilla-central to mozilla-aurora, we saw 26 regression alerts come in and 4 improvement alerts. This prompted us to revisit the process of what we were doing and what could be done better. Here are some of the new things we will be doing:
- For all regressions found, attempt to find the original bug and reopen/comment in the bug
- For some regressions that it is not easy to find the original bug, we will open a new bug
- All bugs that have regression information will be marked as blocking a new tracking bug
- For each release we will create a new tracking bug for all regressions
- After an uplift from central->aurora, we will ensure we have all alerts mapped to existing regressions
As this process goes through a cycle or two, we will refine it to ensure we have less noise for developers and more accuracy in tracking regressions faster