If you have ever received an automated mail about a performance regression, and then 10 more, you probably are frustrated by the volume of alerts. 6 months ago, I started looking at the alerts and filing bugs, and 10 weeks ago a little tool was written to help out.
What have I seen in 10 weeks:
1926 alerts on mozilla.dev.tree-management for Talos resulting in 58 bugs filed (or 1 bug/33 alerts):
*keep in mind that many alerts are improvements, as well as duplicated between trees and pgo/nonpgo
Now for some numbers as we uplift. How are we doing from one release to another? Are we regressing, Improving? These are all questions I would like to answer in the coming weeks.
Firefox 30 uplift, m-c -> Aurora:
Firefox 31 uplift, m-c -> Aurora (tracking bug 990085):
- 23 – regressions (5 CART, 3 TART, 2 SVG, 2 TP5, 2 TResize, and a few others)
- CART regressions are mostly fixed already!
- 10 – improvements (Kraken/V8, Linux64 SVG, OSX 10.8 TART)
Is this useful information?
Are there questions you would rather I answer with this data?
3.1415926535897932384626433... 
I think something that would be *interesting* (I did not say useful) is to have a tool that shows a global view of each 6 week period side-by-side.
Each release has a score for a day that is the aggregate of all talos scores for that day. Maybe we don’t need to do the aggregate of the suites in itself and show them separately. This would give 6*7 aggregated data points. This would allow us to see a release as it goes through the different release channels (m-c -> m-a and on).
Thanks for looking into this topic. I had always wondered.
Thanks for the feedback Armen! I will experiment with the data I provide and 6 weeks later we can see what data looks like. I am sure there is a way to visualize this as it moves from branch to branch.
One thing that changes the values is new tests, disabling tests, and adjusting tests (to produce different values). I am sure that can be accounted for 🙂
We could add a whimsy unicorn jumping of the chart on the days that a new test gets added/removed 🙂