Firefox 32 leaves the train station- what does the performance look like

Now that we have an uplift completed and enough future data has been collected to ensure sustained changes in data automatically, it is time for the triple fortnightly report of what performance looks like.  For reference there is some data in a blog post about general talos numbers.

Firefox 32 uplift, m-c -> Aurora (tracking bug 1004427):

• 20 – regressions (3 CART, 3 TART, 3 SVG, 3 TResize, and some one off tests)
  • 18 regressions are on windows, the majority a result of OMTC being turned on (this is more of a rebaselining of tests than actual regressions)
  • 3 bugs are tracking all 20 regressions!
• 43 – improvements (15 Kraken/V8/Dromaeo, 2 SVG, 7 TScroll, 4 TART/CART, 4 Paint, 9 SessionRestore, and a couple others)
• the Improvements are distributed amongst Windows, Mac, Linux

Firefox 31 uplift, m-c -> Aurora (tracking bug 990085):

• 23 – regressions (5 CART, 3 TART, 2 SVG, 2 TP5, 2 TResize, and a few others)
  • CART regressions are mostly fixed already!
• 10 – improvements (Kraken/V8, Linux64 SVG, OSX 10.8 TART)

Firefox 30 uplift, m-c -> Aurora:

• 26 – regressions (4 TART, 4 SVG, 3 TS, Paint, and many more)
  • 2 remaining bugs not resolved as we are now on Beta (bug 990183, bug 990194)

• 6 – improvements (CART, Windows 8 TPaint)

As you can see Firefox32 has a lot of improvements and fewer regressions (of those 20 about half are related to rebasing numbers).

Lets look at bugs:

• 36 bugs filed to date for Firefox32 Talos regressions
• 16 are resolved (7 as wontfix)
• 20 are open (this means that 17 of them are only showing up on non-pgo)

 

After reviewing the process of investigating alerts, it makes sense that we continue forward with the same process in 6 week intervals and any changes are made on uplift day and they would apply only to trunk.  Some future changes we are considering:

• not filing bugs on minimal regressions (ex. <4%)
• not filing bugs on non-pgo only regressions (since we only build pgo on Aurora, Beta, Release)
• generating alerts for per test (not per suite) regressions (and only file bugs if a single test is >10%)
• adjust the graph server alert calculation to not drop the page with the highest value and to report the geometric mean of the pages instead of the average
• any other great ideas you have on how to be efficient with our time while continuing to identify and document our regressions

Onward to Firefox 33!