Performance Alerts – by the numbers

If you have ever received an automated mail about a performance regression, and then 10 more, you probably are frustrated by the volume of alerts.  6 months ago, I started looking at the alerts and filing bugs, and 10 weeks ago a little tool was written to help out.

What have I seen in 10 weeks:

1926 alerts on mozilla.dev.tree-management for Talos resulting in 58 bugs filed (or 1 bug/33 alerts):

Image

*keep in mind that many alerts are improvements, as well as duplicated between trees and pgo/nonpgo

 

Now for some numbers as we uplift.  How are we doing from one release to another?  Are we regressing, Improving?  These are all questions I would like to answer in the coming weeks.

Firefox 30 uplift, m-c -> Aurora:

  • 26 – regressions (4 TART, 4 SVG, 3 TS, Paint, and many more)
    • 2 remaining bugs not resolved as we are now on Beta (bug 990183, bug 990194)

 

Firefox 31 uplift, m-c -> Aurora (tracking bug 990085):

 

Is this useful information?

Are there questions you would rather I answer with this data?

 

About these ads

3 Comments

Filed under Uncategorized

3 responses to “Performance Alerts – by the numbers

  1. I think something that would be *interesting* (I did not say useful) is to have a tool that shows a global view of each 6 week period side-by-side.
    Each release has a score for a day that is the aggregate of all talos scores for that day. Maybe we don’t need to do the aggregate of the suites in itself and show them separately. This would give 6*7 aggregated data points. This would allow us to see a release as it goes through the different release channels (m-c -> m-a and on).

    Thanks for looking into this topic. I had always wondered.

  2. elvis314

    Thanks for the feedback Armen! I will experiment with the data I provide and 6 weeks later we can see what data looks like. I am sure there is a way to visualize this as it moves from branch to branch.

    One thing that changes the values is new tests, disabling tests, and adjusting tests (to produce different values). I am sure that can be accounted for :)

  3. We could add a whimsy unicorn jumping of the chart on the days that a new test gets added/removed :)

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s