Are there any trends in our Talos regression bugs?

Now that we have a better process for taking action on Talos alerts and pushing them to resolution, it is time to take a step back and see if any trends show up in our bugs.

First I want to look at bugs filed/week:

Image

This is fun to see, now what if we stack this up side by side with the alerts we receive:

Image

We started tracking alerts halfway through this process.  We show that for about 1 out of every 25 alerts we file a bug.  I had previously stated it was closer to 1/33 alerts (it appears that is averaging out the first few weeks).

Lets see where these bugs are filed, here is a view of the different bugzilla products:

Image

The Testing product is used to file bugs that we cannot figure out the exact changeset, so they get filed in testing::talos.  As there are almost 30 unique components bugs are filed in, I took a few minutes to look at the Core product, here is where the bugs live in Core:

Image

Pardon my bad graphing attempt here with the components cut off.  Graphics is the clear winner for regressions (with “graphics: layers” being a large part of it).  Of course the Javascript Engine and DOM would be there (a lot of our tests are sensitive to changes here).  This really shows where our test coverage is more than where bad code lives. 

Now that I know where the bugs are, here is a view of how long the bugs stay open:

Image

The fantastic news is most of our bugs are resolved in <=15 days!  I think this is a metric we can track and get better at- ideally closing all Talos regression bugs in <30 days.

Looking over all the bugs we have, what is the status of them?

Image

Yay for the blue pacman!  We have a lot of new bugs instead of assigned bugs, that might be something we could adjust and assign owners once it is confirmed and briefly discussed- that is still up in the air.

The burning question is what are all the bugs resolved as?

Image

To me this seems healthy, it is a starting point.  Tracking this over time will probably be a useful metric!

 

In summary, many developers have done great work to make improvements and fix patches over the last 6 months that we have been tracking this information.  There are things we can do better, I want to know-

What information provided today is useful to track regularly?

Is there something you would rather see?

 

About these ads

2 Comments

Filed under Uncategorized

2 responses to “Are there any trends in our Talos regression bugs?

  1. Vladan

    Great post, I’d love to see a writeup like this for every release :)

    In particular, I personally would like to see the figures for “# of days bugs stay open” and “all bugs resolution status” in future posts, and maybe a comment on the trend in each. “Regressions by component” as well as “fixed regressions by component” for the last release seem like they could be informative as well.

    I would also like to see info on the change in test scores during the last release, e.g. “ts_paint improved 4% in Firefox 32 with 3 commits improving the score (with bugs linked) and 2 commits regressing it during this cycle”. You’d have to report the noise levels as well. I’m not sure how much work this report would require.

    • elvis314

      Thanks Vladan! I am happy to hear of the interest in this. Right now I don’t track improvements to the specific changeset, although we can do simple queries to give us a general idea. What compounds this is backouts (usually for test failures). A regression or improvement lands, then is backed out, then lands again. That will reduce the quality of a query on alerts.

      Let me plan on creating a similar blog post after the next uplift with some useful data. We can work on making that data more dynamic and easier to access in the month of June!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s