Have you ever been working on a change that you think will affect performance numbers and you were not sure how to verify the impact of your change?
I have had questions on how to do this and recently I needed to do it myself (as I introduced a change to Talos which caused a big old performance regression in everything).
The main use case I needed to do was run a change on Try server and verify that it did in fact fix my performance regression. Normally I would go to tbpl, and click on each of my tests to see the reported number(s). For each of those test:number sets, I would look on graph server (hint: you can get to graph server for a given test by clicking on the reported number) and verify that my numbers were inside the expected range for that test/platform/branch based on the history. If only I was part of a software developers union I could complain that that boring time intensive work was not in my contract.
To simplify my life, I decided to automate this with a python script. I wrote compare.py which will spit out a text based summary of what I described above. Here is a sample output:
python compare.py --revision c094aeea5f73 --branch Try --masterbranch Firefox --test tp5n --platform Linux Linux: tp5n: 292.157 -> 400.444; 308.596
A quick explanation:
- 292.157 is the lowest number reported in the last 7 days for tp5n,linux
- 400.444 is the highest number reported in the last 7 days for tp5n,linux
- 308.596 is the value reported from my test on try server for tp5n,linux
While this doesn’t do the previous 30 changesets and the next 5, it gives a pretty good indicator about what to expect. I can run this on a different time range (to check the 7 days prior to my introduced regression) by adding –skipdays to the command line:
python compare.py --revision c094aeea5f73 --branch Try --masterbranch Firefox --test tp5n --platform Linux --skipdays 6 Linux: **tp5n: 311.975 -> 398.571; 308.596
Here you will see a “**tp5n”, and that indicates that the Try server number is not in the range and should be looked at the old fashioned way.
Hope this helps in debugging.
