2 months ago, we added session restore tests to Talos, today I am pleased to announce 3 new tests:
- media_tests – only runs on linux64 and is our first test to measure audio processing. Much thanks to :jesup, and Suhas Nandaku from Cisco.
- tp5o_scroll - imagine if tscrollx and tp5 had a child- not only do we load the page, but we scroll the page. Big thanks go to :avih for tackling this project.
- glterrain – The first webgl benchmark to show up in Talos. Credit goes to :avih for driving this and delivering it. There are others coming, this was the easiest to get going.
Stay tuned in the coming weeks as we have 2 others tests in the works:
- mainthreadio detection
I have been using alert manager for a few months to help me track performance regressions. It is time to take it to the next level and increase productivity with it.
Yesterday I created a wiki page outlining the project. Today I filed a bug of bugs to outline my roadmap.
Basically we have:
* a bunch of little UI polish bugs
* some optimizations
* addition of reporting
* more work flow for easier management and investigations
In the near future we will work on making this work for high resolution alerts (i.e. each page that we load in talos collects numbers and we need to figure out how to track regressions on those instead of the highly summarized version of a collection).
Thanks for looking at this, improving tools always allows for higher quality work.
Last week I wrote a post with some thoughts on AutoLand and Try Server, this had some wonderful comments and because of that I have continued to think in the same problem space a bit more.
In chatting with Vaibhav1994 (who is really doing an awesome GSoC project this summer for Mozilla), we started brainstorming another way to resolve our intermittent orange problem.
What if we rerun the test case that caused the job to go orange (yes in a crash, leak, shutdown timeout we would rerun the entire job) and if it was green then we could deem the failure as intermittent and ignore it!
With some of the work being done in bug 1014125, we could achieve this outside of buildbot and the job could repeat itself inside the single job instance yielding a true green.
One thought- we might want to ensure that if it is a test failing that we run it 5 times and it only fails 1 time, otherwise it is too intermittent.
A second thought- we would do this by try by default for autoland, but still show the intermittents on integration branches.
I will eventually get to my thoughts on massive chunking, but for now, lets hear more pros and cons of this idea/topic.