Last year we ventured down the path of adding test manifests for xpcshell in bug 616999. Finding a manifest format is not easy because there are plenty of objections to the format, syntax and relevance to the project at hand. At the end of the day, we depend too much on our build system to filter tests and after that we have hardcoded data in tests or harnesses to run or ignore based on certain criteria. So for xpcshell unittests, we have added a manifest so we can start to keep track of all these tests and not depend on iterating directories and sorting or reverse sorting head and tail files.
The first step is to get a manifest format for all existing tests. This was landed today in bug 616999 and is currently on mozilla-central. This requires that all test files in directories be in the manifest file and that the manifest file includes all files in the directory (verified at make time). Basically if you do a build, it will error out if you forget to add a manifest or test file to the manifest. Pretty straightforward.
The manifest we have chosen is the ini format from mozmill. We found that there is no silver bullet for a perfect test manifest, which is why we chose an existing format that met the needs of xpcshell. This is easy to hand edit (as opposed to json), is easy to parse from python and javascript. As compared to reftests which have a custom manifest format, we needed to just have a list of test files and more specifically a way to associate a head and tail script file (not easy with reftest manifests). The format might not work for everything, but it gives us a second format to work with depending on the problem we are solving.
3.1415926535897932384626433... 