Test splitting should account for test startup time
Ben Kraft
As far as I understand, CircleCI currently doesn't take into account test-file startup time when splitting tests, even when the split is at the file level (which, for JavaScript tests, tends to be the only way). This produces extremely poor splits for Jest tests where the test itself is fast but needs to import a lot of code.
This repo provides an example and detailed explanation of the issue: https://github.com/benjaminjkraft/circleci-timing
My suggestion would be that if the level at which the tests are split is above the test suite (e.g. we are splitting by file), the timings should look at the
testsuite.time
in the junit output, not just testcase.time
. (If we are splitting by test cases, there are a few ways to do things, all of which are probably better than nothing.)