Description

rspec --bisect is a great tool for combatting test pollution (having one test pollute the test environment via database records, environment changes, etc) by having a systematic way of tracking down order-based failures. Instead of making this a reactive process that you run when your team is seeing these test pollution problems, we like to get ahead of the problem by actively running rspec --bisect on our test suite (via CirleCI) regularly, that way we have a quick path to finding a fix when it's introduced.

Problem

'rspec --bisect' is only useful if it's given a test ordering that has test pollution in it. This is often both order-based and/or random. To get value out of it, you need to be poking and prodding your groups of tests in different orders to weed out an order-based failure

TLDR ^ repeated runs of the same test orderings doesn't give much value if you're looking for test pollution

If I'm understanding the --split-by=timings option correctly, that would be great here in some respects since there is some element of randomness (assuming your test suite timings are constantly changing), but in our case, this isn't a great option when 'rspec --bisect' is involved since test runs can be _very_ variable...i.e. a test run with no failures will exit immediately, while a test run with failures might take > 5 hours

Solution

Introduce another split-by option that does not take timings into account, but rather does a balanced (by number of tests) random grouping

CCI-I-440