Auto skip flaky tests
A
Aaron
We've found the "rerun from failed" feature [0] to be super useful, and it's made me wonder if y'all have the ability to use the same behind the scenes trickery to let us automatically skip tests that have been flagged as flaky in CircleCI.
Here's how I'm imagining that might work:
- In the UI for advanced project settings, have a switch to opt in to have CircleCI skip flaky tests that are run with circleci tests run.
- Once the new feature is enabled, there’s a new “Flaky Tests” section of the project settings that allows you to set the threshold for how many times a test needs to be flagged as flaky for it to be auto-skipped, and see a list of tests that are currently being skipped (each with a button to re-enable it).
- Once a test crosses the flakiness threshold, automatically add it to the list of tests to skip and send an email/alert to everyone following that project that explains what just happened.
- Ignore the tests as long as they’re on the list of flaky tests to skip.
- When somebody wants to re-enable the test, they can do so using the “Flaky Tests” project settings page, or by using the CircleCI API.