Flaky tests should use the filter branches at the top of the insights page
I
Ivan
When setting the filter branches on the insights page this should extent to the flaky tests below.
Currently, all flaky tests are shown regardless of the filtered branch.
Showing only flaky tests from the filtered branched is not configurable currently.
This makes it hard to differentiate between real/persistent flaky tests vs failed tests introduced by a momentary mistake.
A flag to see it to all branches would be ok, but it can cause confusion when looking for flaky tests on a specific branch.
Iván Navarro 75
In practice, the testing step in CI is done to prevent errors from leaking to the master branch. As such, the test step is likely to fail in PRs because of regressions introduced by the developers during their work. Those regressions need to be fixed, before the PR is deemed fit for merge to master and deployment to production. Those errors are likely to happen, one can even say that it's good they happen in the PR, so that we can catch regressions before they hit production. But they should not be flagged as flakiness in the test, as they were a reflection of a regression that was fixed, and not of the stochastic behavior of the test. The stochastic behavior (aka flakiness) should only be tracked in master, when the user selects "<workflow_name> (master)" in the UI. If the user wants to see the Insights of a workflow on all branches, then we can include failures from all branches in the flakyness calculation. But if the user selects "<workflow_name> (master)" then they would expect to see failures only on master because this is what they selected in the UI.