Option to allow failures in Fan-In/Out Workflow
complete
B
Brandon Page
The requires tag in workflows really limits what is possible. Requires makes a lot of sense for the deployment scenario used to demo it, but waiting for a group of jobs to complete (pass or fail) should be an option.
Scenario: Setup -> Run multiple sets of tests in parallel -> Combine test results/coverage results/artifacts and report to PR
That scenario above isn’t possible because if a single test fails in any of the jobs running in parallel the the fan-in step won’t run.
CCI-I-344
F
Fernando Abreu
We’re planning to extend our requires syntax to support requiring any job that’s in a terminal state, which will now also include skipped (i.e., jobs that will never run because a dependency wasn’t met).
This would allow queuing multiple jobs as part of the same logical group.
Example:
jobs:
- build
- test:
requires:
- build
- deploy:
requires:
- test
- release:
requires:
- deploy
- send_stats:
requires:
- release: terminal
In this setup, send_stats will run regardless of the outcome of release. Even if something fails upstream and release is skipped, send_stats will still execute as long as release reaches a terminal state.
Would this be helpful in your workflows?
Pellizzetti
Fernando Abreu We would love that! It would greatly simplify our workflows. Do you guys have an ETA for this?
F
Fernando Abreu
Pellizzetti: That's great to hear—thanks for the feedback! We'd love to learn more about your potential use case if you're open to sharing.
We don't have a specific ETA just yet, but we'll make sure to keep everyone updated both here and in our changelog: https://circleci.com/changelog/
F
Fernando Abreu
The changelog for this feature is available here: https://circleci.com/changelog/more-flexible-job-required-capabilities/
You can find the documentation here: https://circleci.com/docs/configuration-reference/#requires
t
tachang
Fernando Abreu Did the "completed" status make it in?
Can you model a job that runs if either/or job A or job B is a success?
deploy:
requires:
- a: success
- b: success
F
Fernando Abreu
tachang: It did not. We noticed some confusion about what 'completed' means.
We don’t have a way to model an either/or dependency natively yet
Nathan Fish
complete
This has been released. Documentation and change log entries coming shortly.
S
Sherif
any status on this feature? our org is checking this page daily. we need a job to run after parallelized tests are completed regardless of their outcome.
Nathan Fish
Sherif: We are working on rollout now. Should be released within the next few days, assuming no surprises or bugs found.
F
Fernando Abreu
Sherif: was this able to address the problems you were facing?
Vikram Krishnamurthy
Hey, we could also use a clean up step in our flow when dealing with cleaning up held deployment locks during "fan in". Or some kind of "finally" step that happens no matter what at the end
F
Fernando Abreu
Vikram Krishnamurthy: Were you able to try out the Flexible Requires capability?
seaona
hi there! What's the status on the Requires solution proposed by Nathan?
P
Peter Darton
Another usecase is CircleCI orb testing.
At present, it isn't possible to write a CircleCI CI build to fully built-and-test a CircleCI orb because it isn't possible to "check that a command or job will fail in <situation>" as part of the automated test.
A circleci orb build needs to test that the orb code functions correctly, and that means calling the code and checking that it works as expected, which requires you to check not only "does this work when it should" but also "does it fail when it should" too.
I want to be able to "run this (downstream) job regardless of whether or not the previous (required) job passed or not" so that my orb build can check "did that work? If so,
actually
fail the build for real because that job-under-test should've failed".The easier you can make that, the better.
We would also benefit from this "run this job even if the previous one failed" facility for test-result combining (which we want to do but can't do as-is), but I would've thought that the inability for CircleCI to do CI build-and-test of CircleCI orbs is a bigger issue.
The difference in these two usecases is how the workflow's status is calculated though.
In the case of test-combining, we'd still want the workflow to be declared an overall failure if a fanned-out test failed even if the test-result-combining job succeeded.
In the case of orb job testing, we'd need the workflow to be declared a success if the expected-to-fail job failed and the workflow to fail if the expected-to-fail job succeeded.
TL;DR: We want to be able to say "this job failing doesn't mean the workflow ends here" somehow ... but sometimes we need the workflow to declare a failure and sometimes a success.
Nathan Fish
Interested in everyones thoughts on the following workflow graph when using "Requires". Does it give you enough information to understand why a job ran or did not run?
Robert Peralta
Nathan Fish Wow that is perfect! This is definitely self explanatory imo and solves the case I'm trying to address.
F
Fil Maj
Nathan Fish looking pretty good! Does one get similar information if you hover over the '3 jobs not ran' branch? I assume those were not run because they require job1 to pass?
Nicholas Shaw
Nathan Fish the second image seems to show it requires the prior job to fail, is that correct?
I think the main use case is the automated tests example given, whether the job passes/fails, I don't care, I want a test report generated.
Instead of needing to do things like the following:
- run:
command: |
if [ "$TESTS_PASS" = true ]; then
exit 0
else
exit 1
fi
Nathan Fish
Fil Maj yes that's the intent.
Nathan Fish
Nicholas Shaw in this example that is scenario, requiring failure. If you didn't care about the job passing or failing you could certainly model that with requires with the "completed" keyword. See the example config below for more details on what that would look like.
E
Ezequiel Gonzalez Rial
Nathan Fish In the first screenshot it shows that if a task fails, the workflow will fail. Wonder if there is an option to make the workflow success?
This is something that has been discussed as CCI-I-646
Nathan Fish
Ezequiel Gonzalez Rial at this time you can't override the job status. We have discussed internally how we might meet that need. Most of the use cases we have discussed are directly tied to events reported to github checks. With requires you can create job dependencies regardless of the status by passing both success and failure.
Nathan Fish
in progress
F
Fil Maj
Nathan Fish yay!
Nathan Fish
We are set to start working on a solution for "more flexible requires" and are looking for some feedback from you all. The config syntax would look something like the following:
jobs:
- a
- b
- c:
requires:
- a # defaults to success
- b: completed # success or failed
- d:
requires:
- a: failed
- e:
requires:
- b: canceled
There is one caveat. Canceling workflows does not mean the canceled job logic is reached. Cancelling a workflow still cancels all remaining work in the workflow.
Thoughts or considerations we should take into account?
Augusto Xavier
Nathan Fish Sounds like what we are looking for!
F
Fil Maj
Nathan Fish looking good so far but for clarity, could you describe the various job states that could be specified as part of
requires
? I see success
, failed
, canceled
, completed
as options but it is unclear to me what canceled
actually means? Is that a job state or a workflow outcome?Nathan Fish
Fil Maj great question! Cancelled will be a new option where an "approval job" can be cancelled, not just approved. It will be possible to cancel an approval job via API or via the UI for approval jobs.
F
Fil Maj
Nathan Fish Thanks for clarifying, LGTM 🚀 :shipit:
Load More
→