Sequentially run jobs in a branch
planned
K
Kyle Tryon
In some instances it may be that a user makes multiple commits to a branch in a short amount of time, causing multiple Workflows to execute at the same time, possibly deploying in an incorrect order. "Auto Cancel Redundant Builds" may not be suitable for all situations in which multiple deployments are expected. Solution: Enable a way to ensure that commits to a new branch are queued for sequential execution.
CCI-I-840
I
Info
Show this "idea" to your CEO and tell them that people are leaving CircleCI because it fails to do what it claims - produce consistent builds.
Nathan Fish
How would folks feel about something like the following config syntax to support this? Within a job you can specify that the job should run sequentially based on a unique key that can be generated from pipeline parameters or values. This would prevent race conditions and ensure that the job is run to completion in the order in which it was received.
...
deploy:
concurrency:
key: <<parameters.projectname>>-<<git.branch>>-<<parameters.target>>
steps:
- checkout
...
validate:
concurrency:
key: <<parameters.projectname>>-<<git.branch>>-<<parameters.target>>
steps:
...
or
...
workflows:
version: 2
build-and-deploy:
jobs:
- build
- deploy
concurrency:
key: <<parameters.env>>-<<git.branch>>
- validate
concurrency:
key: <<parameters.env>>-<<git.branch>>
J
James Taylor
Nathan Fish That would be great
Victor Sudakov
Nathan Fish for our purposes it would be sufficient to have a marker in the config that this workflow should run sequentially, and the workflows are queued in the order of received Github hooks. If your suggestion can be used like that, I'm fine with it. Will this syntax be supported for jobs, or workflows, or both?
Also, can you comment in what order the jobs with identical keys will be queued?
Nathan Fish
Victor Sudakov yes this would be based on received github hooks. We were looking specifically at job specific queues, which could be scoped to the org, workflow, or other pipeline parameters. It would not be an entire workflow, it would be the individual jobs that are queued. We can look at entire workflows as well but the plan right now is for jobs.
B
Ben Limmer
Nathan Fish Looks good for MVP.
Would the default behavior be to not cancel in-progress jobs? In GitHub actions, the default is to cancel-in-progress unless set otherwise: https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idconcurrency . Having a different behavior here might be confusing to folks.
Also +1 to be able to wait for other workflows. Today, I'm using the queue orb (https://circleci.com/developer/orbs/orb/eddiewebb/queue#jobs-block_workflow) to run some jobs (test, build docker image) but wait to execute a deployment until other in-progress deployment workflows are complete.
Nathan Fish
Ben Limmer I'll take that into consideration with the team, I agree similar behavior would be beneficial. It would also be valuable to enable customers to define "non-cancellable" jobs, to ensure that something can't be canceled once started. While not related directly to concurrency as a capability I think some of the use cases for concurrency also benefit from not allowing jobs to be canceled or to be protected from being canceled.
Nathan Fish
planned
Nathan Fish
Merged in a post:
Queue parallel builds to deploy and test in sequence
R
Rob Grant
As a software engineer
I want to run end to end tests for each change
So I can quickly identify breaking changes
Example scenario
There are 5 repos to build, for services 1-5. Services 1, 2, 3 and 4 have a large number of inter-service tests; service 5 is standalone.
Upon building code for service 1, 2, 3 or 4 in Circle CI, the build enters a step that deploys the build to an integration test environment (ITE), and triggers the full suite of tests. If more than pull request for Service 1, 2, 3 or 4 triggers this step at a similar time, the deploy to ITE is FIFO queued pending a successful outcome for the currently running integration tests.
Upon building the code for service 5, the build and deploy waits for any other Service 5 builds to pass tests, and then it deploys and runs the tests.
If Service 1, 2, 3 or 4 fails tests, their queue is paused until the problem is fixed (a successful rerun of that branch's tests) and then the builds continue.
If Service 5 fails its tests, its queue is paused until the problem is fixed (a successful rerun of that branch's tests) and then any other Service 5 builds continue.
Notes
Ideally builds should be able to run independently to a point (so unit tests can run and Docker images can build) but then should enter a sequencing queue.
Builds with interdependent repos could be gathered into Build Groups for easy configuration of sequencing.
CCI-I-1056
Nathan Fish
Victor Sudakov Right now we are looking at a few different options. The first is a workflow triggering a new pipeline. So work can be sequenced. We are also considering the concept of a locking job. Which would force work to wait until the job is completed.
One of the challenges we have is there is no guarantee that we receive the triggering event from the VCS or webhook in the correct order. Github now has a merge queue so for customers using that there is some confidence that a locking job would address the issue and avoid a job completing before another in a pipeline that started earlier.
Victor Sudakov
Nathan Fish Thank you for replying. I've experimented with locks in Redis trying to prevent simultaneous jobs, and the correct sequence is indeed the main challenge.
B
Ben Limmer
Nathan Fish I like the idea of a locking job. It would be great if this worked just like it does in GitHub actions where I just say "only one of this type of workflow can be running at once". https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idconcurrency Their feature works for like 90%+ of the use cases I've come up with.
Victor Sudakov
Is there any news on implementing this? It is a very important feature, workflows should run in a predictable order, preferably with the order of commits.
Fredrik A. Madsen-Malmo
Are there any updates on this project?
D
Devin Riegle
Yes, this is a necessity for deploying applications through Circle. Please add!
Chris Morris
Yes please.
(This request was put "under review" in Sept 2020, what does that mean and is it still current? I'm guessing it's not?)
B
Ben Limmer
For reference, this is how they do it in GitHub Actions: https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idconcurrency
It's a pretty simple/elegant solution IMO
Load More
→