Sequentially run jobs in a branch
under review
K
Kyle Tryon
In some instances it may be that a user makes multiple commits to a branch in a short amount of time, causing multiple Workflows to execute at the same time, possibly deploying in an incorrect order. "Auto Cancel Redundant Builds" may not be suitable for all situations in which multiple deployments are expected. Solution: Enable a way to ensure that commits to a new branch are queued for sequential execution.
CCI-I-840
Nathan Fish
Merged in a post:
Queue parallel builds to deploy and test in sequence
R
Rob Grant
As a software engineer
I want to run end to end tests for each change
So I can quickly identify breaking changes
Example scenario
There are 5 repos to build, for services 1-5. Services 1, 2, 3 and 4 have a large number of inter-service tests; service 5 is standalone.
Upon building code for service 1, 2, 3 or 4 in Circle CI, the build enters a step that deploys the build to an integration test environment (ITE), and triggers the full suite of tests. If more than pull request for Service 1, 2, 3 or 4 triggers this step at a similar time, the deploy to ITE is FIFO queued pending a successful outcome for the currently running integration tests.
Upon building the code for service 5, the build and deploy waits for any other Service 5 builds to pass tests, and then it deploys and runs the tests.
If Service 1, 2, 3 or 4 fails tests, their queue is paused until the problem is fixed (a successful rerun of that branch's tests) and then the builds continue.
If Service 5 fails its tests, its queue is paused until the problem is fixed (a successful rerun of that branch's tests) and then any other Service 5 builds continue.
Notes
Ideally builds should be able to run independently to a point (so unit tests can run and Docker images can build) but then should enter a sequencing queue.
Builds with interdependent repos could be gathered into Build Groups for easy configuration of sequencing.
CCI-I-1056
Nathan Fish
Victor Sudakov Right now we are looking at a few different options. The first is a workflow triggering a new pipeline. So work can be sequenced. We are also considering the concept of a locking job. Which would force work to wait until the job is completed.
One of the challenges we have is there is no guarantee that we receive the triggering event from the VCS or webhook in the correct order. Github now has a merge queue so for customers using that there is some confidence that a locking job would address the issue and avoid a job completing before another in a pipeline that started earlier.
Victor Sudakov
Nathan Fish Thank you for replying. I've experimented with locks in Redis trying to prevent simultaneous jobs, and the correct sequence is indeed the main challenge.
B
Ben Limmer
Nathan Fish I like the idea of a locking job. It would be great if this worked just like it does in GitHub actions where I just say "only one of this type of workflow can be running at once". https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idconcurrency Their feature works for like 90%+ of the use cases I've come up with.
Victor Sudakov
Is there any news on implementing this? It is a very important feature, workflows should run in a predictable order, preferably with the order of commits.
Fredrik A. Madsen-Malmo
Are there any updates on this project?
D
Devin Riegle
Yes, this is a necessity for deploying applications through Circle. Please add!
Chris Morris
Yes please.
(This request was put "under review" in Sept 2020, what does that mean and is it still current? I'm guessing it's not?)
B
Ben Limmer
For reference, this is how they do it in GitHub Actions: https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idconcurrency
It's a pretty simple/elegant solution IMO
Jordan
We need this as well. We've seen out-of-order deploys end up taking our production state backward to an older commit, when two close-together commits happen close together.
Ideally we'd like to allow the
build
of our workflow to proceed, but then pause until it can acquire a lock of some sort to proceed with the actual deploy.Alex Sweeney
This is 100% needed for us. The queue orb is not a good hack and won't work for our use case. As our company scales, using CircleCI for CI is going to become harder and harder for us and we may have to move to some other solution. People are getting blocked by their deploys failing because of this issue and it'll only get worse as we hire more people.
I'm kind of shocked that this isn't already an option, for CI this seems like a no-brainer.
C
Caran Chris
This is a big need for us. I've brought it up with our CSM.
Load More
→