Sequentially run jobs in a branch | Feature Requests

Sequentially run jobs in a branch

complete

Kyle Tryon

In some instances it may be that a user makes multiple commits to a branch in a short amount of time, causing multiple Workflows to execute at the same time, possibly deploying in an incorrect order. "Auto Cancel Redundant Builds" may not be suitable for all situations in which multiple deployments are expected. Solution: Enable a way to ensure that commits to a new branch are queued for sequential execution.

CCI-I-840

January 24, 2019

Fernando Abreu

(Copying this from a reply for visibility):
We’re planning to extend our requires syntax to support requiring any job that’s in a terminal state, which will now also include blocked (i.e., jobs that will never run because a dependency wasn't met).
This would allow queuing multiple jobs as part of the same logical group.
Example:
  jobs:
    - build
    - lock:
        key: << pipeline.git.branch >>
        requires: build
    - deploy:
        requires:
          - lock
    - release:
        requires:
          - deploy
    - unlock:
        requires:
          - lock: success
          - release: terminal
In this setup, lock indicates the start of a Serial Group. With the new terminal keyword, you can include deploy and release in the same group, regardless of whether release succeeds, fails, or is blocked. Of course, you could have any number of arbitrary jobs between lock and unlock jobs.
Would this be helpful in your workflows?

Victor Sudakov

Fernando Abreu Not immediately. Our workflows are not that sophisticated now. But might be good to have for the future.

Nacho Retamoza

We are trying to use this new feature (very useful!)
But we are facing a problem.
We are using the same key for 2 jobs in the same repository. One after the other: deploy and test.
So we have deploy and test for pipeline1 and pipeline2: p1_d, p1_t, p2_d and p2_t
p1_d is blocking p2_d, that is fine.
But after p1_d finishes, p1_t is auto canceled and p2_d starts running.
We were expecting p1_t to start running right after p1_d, as it has a lower pipeline number than p2

Fernando Abreu

Nacho Retamoza: Thanks for reaching out!
Serial groups don’t enforce ordering when jobs are added to the queue, but they ensure that within the same project, jobs with lower pipeline numbers will not run after a job within the same serial group that has a higher pipeline number.
In your case, here’s what’s happening:
p1_d gets queued, followed by p2_d.
p1_t and p2_t only get queued once p1_d and p2_d finish, respectively.
By the time p1_t enters the queue, p2_d is already ahead, causing p1_t to be skipped.
We understand that this isn't the expected behavior for your use case. We’re actively working on a feature that will allow multiple jobs to be queued together. With this, you’d be able to queue p1_d and p1_t together, ensuring that p2_d and p2_t will only run after p1_t finishes.
Would this solution address your issue?

Fernando Abreu

For more details on Serial Groups, you can check out the official documentation here: https://circleci.com/docs/configuration-reference/#serial-group

Nathan Fish

marked this post as

complete

The first release of this is complete and available to customers. Documentation is updated and a change log entry will go out today. We have more planned to extend this to groups of jobs. We will update this ticket and the change log when that is available.

Fernando Abreu

We're excited to share that our new serial grouping feature is ready for testing. This feature allows you to queue jobs under the same serial-group
 key. 
To get started, simply add serial-group
 to your job configuration. Here's a quick example:
workflows:
  main-workflow:
    jobs:
      - test
      - deploy:
          serial-group: << pipeline.git.branch >>
          requires:
            - test
We're eager to hear your thoughts and experiences with this feature. If you have any questions or feedback, please feel free to share them with us.
We're still working on the UI, so some of the user experience may change in the upcoming days and weeks..
Thank you for helping us improve!

Jordan

Thanks for the new feature!

We have a workflow like this:

- build
- [several deploy-related jobs, some in parallel, some sequential]

"build" should be free to run in parallel across all workflows, but the subsequent deploy jobs should be done

as a group

sequentially, so that only one Workflow at a time is performing any of those steps, and the Workflows run in the order they were created.

Are we able to accomplish that with this new feature?

Fernando Abreu

Jordan: Thank you for reaching out and sharing your use case. Currently, this feature doesn't support queuing multiple jobs together, but we plan to address this in the future.

Victor Sudakov

Fernando Abreu When I tried this configuration:
workflows:
  dev:
    jobs:
      - deploy-dev:
          serial-group: << pipeline.git.branch >>
          context: XXX
          filters:
            branches:
              only:
                - /^STAGE-.*/
I got the CircleCI error:
Error calling workflow: 'dev'
Error calling job: 'null'
Arguments referenced without declared parameters: parameters.post-steps, parameters.pre-steps
Am I missing something?

Victor Sudakov

Fernando Abreu

circleci config validate

does not complain about the config with the

serial-group: << pipeline.git.branch >>

line added.

Fernando Abreu

Victor Sudakov Thank you for testing our feature and for your feedback! We've addressed the issue you mentioned. Could you please check if it's now working as expected?

Let us know if you encounter any more issues or have additional feedback. We appreciate your help!

Victor Sudakov

Fernando Abreu Looks like the error is fixed. I will continue testing tomorrow with simultaneous git pushes.

Victor Sudakov

Fernando AbreuI It would be nice if the jobs waiting for their turn in the queue were shown as "Waiiting", not "Running" in the UI. Is it possible?

Fernando Abreu

Victor Sudakov: Thank you for testing the feature again! I'm glad to hear it's working now. Regarding the UI, yes, we’re still working on improving the status display. Jobs in the queue will have a different status, possibly "Waiting," to better reflect their state.

Timo Sand

Fernando Abreu Does this feature only impact jobs that run at the same time or would it also impact jobs that are run later out-of-order? We sometimes have humans mis-clicking rerun on an older workflow execution, which then deploys an older version

Timo Sand

Fernando Abreu Is the

serial-group

scoped to one project or workflow or job only? Or could this allow us to coordinate jobs between multiple projects, like a semaphore?

Fernando Abreu

Timo Sand: Yes, this feature only impacts jobs that are currently running. Once the queue is empty, you would be able to rerun an older workflow execution without restriction.

The serial group queue is scoped at the organization level, meaning it can act as a semaphore for a shared resource across multiple projects. However, note that only one job can run at a time within the serial group—any additional jobs will be queued until the running job completes.

Fernando Abreu

Jordan: We’re planning to extend our requires syntax to support requiring any job that’s in a terminal state, which will now also include blocked (i.e., jobs that will never run because a dependency wasn't met).
This would allow queuing multiple jobs as part of the same logical group.
Example:
  jobs:
    - build
    - lock:
        key: << pipeline.git.branch >>
        requires: build
    - deploy:
        requires:
          - lock
    - release:
        requires:
          - deploy
    - unlock:
        requires:
          - lock: success
          - release: terminal
In this setup, lock indicates the start of a Serial Group. With the new terminal keyword, you can include deploy and release in the same group, regardless of whether release succeeds, fails, or is blocked. Of course, you could have any number of arbitrary jobs between lock and unlock jobs.
Would this be helpful in your workflows?

Fernando Abreu

We’re introducing serial groups as a way to queue jobs that share the same key, ensuring that only one runs at a time while others wait in a queue. For example, if multiple jobs across workflows share the same serial group key (e.g., the branch name), only one will proceed at a time, while others remain queued until their turn.

Example Configuration:

jobs:
  test:
    type: no-op
  deploy:
    type: no-op
workflows:
  main-workflow:
    jobs:
      - test
      - deploy:
          serial-group: << pipeline.git.branch >>
          requires:
            - test

In this example, the deploy job is queued based on the branch name. If three workflows are running on the main branch, only one deploy will execute at a time, while the others wait in the queue.

There is a UI mockup attached to this post. In the mockup, you can see that the deploy job waited 5 min and 11 sec in the queue and it’s currently running.

Looking for Feedback on:

Is the serial group concept clear?
Is the UI intuitive in showing when a job is running?
Are there any additional details you’d expect to see in the UI?

- Any concerns?

Victor Sudakov

Fernando Abreu It looks clear to me for the present, I cannot wait to start using this feature.

Ben Limmer

Fernando Abreu I slightly prefer a term like

concurrency-key

or similar, but that's certainly not a blocker. As described, this is exactly what my team is looking for! I'd love to get rid of my hacky

queue

orb.

Fernando Abreu

Victor Sudakov: thank you for the feedback!

Fernando Abreu

Ben Limmer: thanks for the feedback. I will chat with the team about the name

Jean-Christophe Delmas

Fernando Abreu Thanks for this new feature! Could you clarify if the serial group would only prevent two jobs from running at the time, or if it also ensures that the jobs are executed in the same order as the pipeline?

Fernando Abreu

Jean-Christophe Delmas Thanks for the question! To clarify, the serial group can be configured to prevent any number of jobs from running simultaneously under the same serial-group key. For example, if you have 10 workflows attempting to deploy, only one will run at a time. Currently, we use the pipeline number order to determine this sequence. Therefore, if a job from a pipeline with a higher number is queued first, the job from the lower-numbered pipeline will not execute.

Info

Show this "idea" to your CEO and tell them that people are leaving CircleCI because it fails to do what it claims - produce consistent builds.

Nathan Fish

How would folks feel about something like the following config syntax to support this? Within a job you can specify that the job should run sequentially based on a unique key that can be generated from pipeline parameters or values. This would prevent race conditions and ensure that the job is run to completion in the order in which it was received.

...
  deploy:
    concurrency: 
      key: <<parameters.projectname>>-<<git.branch>>-<<parameters.target>>
    steps:
      - checkout
      ...
  validate:
    concurrency: 
      key: <<parameters.projectname>>-<<git.branch>>-<<parameters.target>>
    steps:
      ...

...
workflows:
  version: 2
  build-and-deploy:
    jobs:
      - build
      - deploy
	   concurrency: 
      	     key: <<parameters.env>>-<<git.branch>>
      - validate
	   concurrency: 
      	     key: <<parameters.env>>-<<git.branch>>

James Taylor

Nathan Fish That would be great

Victor Sudakov

Nathan Fish for our purposes it would be sufficient to have a marker in the config that this workflow should run sequentially, and the workflows are queued in the order of received Github hooks. If your suggestion can be used like that, I'm fine with it. Will this syntax be supported for jobs, or workflows, or both?

Also, can you comment in what order the jobs with identical keys will be queued?

Nathan Fish

Victor Sudakov yes this would be based on received github hooks. We were looking specifically at job specific queues, which could be scoped to the org, workflow, or other pipeline parameters. It would not be an entire workflow, it would be the individual jobs that are queued. We can look at entire workflows as well but the plan right now is for jobs.

Ben Limmer

Nathan Fish Looks good for MVP.
Would the default behavior be to not cancel in-progress jobs? In GitHub actions, the default is to cancel-in-progress unless set otherwise: https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#jobsjob_idconcurrency . Having a different behavior here might be confusing to folks.
Also +1 to be able to wait for other workflows. Today, I'm using the queue orb (https://circleci.com/developer/orbs/orb/eddiewebb/queue#jobs-block_workflow) to run some jobs (test, build docker image) but wait to execute a deployment until other in-progress deployment workflows are complete.

Nathan Fish

Ben Limmer I'll take that into consideration with the team, I agree similar behavior would be beneficial. It would also be valuable to enable customers to define "non-cancellable" jobs, to ensure that something can't be canceled once started. While not related directly to concurrency as a capability I think some of the use cases for concurrency also benefit from not allowing jobs to be canceled or to be protected from being canceled.

Nathan Fish

marked this post as

planned

Nathan Fish

Merged in a post:

Queue parallel builds to deploy and test in sequence

Rob Grant

As a software engineer
I want to run end to end tests for each change
So I can quickly identify breaking changes
Example scenario
There are 5 repos to build, for services 1-5. Services 1, 2, 3 and 4 have a large number of inter-service tests; service 5 is standalone.
Upon building code for service 1, 2, 3 or 4 in Circle CI, the build enters a step that deploys the build to an integration test environment (ITE), and triggers the full suite of tests. If more than pull request for Service 1, 2, 3 or 4 triggers this step at a similar time, the deploy to ITE is FIFO queued pending a successful outcome for the currently running integration tests.
Upon building the code for service 5, the build and deploy waits for any other Service 5 builds to pass tests, and then it deploys and runs the tests.
If Service 1, 2, 3 or 4 fails tests, their queue is paused until the problem is fixed (a successful rerun of that branch's tests) and then the builds continue.
If Service 5 fails its tests, its queue is paused until the problem is fixed (a successful rerun of that branch's tests) and then any other Service 5 builds continue.
Notes
Ideally builds should be able to run independently to a point (so unit tests can run and Docker images can build) but then should enter a sequencing queue.
Builds with interdependent repos could be gathered into Build Groups for easy configuration of sequencing.
CCI-I-1056

October 16, 2024

→