Allow serial groups to wrap multiple jobs
planned
l
leah.keith@gousto.co.uk
We would like to wrap multiple consecutive jobs in one serial group. For example, we run automated testing against a deployed ECS container. We don't want to one run to deploy the service while a previous run is in the middle of testing against the deployed service. This causes inconsistent test results as some of the tests could be running against the new instance of the service which could have different behaviour.
Current pseudo-config:
main-workflow:
jobs:
- checkout-and-build
- test-unit
- deploy-service <- serial group 1
- test-end-to-end <- serial group 2
- deploy-prod <- serial group 3
proposed:
main-workflow:
jobs:
- checkout-and-build
- test-unit
- deploy-service <- serial group 1
- test-end-to-end <- serial group 1
- deploy-prod <- serial group 2
Minjun Seong
marked this post as
planned
Hi folks, I'm a product manager at CircleCI. I'd love to understand the problem you're trying to solve a bit better:
Whats the situation you are currently running into that requires the need to wrap multiple jobs together when using serial groups?
Do you need to wrap groups of jobs or the whole workflow?
What is your current workaround to solve this problem?
Would appreciate as much detail as possible about your current experience!
l
leah.keith@gousto.co.uk
Minjun Seong Thanks for the response! I've provided a bit more detail about our situation below, let me know if you have any questions.
Here is our current workflow
- checkout and build
- unit tests
- functional tests
- deploy to staging
- end-to-end tests (these run against staging)
- deploy to production
In order for our service to be deployed to production, end-to-end tests must run and pass against our service deployed in a staging environment. We push code frequently, so we often have multiple pipeline runs going at a time. If Pipeline A is running end-to-end tests against staging while Pipeline B is deploying to staging, Pipeline A's e2e tests often fail while services are deploying, or they fail because the behavior of the service has changed in the commit that triggered Pipeline B which the tests in Pipeline A don't know about.
If we were able to put the deploy to staging and e2e test jobs in one serial group, it would ensure that every pipeline run is running e2e tests against the version of staging it expects. It also would mean no disruption due to intermittent down time while services deploy.
We don't need to wrap the full workflow, in our situation it is only these two specific jobs - deploy to staging and end-to-end tests. We want the rest of the pipeline to continue running in parallel where possible.
Our current workaround: we have combined our deploy and end-to-end testing jobs into one so that we can wrap it in a single serial group. However this is far from ideal. Our e2e tests need to run on a self-hosted runner while our deploy etc jobs need to run on a machine/image that our platform team manages. So we've had to write a custom orb that triggers the end-to-end tests in a separate pipeline from our main pipeline and waits for it to complete. This has slowed down our pipeline significantly because we don't have the context of the single pipeline and just the overhead of spinning up another pipeline. It is also harder to understand what is going on when looking at the CircleCI UI.
We considered using https://eddiewebb.github.io/circleci-queue/ but our organization does not allow the use of public uncertified orbs for security reasons.
Hope this helps, let me know if you need anything else!
e
eric.mueller@framebridge.com
Minjun Seong We have a similar need - in our case we have several concurrent jobs that each deploy a different service from the monorepo, and we need to prevent the deployment of any of the services until _all of them_ are finished with their prior deployment.
I'd be satisfied with just serializing the entire workflow (on the master branch only), but if I could set all of those deployment jobs as a single entry in a serializable group, that would be ideal (and increase maximum throughput by about 2x)
E
Eddie
Merged in a post:
CircleCI tail
A
Andriy Semenchenko
Can we please get https://eddiewebb.github.io/circleci-queue/ as default feature to avoid additional costs while waiting for previous builds?