Update Path Filtering Orb to be configurable
Vincent Taverna
Recently I learned of the circleci/path-filtering orb that uses the continuation API in a Setup Workflow. This is a really powerful set of tools, except for some reason the
base-revision
is not something that can be dynamically controlled. While you can set the base-revision for the path filtering orb to any valid value, including other parameters, it is not possible to dynamically create a value for the base-revision. For example, if the base-revision is set to a value of master
it works fine. But if we create a pre-step that gets the base revision from Github, the value cannot be used as the base revision. Relying on the pipeline.git.base-revision
as a dynamic revision is not reliable, and doesn't work the way one might hope it would. In practice this means that developers using CircleCI in my project need to manually update the base-revision whenever they are working on a different base from main
/ master
. Can the orb be updated to accept an environment variable exported into $BASH_ENV from a pre-step, or can we have some other mechanism for dynamically controlling this parameter?Ideally I would be able to do something like this
version: 2.1
setup: true
orbs:
path-filtering: circleci/path-filtering@0.0.2
workflows:
generate-config:
jobs:
- path-filtering/filter:
pre-steps:
- run:
name: Fetch Base Branch from Github
command: |
BASE_BRANCH=$(curl -X GET \
-H "Authorization: token $GITHUB_TOKEN" \
-H "Accept: application/vnd.github.groot-preview+json" \
-L "https://api.github.com/repos/$CIRCLE_PROJECT_USERNAME/$CIRCLE_PROJECT_REPONAME/commits/$CIRCLE_SHA1/pulls" \
| jq '.[].base.ref')
echo "export BASE_BRANCH=$BASE_BRANCH" >> $BASH_ENV
base-revision: $BASE_BRANCH
mapping: |
path/to/dir/.* build-my-job true
Path Filtering orb
Philipp Kyeck
Did you get this to work somehow?
Poya Moshiry-Zavieh
Hey Vincent,
I'd also like the configuration to be configurable but I found I could get much of the same effect by either using filters
```
- path-filtering/filter:
name: qa-build
filters:
branches:
ignore:
- /^prod.*/
- /^master.*/
- /^release.*/
base-revision: origin/dev
mapping: |
frontend/.* frontend true
backend/.* backend true
config-path: .circleci/workflows.yml
- path-filtering/filter:
name: prod-build
filters:
branches:
only:
- /^prod.*/
- /^master.*/
- /^release.*/
base-revision: origin/master
mapping: |
frontend/.* frontend true
backend/.* backend true
config-path: .circleci/workflows.yml
```
or by making use of the path-filtering/set-parameters command inside of my own custom job (in conjunction with continuation/continue) and passing the base-revision as a parameter.
Philipp Kyeck
Poya Moshiry-Zavieh: Do you have an example for the second part of your answer?
J
James Nguyen
Philipp Kyeck: Hope this helps, but I got this working from Poya's answer above.
>> config.yml
version: '2.1'
setup: true
orbs:
path-filtering: circleci/path-filtering@0.1.3
workflows:
generate-config:
jobs:
- path-filtering/filter:
name: staging-build
base-revision: main
config-path: .circleci/continue-config.yml
mapping: |
applications/project01/.* project-one-build true
applications/project02/.* project-two-build true
applications/project03/.* project-three-build true
common/.* build-them-all true
filters:
branches:
ignore:
- /^integration.*/
- /^prod.*/
- path-filtering/filter:
name: integration-deploy
base-revision: origin/integration
config-path: .circleci/continue-config.yml
mapping: |
applications/project01/.* project-one-deploy true
applications/project02/.* project-two-deploy true
applications/project03/.* project-three-deploy true
common/.* deploy-them-all true
filters:
branches:
only:
- /^integration.*/
- path-filtering/filter:
name: prod-deploy
base-revision: origin/prod
config-path: .circleci/continue-config.yml
mapping: |
applications/project01/.* project-one-deploy true
applications/project02/.* project-two-deploy true
applications/project03/.* project-three-deploy true
common/.* deploy-them-all true
filters:
branches:
only:
- /^prod.*/
>> continue-config.yml
version: 2.1
executors:
base:
docker:
- image: cimg/base:stable
parameters:
project-one-build:
type: boolean
default: false
project-two-build:
type: boolean
default: false
project-three-build:
type: boolean
default: false
build-them-all:
type: boolean
default: false
project-one-deploy:
type: boolean
default: false
project-two-deploy:
type: boolean
default: false
project-three-deploy:
type: boolean
default: false
deploy-them-all:
type: boolean
default: false
jobs:
project_one_build:
executor: base
steps:
- run:
command: |
echo "project one"
project_two_build:
executor: base
steps:
- run:
command: |
echo "project two"
project_three_build:
executor: base
steps:
- run:
command: |
echo "project three"
all_projects_build:
executor: base
steps:
- run:
command: |
echo "all"
project_one_deploy:
executor: base
steps:
- run:
command: |
echo "project 1 deploy time!"
project_two_deploy:
executor: base
steps:
- run:
command: |
echo "project 2 deploy time!"
project_three_deploy:
executor: base
steps:
- run:
command: |
echo "project 3 deploy time!"
all_projects_deploy:
executor: base
steps:
- run:
command: |
echo "release the hounds!"
workflows:
build-1:
when:
or:
- << pipeline.parameters.project-one-build >>
- << pipeline.parameters.build-them-all >>
jobs:
- project_one_build
build-2:
when:
or:
- << pipeline.parameters.project-two-build >>
- << pipeline.parameters.build-them-all >>
jobs:
- project_two_build
build-3:
when:
or:
- << pipeline.parameters.project-three-build >>
- << pipeline.parameters.build-them-all >>
jobs:
- project_three_build
build-shared-other:
when: << pipeline.parameters.build-them-all >>
jobs:
- all_projects_build
deploy-1:
when:
or:
- << pipeline.parameters.project-one-deploy >>
- << pipeline.parameters.deploy-them-all >>
jobs:
- project_one_deploy
deploy-2:
when:
or:
- << pipeline.parameters.project-two-deploy >>
- << pipeline.parameters.deploy-them-all >>
jobs:
- project_two_deploy
deploy-3:
when:
or:
- << pipeline.parameters.project-three-deploy >>
- << pipeline.parameters.deploy-them-all >>
jobs:
- project_three_deploy
deploy-shared-other:
when: << pipeline.parameters.deploy-them-all >>
jobs:
- all_projects_deploy