Provide the capability to source / run a file before every RUN step
M
Mtwomey
In CircleCI 1.0 you could do this by adding commands to the .circlerc file. This was great - because it was driven by CirclCI itself and didn't rely on shell specific tricks (e.g. like $BASH_ENV). In CircleCI 2.0 this capability is not present and there is currently no way to do this for the official Alpine docker images. You have to include your step(s) in every single run command.
We have large modular build and deploy scripts with many run commands and having this capability keep things simple and follows DRY principals.
It's also a very flexible feature
- People can easily carry environment variables across step (or even set them from within a step and have them available to other steps)
- You can setup you shell the way you need to before each run step
More detail here: https://discuss.circleci.com/t/circlerc-bash-env-env/25011/4
CCI-I-600
Brandon Chinn
Some additional context, copied from a separate request:
We would like to do certain actions before every step in a CI job, such as turning on
set -ux
and setting environment variables (like VERSION
). Currently, as I understand it, there are two ways to do this:- Append to $BASH_ENV, which gets sourced at the beginning of each file
- Customize the shellparameter of the executor (only relevant for theset -uxaction, doesn't help with the env var case)
Putting
set -ux
in either BASH_ENV
or shell
doesn't work great out-of-the-box, however. Because BASH_ENV
is sourced for every bash shell, that means set -ux
is run when a shell script is run, when SSHing into a job, and even potentially before ~/.bashrc
is sourced. This is problematic because third-party shell scripts (and some executors' ~/.bashrc
) are not set -u
safe. As an example, the cimg/python
images have a pip
shim which is not set -u
safe, which errors out due to the pip
shim also setting set -e
. It's also troublesome in the machine
executor because the cd
command in machine
is also a shim and is not set -u
safe (workaroundable using builtin cd
).It would be great if Circle CI could have a dedicated env var like
CIRCLE_BASH_ENV
that it explicitly sources at the beginning of every step. This solves the initial issue because running shell scripts within a step would not automatically source CIRCLE_BASH_ENV
. As a bonus, it would work regardless of the shell, so the Circle CI docs could remove notes about it being bash specific (e.g. the end of the section at: https://circleci.com/docs/2.0/env-vars#using-parameters-and-bash-environment)------------------------------------------------------------
a
.circlecirc
file would also work. In general, there needs to be something Circle CI-specific, not something that hooks into bashM
Mtwomey
[Additional note for item 3] You also can't force the shell command to bash at the job level, because it's not yet present before the first run step in which you actually install bash.
M
Mtwomey
Nathan, this is great functionality! I think it will be useful in many ways. I do think this makes things a little easier. However, I still hold to the idea that there should be a known file sourced with every container - there are just so many uses for it - and frankly, it's the most simple way to solve this (and several other) problems.
In the research and testing I've done - I've identified several possible solutions to this:
Bring back the concept of a file that is sourced every time a container is launched (e.g. the .circelcirc approach).
Set $ENV=$BASH_ENV in the parent environment, this fixes the issue for Alpine images (because ash sources the file pointed to by $ENV). However, due to the lack of variable interpolation in circle.yml, there is no way to set this.
Update this functionality here:
"The default value of shell option is /bin/bash -eo pipefail if /bin/bash is present in the build container. Otherwise it is /bin/sh -eo pipefail. The default shell is not a login shell (--login or -l are not specified by default). Hence, the default shell will not source your ~/.bash_profile, ~/.bash_login, ~/.profile files. Descriptions of the -eo pipefail options are provided below."
This appears to only check for the presence of bash one time (at the very beginning). For example if you use an Alpine image (which doesn't have bash by default), and immediately install bash as your first step, it will still use /bin/sh -eo pipefail on subsequent containers. If it checked for the presence of bash every time it spun up a container, that would also fix this issue.
N
Nathan Dintenfass
You can read more about the new 2.1 features here: https://github.com/CircleCI-Public/config-preview-sdk/blob/master/docs/whats-new.md
N
Nathan Dintenfass
Curious if you'd be OK using the new 2.1 config (currently in preview) that has a new
commands
key where you can define reusable steps that take parameters. One of the parameter types is "steps" that lets you pass a sequence of steps to the command and then reference them in a flow of other steps. In that way you could create a command that does whatever think you want to do before a series of steps, then pass the steps you want to run. Using that technique it would end up looking something like this:commands: prefixed: parameters: steps: type: steps steps: - run: echo "whatever you want to do here" - steps: << parameters.steps >>jobs: build: docker: image: alpine:latest steps: - prefixed: steps: - run: "some step" - prefixed: steps: - run: "some other step"