Add support for sharing entire workspace amongst jobs
R
Robby Marston
Currently, my build and test steps are both part of the same lengthy job that features steps to build a Docker image, checkout code, install Ruby gems, install Node.js packages, etc. That said, I'd like to move my test steps to a new job so that I can leverage the benefits of Workflows. It's my understanding that the context or progress made by the initial build job cannot be leveraged by subsequent jobs. What I mean by that is, my new test job would have to do everything over again (rebuild the Docker image, check out the code, etc.) in order for my application to be in a state where tests can execute successfully. In the future, I'd like to add additional Workflows for deployment etc.; however, this level of redundancy will make for quite a bloated config file and lengthy execution time (when all jobs are ran in succession).
CCI-I-878
V
Vinny Thanh
To give context, a "workspace" or "cache" is just us tarring up a bunch of files, uploading them, and then downloading them again in the next job. There's no persistent environment or disk being shared between jobs. So transferring work or artifacts between them must use either workspaces or caches, or something external you manually implement.
For your node.js packages and ruby gems, I'd recommend using the dependency caching. Those persist beyond just a single workflow run, so many runs on many different branches can make use of the cache if the dependencies didn't change. I'd also recommend using a separate cache each for node modules and ruby gems (e.g., keying on the SHA of your package.json and Gemfiles), that way you can cache and update each independently.
For Docker images, we do have Docker Layer Caching, a premium feature. But if you'd prefer to transfer a built image, you can just run docker save / docker load steps and then transfer the image tarball between jobs in a workspace. Alternatively, you could just push a built image to a registry and re-pull it in the next job.
If you really wanted to, you could store
everything
in a workspace and transfer it all over to the next job. But depending on how much content there is, it might be really slow to do so and would be easier to use the strategies I mentioned above.Hope this helps!
Documentation:
- https://circleci.com/docs/2.0/caching/#basic-example-of-dependency-caching
- https://circleci.com/docs/2.0/workflows/#using-workspaces-to-share-data-among-jobs
- https://circleci.com/docs/2.0/docker-layer-caching/
- https://circleci.com/blog/optimizing-open-source-projects-on-circleci/ (scroll down/search for Reusing previously-built Docker images)