Provide a CircleCI DSL alternative to YAML
Aaron Peschel
Can we address the elephant in the room, please? YAML is not a programming language, no matter how many special keys are bolted onto it. It's literally in the acronym. YAML Ain’t Markup Language.
Here are things YAML doesn't have that should be present in a decent declarative DSL:
code reuse
functions
variables
libraries
YAML anchors don't count for much.
conditionals and control flow
boolean logic
if blocks
string interpolation
arithmetic
regular expressions
string matching
data structure methods
joining lists
list iteration
etc
CircleCI is trying to fix problems with YAML by bolting more and more special keys on to emulate these things YAML doesn't have, but this just ends up uncovering more problems.
For example, YAML doesn't support functions or libraries, so CircleCI came up with "commands" and "orbs" to attempt to do the same thing. This is a nice addition, but it has now made YAML's lack of other features even more painful. There is no way to pass an interpolated value to a command or an orb, and you're left either having to use static values or really "clever" workarounds to deal with the problem.
CircleCI can probably fix these new problems by adding additional functionality, then rinse and repeat until everything is golden. But if we take this to its extreme conclusion, CircleCI is just going to essentially recreate Jsonnet, and I'm guessing there's a reason CircleCI didn't just use Jsonnet in the first place.
So rather than continuing this insane dance around the core problem of YAML not actually being suited for use as a declarative DSL, can we have CircleCI maybe consider creating a proper DSL for configuring a CI run on their platform?
CCI-I-950
[
[not provided] apujari-hippo
I think a language-based DSL (like Kotlin & Groovy DSL) would be much more useful.
Python is very popular nowadays... Maybe we can have Python DSL for Circle CI.
V
Vinny Thanh
Hi Aaron,
Totally feel you here. CircleCI has rolled out several items you've mentioned for our configuration. See:
- Config Reuse (https://circleci.com/docs/2.0/reusing-config/)
- Pipeline Parameters (https://circleci.com/docs/2.0/pipeline-variables/)
- Conditional Steps (https://circleci.com/docs/2.0/reusing-config/#defining-conditional-steps)
- Matrix Jobs (https://circleci.com/docs/2.0/configuration-cookbook/#use-matrix-jobs-to-run-multiple-os-tests)
The parameters should allow for interpolating items into your configuration. In addition, we're also working on a feature called Setup Workflows, which allows for dynamic construction of configuration. See this idea here: https://ideas.circleci.com/cloud-feature-requests/p/dynamic-config-aka-setup-jobs
Hope this helps!
T
Timothée GERMAIN
Hi,is there a github issue / circleCI idea to track the dynamic config generation feature ?It sounds great but it is not a trivial addition.
V
Vinny Thanh
Timothée GERMAIN: Yes, see this idea here: https://ideas.circleci.com/cloud-feature-requests/p/dynamic-config-aka-setup-jobs
N
Nathan Dintenfass
Our general approach here is likely to be a way to dynamically generate config when you trigger work on CircleCI. We do still believe in a declarative model for defining your pipelines, so building out a full-blown DSL in a language like Javascript would create a new category of complications and debugging. Instead, our hope is to make it possible to let you perform arbitrary work to conjure the configuration, at which point we could also have a DSL (or several) in a popular language that would generate the config in a way you could run locally to test, for instance. This same technique (dynamic config generation) will also be useful for those wanting to store multiple configs in a monorepo or those that want to do dynamic work based on arbitrary state of code or of the commit tree.