Hi DevOps, how do you think your ideal programming language would look like? I mean a language in which you would write pipeline logic, like Python or Bash, not define pipeline steps itself, like YAML.
I think for me it would have:
- very clean and readable syntax
- immutable state by default
- strong typing
- strong tooling and IDE support
- focus on DevOps-need things, like JSON and files manipulation
- absence of danger things like pointers
You must log in or register to comment.
Nix comes closest. The biggest issue with Nix is that it does not admit a clean simple LL/LR grammar, but I think it admits a PEG, which is fine for practical work. The ecosystem could use more independent implementations and tooling, but I haven’t found any deficiencies with the language that would make me prefer e.g. Guile Scheme’s larger standard library and richer types.
HCL is another option. It’s not awful, but it’s verbose when trying to do anything higher-order, and it wasn’t intended to directly represent lambda-style subroutines. Guile Scheme, as used in Guix, is clearly capable, but requires embedding a fairly large standard library or having to NIH common routines; similar problems plague Lua or Python.
I think that your question has an interesting framing. My first answer was to mention jq and the relational pipes toolkit, but those are still run from a top-level shell. For example, I suppose that there’s two things that you can do with JSON: you can manipulate it and I would use jq for that, or you can load it as a datastructure into a (build) action and I would use Nix for that. jq is close to ideal for its particular formalization, but relational pipes are still evolving and I think that there are further simplifications that could be made.
The advantage of yaml is idempotency, so you really should be using generators to produce outputs that you then check in. This gives you a state repo, which when tools feed from to produce a final state, you always have the same result.
Check out CDK for an example.
There isn’t a universal language to fill all these needs. DevOps covers a ton of areas so you want several tools in your toolbelt instead of just one (granted if you’re an incredibly talented greybeard bash can honestly do everything and do it fairly fast; I’ve never taken this analysis too seriously).
- bash is something you need for every day tasks and quick tooling to improve everything else in this list
- POSIX tooling is something you should know well to improve everything else on this list (eg awk is something you’ll use a lot in bash)
- Python with or without types will handle more advanced scripting in a cleaner and more maintainable way for your team (explaining bashisms slows things down. Does your junior know what
:is or why we currently prefer$()over backticks?) - Go is something you want to use for containerization and scaling servers
- Rust is something you want to use for process work and embeddables
Each of these languages meets some of the requirements on your list but not all. That’s because nothing is a silver bullet. Depending on what you’re doing, you might even want to introduce, say, Java or C# in your pipelines if that’s what you’re maintaining. DevOps should support not alienate and overall it needs to be flexible.
I think the point of the question is what a hypothetical ideal language for CI/CD pipelines would look like.
☝️
Declarative systems are so much better to work in the domain, why would you force yourself into “pipeline logic”?
Sometimes you need to execute some logic in a pipeline step, like rename a file or push JSON somewhere.
like rename a file or push JSON somewhere.
Both of these can be done in a declarative fashion.
Yeah, yaml sucks. But we don’t need to throw the baby with the bathwater. You can have tools based in declarative systems that use saner languages, e.g, nix or TCL, and never have to deal with yaml issues.
The post doesn’t say “imperative”, it just differentiates between defining pipeline steps and defining the logic within a step.
…also, TCL? I haven’t used it for ops, but my memory of tcl/tk is extremely negative.
…also also: a core part of a build, CI, or, CD pipeline is almost always invoking binaries to run a command. That’s why shell scripts are so ubiquitous in pipeline-logic: invoking binaries is what they’re for. And it’s very difficult to do that a declarative way: Make comes close, but it’s difficult to track any side-effects that aren’t “update these files”, and a huge amount of CI/CD is no longer just “update a file”.
TCL as in Terraform Configuration Language.
Oh, that makes much more sense; thanks.
I don’t have an ideal, but a go to Usually python scripts…
Same here I use a mix of python without libs and bash for DevOps.
I prefer pyinfra over ansible.
I like https://github.com/bazelbuild/starlark as a concept, but it is sadly quite unpopular.
Tilt utilizes it and it’s interesting.
Most of them are bash scripts as they usually don’t need any specific tools installed and run out of the box. When it gets more complicated (especially if there are lots of asynchronous and parallel tasks) I go to Node.js / Typescript. This is maybe a little bit of an oddball, but there are plenty of tools to create CLI binaries that can run independently and working asynchronously is really easy. I like the language a lot more than I like Python – I haven’t tried Go yet.
I personally like golang a lot for small-ish scripting and programming. It has strong typing and few foot guns (e.g. bash with the weird way arrays are handled, implicit splitting of array items etc). Downside is it needs to be compiled up front and is basically a black box from the CI configuration’s POV (you can inspect the source of a circleci orb or GitHub action for example).
I use terraform, helm, a bit of ansible and gitlab/forgejo (for cicd). The minimum amount of scripting I do is shell, I don’t have a need for more…
Back in the days of manual deployments and semi-automated configurations, I did use bash.
What is your need for complex scripting/programming?
I mainly do things for Jenkins so I mainly code in Groovy. Most complex things I need to do is to manipulate some JSON, files, or strings. Send some messages to Slack, or push some data to DB.
Most things I’d say is bash, easy just to write a simple task in it. If you have to use some more logic I use python. But python is far from great for it. It is easy for me to run a venv and test the stuff. But “this script should do a cron thing”, “this is an filter for ansible”, “this performs a job here or there”, managing a uniform consistent env for all of it is a mess. Bjt yeah, if it is just pipeline logic, ok that env can be managed.
