Matthew Palmer: Your Release Process Sucks
For the past decade-plus, every piece of software I write has had one of two release processes.
Software that gets deployed directly onto servers (websites, mostly, but also the infrastructure that runs Pwnedkeys, for example) is deployed with nothing more than git push prod main.
I ll talk more about that some other day.
Today is about the release process for everything else I maintain Rust / Ruby libraries, standalone programs, and so forth.
To release those, I use the following, extremely intricate process:
- Create an annotated git tag, where the name of the tag is the software version I m releasing, and the annotation is the release notes for that version.
-
Run
git releasein the repository. - There is no step 3.
Sidebar: annotated what-now?!?
The annotated tag is one git s best-kept secrets.
They ve been available in git for practically forever (I ve been using them since at least 2014, which is practically forever in software development), yet almost everyone I mention them to has never heard of them.
A tag , in git parlance, is a repository-unique named label that points to a single commit (as identified by the commit s SHA1 hash).
Annotating a tag is simply associating a block of free-form text with that tag.
Creating an annotated tag is simple-sauce: git tag -a tagname will open up an editor window where you can enter your annotation, and git tag -a -m "some annotation" tagname will create the tag with the annotation some annotation .
Retrieving the annotation for a tag is straightforward, too: git show tagname will display the annotation along with all the other tag-related information.
Now that we know all about annotated tags, let s talk about how to use them to make software releases freaking awesome.
Step 1: Create the Annotated Git Tag
As I just mentioned, creating an annotated git tag is pretty simple: just add a -a (or --annotate, if you enjoy typing) to your git tag command, and WHAM! annotation achieved.
Releases, though, typically have unique and ever-increasing version numbers, which we want to encode in the tag name.
Rather than having to look at the existing tags and figure out the next version number ourselves, we can have software do the hard work for us.
Enter: git-version-bump.
This straightforward program takes one mandatory argument: major, minor, or patch, and bumps the corresponding version number component in line with Semantic Versioning principles.
If you pass it -n, it opens an editor for you to enter the release notes, and when you save out, the tag is automagically created with the appropriate name.
Because the program is called git-version-bump, you can call it as a git command: git version-bump.
Also, because version-bump is long and unwieldy, I have it aliased to vb, with the following entry in my ~/.gitconfig:
[alias]
vb = version-bump -n
Of course, you don t have to use git-version-bump if you don t want to (although why wouldn t you?).
The important thing is that the only step you take to go from here is our current codebase in main to everything as of this commit is version X.Y.Z of this software , is the creation of an annotated tag that records the version number being released, and the metadata that goes along with that release.
Step 2: Run git release
As I said earlier, I ve been using this release process for over a decade now.
So long, in fact, that when I started, GitHub Actions didn t exist, and so a lot of the things you d delegate to a CI runner these days had to be done locally, or in a more ad-hoc manner on a server somewhere.
This is why step 2 in the release process is run git release .
It s because historically, you can t do everything in a CI run.
Nowadays, most of my repositories have this in the .git/config:
[alias]
release = push --tags
Older repositories which, for one reason or another, haven t been updated to the new hawtness, have various other aliases defined, which run more specialised scripts (usually just rake release, for Ruby libraries), but they re slowly dying out.
The reason why I still have this alias, though, is that it standardises the release process.
Whether it s a Ruby gem, a Rust crate, a bunch of protobuf definitions, or whatever else, I run the same command to trigger a release going out.
It means I don t have to think about how I do it for this project, because every project does it exactly the same way.
The Wiring Behind the Button
It wasn t the button that was the problem. It was the miles of wiring, the hundreds of miles of cables, the circuits, the relays, the machinery. The engine was a massive, sprawling, complex, mind-bending nightmare of levers and dials and buttons and switches. You couldn t just slap a button on the wall and expect it to work. But there should be a button. A big, fat button that you could press and everything would be fine again. Just press it, and everything would be back to normal.
- Red Dwarf: Better Than Life
Once you ve accepted that your release process should be as simple as creating an annotated tag and running one command, you do need to consider what happens afterwards.
These days, with the near-universal availability of CI runners that can do anything you need in an isolated, reproducible environment, the work required to go from annotated tag to release artifacts can be scripted up and left to do its thing.
What that looks like, of course, will probably vary greatly depending on what you re releasing.
I can t really give universally-applicable guidance, since I don t know your situation.
All I can do is provide some of my open source work as inspirational examples.
For starters, let s look at a simple Rust crate I ve written, called strong-box.
It s a straightforward crate, that provides ergonomic and secure cryptographic functionality inspired by the likes of NaCl.
As it s just a crate, its release script is very straightforward.
Most of the complexity is working around Cargo s inelegant mandate that crate version numbers are specified in a TOML file.
Apart from that, it s just a matter of building and uploading the crate.
Easy!
Slightly more complicated is action-validator.
This is a Rust CLI tool which validates GitHub Actions and Workflows (how very meta) against a published JSON schema, to make sure you haven t got any syntax or structural errors.
As not everyone has a Rust toolchain on their local box, the release process helpfully build binaries for several common OSes and CPU architectures that people can download if they choose.
The release process in this case is somewhat larger, but not particularly complicated.
Almost half of it is actually scaffolding to build an experimental WASM/NPM build of the code, because someone seemed rather keen on that.
Moving away from Rust, and stepping up the meta another notch, we can take a look at the release process for git-version-bump itself, my Ruby library and associated CLI tool which started me down the Just Tag It Already rabbit hole many years ago.
In this case, since gemspecs are very amenable to programmatic definition, the release process is practically trivial.
Remove the boilerplate and workarounds for GitHub Actions bugs, and you re left with about three lines of actual commands.
These approaches can certainly scale to larger, more complicated processes.
I ve recently implemented annotated-tag-based releases in a proprietary software product, that produces Debian/Ubuntu, RedHat, and Windows packages, as well as Docker images, and it takes all of the information it needs from the annotated tag.
I m confident that this approach will successfully serve them as they expand out to build AMIs, GCP machine images, and whatever else they need in their release processes in the future.
Objection, Your Honour!
I can hear the howl of the but, actuallys coming over the horizon even as I type.
People have a lot of Big Feelings about why this release process won t work for them.
Rather than overload this article with them, I ve created a companion article that enumerates the objections I ve come across, and answers them.
I m also available for consulting if you d like a personalised, professional opinion on your specific circumstances.
DVD Bonus Feature: Pre-releases
Unless you re addicted to surprises, it s good to get early feedback about new features and bugfixes before they make it into an official, general-purpose release.
For this, you can t go past the pre-release.
The major blocker to widespread use of pre-releases is that cutting a release is usually a pain in the behind.
If you ve got to edit changelogs, and modify version numbers in a dozen places, then you re entirely justified in thinking that cutting a pre-release for a customer to test that bugfix that only occurs in their environment is too much of a hassle.
The thing is, once you ve got releases building from annotated tags, making pre-releases on every push to main becomes practically trivial.
This is mostly due to another fantastic and underused Git command: git describe.
How git describe works is, basically, that it finds the most recent commit that has an associated annotated tag, and then generates a string that contains that tag s name, plus the number of commits between that tag and the current commit, with the current commit s hash included, as a bonus.
That is, imagine that three commits ago, you created an annotated release tag named v4.2.0.
If you run git describe now, it will print out v4.2.0-3-g04f5a6f (assuming that the current commit s SHA starts with 04f5a6f).
You might be starting to see where this is going.
With a bit of light massaging (essentially, removing the leading v and replacing the -s with .s), that string can be converted into a version number which, in most sane environments, is considered newer than the official 4.2.0 release, but will be superceded by the next actual release (say, 4.2.1 or 4.3.0).
If you re already injecting version numbers into the release build process, injecting a slightly different version number is no work at all.
Then, you can easily build release artifacts for every commit to main, and make them available somewhere they won t get in the way of the official releases.
For example, in the proprietary product I mentioned previously, this involves uploading the Debian packages to a separate component (prerelease instead of main), so that users that want to opt-in to the prerelease channel simply modify their sources.list to change main to prerelease.
Management have been extremely pleased with the easy availability of pre-release packages; they ve been gleefully installing them willy-nilly for testing purposes since I rolled them out.
In fact, even while I ve been writing this article, I was asked to add some debug logging to help track down a particularly pernicious bug.
I added the few lines of code, committed, pushed, and went back to writing.
A few minutes later (next week s job is to cut that in-process time by at least half), the person who asked for the extra logging ran apt update; apt upgrade, which installed the newly-built package, and was able to progress in their debugging adventure.
Continuous Delivery: It s Not Just For Hipsters.
+1, Informative
Hopefully, this has spurred you to commit your immortal soul to the Church of the Annotated Tag.
You may tithe by buying me a refreshing beverage.
Alternately, if you re really keen to adopt more streamlined release management processes, I m available for consulting engagements.
-a (or --annotate, if you enjoy typing) to your git tag command, and WHAM! annotation achieved.
Releases, though, typically have unique and ever-increasing version numbers, which we want to encode in the tag name.
Rather than having to look at the existing tags and figure out the next version number ourselves, we can have software do the hard work for us.
Enter: git-version-bump.
This straightforward program takes one mandatory argument: major, minor, or patch, and bumps the corresponding version number component in line with Semantic Versioning principles.
If you pass it -n, it opens an editor for you to enter the release notes, and when you save out, the tag is automagically created with the appropriate name.
Because the program is called git-version-bump, you can call it as a git command: git version-bump.
Also, because version-bump is long and unwieldy, I have it aliased to vb, with the following entry in my ~/.gitconfig:
[alias]
vb = version-bump -n
Of course, you don t have to use git-version-bump if you don t want to (although why wouldn t you?).
The important thing is that the only step you take to go from here is our current codebase in main to everything as of this commit is version X.Y.Z of this software , is the creation of an annotated tag that records the version number being released, and the metadata that goes along with that release.
Step 2: Run git release
As I said earlier, I ve been using this release process for over a decade now.
So long, in fact, that when I started, GitHub Actions didn t exist, and so a lot of the things you d delegate to a CI runner these days had to be done locally, or in a more ad-hoc manner on a server somewhere.
This is why step 2 in the release process is run git release .
It s because historically, you can t do everything in a CI run.
Nowadays, most of my repositories have this in the .git/config:
[alias]
release = push --tags
Older repositories which, for one reason or another, haven t been updated to the new hawtness, have various other aliases defined, which run more specialised scripts (usually just rake release, for Ruby libraries), but they re slowly dying out.
The reason why I still have this alias, though, is that it standardises the release process.
Whether it s a Ruby gem, a Rust crate, a bunch of protobuf definitions, or whatever else, I run the same command to trigger a release going out.
It means I don t have to think about how I do it for this project, because every project does it exactly the same way.
The Wiring Behind the Button
It wasn t the button that was the problem. It was the miles of wiring, the hundreds of miles of cables, the circuits, the relays, the machinery. The engine was a massive, sprawling, complex, mind-bending nightmare of levers and dials and buttons and switches. You couldn t just slap a button on the wall and expect it to work. But there should be a button. A big, fat button that you could press and everything would be fine again. Just press it, and everything would be back to normal.
- Red Dwarf: Better Than Life
Once you ve accepted that your release process should be as simple as creating an annotated tag and running one command, you do need to consider what happens afterwards.
These days, with the near-universal availability of CI runners that can do anything you need in an isolated, reproducible environment, the work required to go from annotated tag to release artifacts can be scripted up and left to do its thing.
What that looks like, of course, will probably vary greatly depending on what you re releasing.
I can t really give universally-applicable guidance, since I don t know your situation.
All I can do is provide some of my open source work as inspirational examples.
For starters, let s look at a simple Rust crate I ve written, called strong-box.
It s a straightforward crate, that provides ergonomic and secure cryptographic functionality inspired by the likes of NaCl.
As it s just a crate, its release script is very straightforward.
Most of the complexity is working around Cargo s inelegant mandate that crate version numbers are specified in a TOML file.
Apart from that, it s just a matter of building and uploading the crate.
Easy!
Slightly more complicated is action-validator.
This is a Rust CLI tool which validates GitHub Actions and Workflows (how very meta) against a published JSON schema, to make sure you haven t got any syntax or structural errors.
As not everyone has a Rust toolchain on their local box, the release process helpfully build binaries for several common OSes and CPU architectures that people can download if they choose.
The release process in this case is somewhat larger, but not particularly complicated.
Almost half of it is actually scaffolding to build an experimental WASM/NPM build of the code, because someone seemed rather keen on that.
Moving away from Rust, and stepping up the meta another notch, we can take a look at the release process for git-version-bump itself, my Ruby library and associated CLI tool which started me down the Just Tag It Already rabbit hole many years ago.
In this case, since gemspecs are very amenable to programmatic definition, the release process is practically trivial.
Remove the boilerplate and workarounds for GitHub Actions bugs, and you re left with about three lines of actual commands.
These approaches can certainly scale to larger, more complicated processes.
I ve recently implemented annotated-tag-based releases in a proprietary software product, that produces Debian/Ubuntu, RedHat, and Windows packages, as well as Docker images, and it takes all of the information it needs from the annotated tag.
I m confident that this approach will successfully serve them as they expand out to build AMIs, GCP machine images, and whatever else they need in their release processes in the future.
Objection, Your Honour!
I can hear the howl of the but, actuallys coming over the horizon even as I type.
People have a lot of Big Feelings about why this release process won t work for them.
Rather than overload this article with them, I ve created a companion article that enumerates the objections I ve come across, and answers them.
I m also available for consulting if you d like a personalised, professional opinion on your specific circumstances.
DVD Bonus Feature: Pre-releases
Unless you re addicted to surprises, it s good to get early feedback about new features and bugfixes before they make it into an official, general-purpose release.
For this, you can t go past the pre-release.
The major blocker to widespread use of pre-releases is that cutting a release is usually a pain in the behind.
If you ve got to edit changelogs, and modify version numbers in a dozen places, then you re entirely justified in thinking that cutting a pre-release for a customer to test that bugfix that only occurs in their environment is too much of a hassle.
The thing is, once you ve got releases building from annotated tags, making pre-releases on every push to main becomes practically trivial.
This is mostly due to another fantastic and underused Git command: git describe.
How git describe works is, basically, that it finds the most recent commit that has an associated annotated tag, and then generates a string that contains that tag s name, plus the number of commits between that tag and the current commit, with the current commit s hash included, as a bonus.
That is, imagine that three commits ago, you created an annotated release tag named v4.2.0.
If you run git describe now, it will print out v4.2.0-3-g04f5a6f (assuming that the current commit s SHA starts with 04f5a6f).
You might be starting to see where this is going.
With a bit of light massaging (essentially, removing the leading v and replacing the -s with .s), that string can be converted into a version number which, in most sane environments, is considered newer than the official 4.2.0 release, but will be superceded by the next actual release (say, 4.2.1 or 4.3.0).
If you re already injecting version numbers into the release build process, injecting a slightly different version number is no work at all.
Then, you can easily build release artifacts for every commit to main, and make them available somewhere they won t get in the way of the official releases.
For example, in the proprietary product I mentioned previously, this involves uploading the Debian packages to a separate component (prerelease instead of main), so that users that want to opt-in to the prerelease channel simply modify their sources.list to change main to prerelease.
Management have been extremely pleased with the easy availability of pre-release packages; they ve been gleefully installing them willy-nilly for testing purposes since I rolled them out.
In fact, even while I ve been writing this article, I was asked to add some debug logging to help track down a particularly pernicious bug.
I added the few lines of code, committed, pushed, and went back to writing.
A few minutes later (next week s job is to cut that in-process time by at least half), the person who asked for the extra logging ran apt update; apt upgrade, which installed the newly-built package, and was able to progress in their debugging adventure.
Continuous Delivery: It s Not Just For Hipsters.
+1, Informative
Hopefully, this has spurred you to commit your immortal soul to the Church of the Annotated Tag.
You may tithe by buying me a refreshing beverage.
Alternately, if you re really keen to adopt more streamlined release management processes, I m available for consulting engagements.
It wasn t the button that was the problem. It was the miles of wiring, the hundreds of miles of cables, the circuits, the relays, the machinery. The engine was a massive, sprawling, complex, mind-bending nightmare of levers and dials and buttons and switches. You couldn t just slap a button on the wall and expect it to work. But there should be a button. A big, fat button that you could press and everything would be fine again. Just press it, and everything would be back to normal.Once you ve accepted that your release process should be as simple as creating an annotated tag and running one command, you do need to consider what happens afterwards. These days, with the near-universal availability of CI runners that can do anything you need in an isolated, reproducible environment, the work required to go from annotated tag to release artifacts can be scripted up and left to do its thing. What that looks like, of course, will probably vary greatly depending on what you re releasing. I can t really give universally-applicable guidance, since I don t know your situation. All I can do is provide some of my open source work as inspirational examples. For starters, let s look at a simple Rust crate I ve written, called
- Red Dwarf: Better Than Life
strong-box.
It s a straightforward crate, that provides ergonomic and secure cryptographic functionality inspired by the likes of NaCl.
As it s just a crate, its release script is very straightforward.
Most of the complexity is working around Cargo s inelegant mandate that crate version numbers are specified in a TOML file.
Apart from that, it s just a matter of building and uploading the crate.
Easy!
Slightly more complicated is action-validator.
This is a Rust CLI tool which validates GitHub Actions and Workflows (how very meta) against a published JSON schema, to make sure you haven t got any syntax or structural errors.
As not everyone has a Rust toolchain on their local box, the release process helpfully build binaries for several common OSes and CPU architectures that people can download if they choose.
The release process in this case is somewhat larger, but not particularly complicated.
Almost half of it is actually scaffolding to build an experimental WASM/NPM build of the code, because someone seemed rather keen on that.
Moving away from Rust, and stepping up the meta another notch, we can take a look at the release process for git-version-bump itself, my Ruby library and associated CLI tool which started me down the Just Tag It Already rabbit hole many years ago.
In this case, since gemspecs are very amenable to programmatic definition, the release process is practically trivial.
Remove the boilerplate and workarounds for GitHub Actions bugs, and you re left with about three lines of actual commands.
These approaches can certainly scale to larger, more complicated processes.
I ve recently implemented annotated-tag-based releases in a proprietary software product, that produces Debian/Ubuntu, RedHat, and Windows packages, as well as Docker images, and it takes all of the information it needs from the annotated tag.
I m confident that this approach will successfully serve them as they expand out to build AMIs, GCP machine images, and whatever else they need in their release processes in the future.
Objection, Your Honour!
I can hear the howl of the but, actuallys coming over the horizon even as I type.
People have a lot of Big Feelings about why this release process won t work for them.
Rather than overload this article with them, I ve created a companion article that enumerates the objections I ve come across, and answers them.
I m also available for consulting if you d like a personalised, professional opinion on your specific circumstances.
DVD Bonus Feature: Pre-releases
Unless you re addicted to surprises, it s good to get early feedback about new features and bugfixes before they make it into an official, general-purpose release.
For this, you can t go past the pre-release.
The major blocker to widespread use of pre-releases is that cutting a release is usually a pain in the behind.
If you ve got to edit changelogs, and modify version numbers in a dozen places, then you re entirely justified in thinking that cutting a pre-release for a customer to test that bugfix that only occurs in their environment is too much of a hassle.
The thing is, once you ve got releases building from annotated tags, making pre-releases on every push to main becomes practically trivial.
This is mostly due to another fantastic and underused Git command: git describe.
How git describe works is, basically, that it finds the most recent commit that has an associated annotated tag, and then generates a string that contains that tag s name, plus the number of commits between that tag and the current commit, with the current commit s hash included, as a bonus.
That is, imagine that three commits ago, you created an annotated release tag named v4.2.0.
If you run git describe now, it will print out v4.2.0-3-g04f5a6f (assuming that the current commit s SHA starts with 04f5a6f).
You might be starting to see where this is going.
With a bit of light massaging (essentially, removing the leading v and replacing the -s with .s), that string can be converted into a version number which, in most sane environments, is considered newer than the official 4.2.0 release, but will be superceded by the next actual release (say, 4.2.1 or 4.3.0).
If you re already injecting version numbers into the release build process, injecting a slightly different version number is no work at all.
Then, you can easily build release artifacts for every commit to main, and make them available somewhere they won t get in the way of the official releases.
For example, in the proprietary product I mentioned previously, this involves uploading the Debian packages to a separate component (prerelease instead of main), so that users that want to opt-in to the prerelease channel simply modify their sources.list to change main to prerelease.
Management have been extremely pleased with the easy availability of pre-release packages; they ve been gleefully installing them willy-nilly for testing purposes since I rolled them out.
In fact, even while I ve been writing this article, I was asked to add some debug logging to help track down a particularly pernicious bug.
I added the few lines of code, committed, pushed, and went back to writing.
A few minutes later (next week s job is to cut that in-process time by at least half), the person who asked for the extra logging ran apt update; apt upgrade, which installed the newly-built package, and was able to progress in their debugging adventure.
Continuous Delivery: It s Not Just For Hipsters.
+1, Informative
Hopefully, this has spurred you to commit your immortal soul to the Church of the Annotated Tag.
You may tithe by buying me a refreshing beverage.
Alternately, if you re really keen to adopt more streamlined release management processes, I m available for consulting engagements.
main becomes practically trivial.
This is mostly due to another fantastic and underused Git command: git describe.
How git describe works is, basically, that it finds the most recent commit that has an associated annotated tag, and then generates a string that contains that tag s name, plus the number of commits between that tag and the current commit, with the current commit s hash included, as a bonus.
That is, imagine that three commits ago, you created an annotated release tag named v4.2.0.
If you run git describe now, it will print out v4.2.0-3-g04f5a6f (assuming that the current commit s SHA starts with 04f5a6f).
You might be starting to see where this is going.
With a bit of light massaging (essentially, removing the leading v and replacing the -s with .s), that string can be converted into a version number which, in most sane environments, is considered newer than the official 4.2.0 release, but will be superceded by the next actual release (say, 4.2.1 or 4.3.0).
If you re already injecting version numbers into the release build process, injecting a slightly different version number is no work at all.
Then, you can easily build release artifacts for every commit to main, and make them available somewhere they won t get in the way of the official releases.
For example, in the proprietary product I mentioned previously, this involves uploading the Debian packages to a separate component (prerelease instead of main), so that users that want to opt-in to the prerelease channel simply modify their sources.list to change main to prerelease.
Management have been extremely pleased with the easy availability of pre-release packages; they ve been gleefully installing them willy-nilly for testing purposes since I rolled them out.
In fact, even while I ve been writing this article, I was asked to add some debug logging to help track down a particularly pernicious bug.
I added the few lines of code, committed, pushed, and went back to writing.
A few minutes later (next week s job is to cut that in-process time by at least half), the person who asked for the extra logging ran apt update; apt upgrade, which installed the newly-built package, and was able to progress in their debugging adventure.
Continuous Delivery: It s Not Just For Hipsters.
And all of this is because I can read, write, articulate. Perhaps many of them may not even have a voice or a platform.
Even to get this temporary progressive disability certificate there is more than 4 months of running from one place to the other, 4 months of culmination of work. This I can share and tell from my experience, who knows how much else others might have suffered for the same. In my case a review will happen after 5 years, in most other cases they have given only 1 year. Of course, this does justify people s jobs and perhaps partly it may be due to that. Such are times where I really miss that I am unable to hear otherwise could have fleshed out lot more other people s sufferings.
And just so people know/understand this is happening in the heart of the city whose population easily exceeds 6 million plus and is supposed to be a progressive city. I do appreciate and understand the difficulties that the doctors are placed under.
As part of the continuing work to 
