Making deploys boring

The best compliment a deployment process can get is that nobody talks about it. No war room, no person who is the only one who knows the steps, no Friday afternoon rule.

Getting there is mostly a process problem wearing a tooling costume. You can have excellent pipelines and still take two weeks to ship, because the pipeline was never the slow part.

What is actually slow

When a release takes a long time, the delay is rarely compute. It is usually one of these:

• Waiting for a person to manually verify something a machine could check.
• Waiting for a meeting where a group decides whether the release is safe.
• Waiting for an environment that somebody else is currently using.
• Re-doing work because a problem surfaced after the expensive part of the process.

None of those get faster by optimizing build times. They get faster by moving checks earlier and by replacing human judgment with an automated gate wherever the judgment was mechanical to begin with.

The order of the stages is the design

A pipeline is a sequence of filters, and the useful principle is that the cheapest check that can reject a change should run first.

The billing service I wrote about earlier ships through GitHub Actions in this order:

1. Ruff lint. Takes seconds and catches the class of problem that would otherwise waste a reviewer's attention.
2. The test suite, against a real PostgreSQL service container rather than a mock, so the environment resembles production in the ways that matter.
3. Build the Docker image, which is only worth doing for a change that already passed the first two.
4. Push the image to ECR.
5. Deploy to ECS Fargate.

Every stage is more expensive than the one before it. A change that is going to fail should fail in stage one, thirty seconds in, rather than after a five minute build.

The other property worth having is that the pipeline produces the artifact that gets deployed. If the thing you tested and the thing you shipped were built separately, you have two artifacts and a theory that they match.

Automate the decision, not just the steps

This is the part that took me longest to understand.

Scripting the deploy steps makes the deploy fast. It does not make releases frequent, because the slow part was the decision, and a script does not make decisions.

Releases get frequent when the pipeline's verdict is trusted enough to be the decision. Green means ship. That only works if the suite is good enough that people believe it, which is why test quality and deploy frequency are the same problem viewed from different ends. A flaky suite forces a human back into the loop, and now you have a meeting again.

So the sequence is: make the tests trustworthy, then let them gate, and then the schedule moves. I have watched the other order in action, and it produces an automated pipeline that everyone quietly overrides.

Make going back cheap

The other half of shipping confidently is knowing you can undo it.

If rolling back means a redeploy of the previous image, it takes as long as a deploy and everyone knows the procedure. If rolling back means reversing a database migration under pressure, nobody wants to ship on a Thursday.

That is why backward-compatible schema changes are worth the extra step of deploying them separately from the code that uses them. It is more work in the moment and it keeps the rollback path simple, which is what you are actually buying.

A deploy you can reverse in two minutes changes how the team behaves long before you ever need to reverse one.