T

Nobody in the company can explain why step 12 exists. It was added after an incident in 2023 and the engineer who wrote it has left. It costs about ten minutes each release and it will still be there next year, because deleting it requires somebody to be the person who deleted it.

When deployment takes too long at a startup, the pipeline is almost never the cause. The ceremony grew because each piece of it was a reasonable answer to something that actually went wrong, and it sustains itself through a loop nobody designed: fear makes batches larger, larger batches make incidents worse, worse incidents justify more gates, and more gates slow releases until the next batch is larger still. Breaking that needs three specific capabilities, and for most teams the one genuinely blocking progress is database migrations rather than anything in CI.

Why do deploys get slower every quarter?

The loop is worth naming out loud in a meeting, because it is much easier to argue against something with a name.

Call it the deploy ratchet. Every gate in that checklist was added in response to a real incident, by a competent person, with a sound local argument. A review would have caught it, so add a review. A staging soak would have caught it, so add a soak. Each addition is individually correct and the aggregate is a release process that takes a day and four people.

What makes it a ratchet rather than a dial is the asymmetry in who carries the risk. Adding a step is careful work that makes you look diligent. Removing one means owning the next incident personally, in front of everyone, with a git blame pointing at your name. Nobody has ever been promoted for deleting step 12.

The mechanism connecting fear to actual risk runs through batch size. Ship three changes and something breaks, you know which one within minutes. Ship forty changes and something breaks, you are bisecting under pressure while customers are affected and four people are watching, and the diagnosis takes an hour instead of five minutes. Time to identify a bad change scales with the number of changes shipped alongside it, and the stress scales considerably faster than that.

This is the part that reverses most people's intuition. Accelerate, published by Forsgren, Humble and Kim in 2018, along with the DORA research programme that produced it, has repeatedly found that teams deploying most frequently also have the lowest change failure rate. More deploys correlating with fewer failures sounds backwards until you account for batch size, at which point it is close to inevitable. Elite performers ship small, so their failures are small and quickly attributed. Low performers ship quarterly, so every release is an event with an unknown number of interacting changes in it.

The team that deploys twice a day is not braver. They have arranged things so that being wrong is cheap.

What has to be true before a deploy is boring?

Rollback comes first, and the standard is higher than having one documented. It has to be something you have performed, ideally within the last quarter, ideally by accident. A rollback procedure that has never executed is in exactly the same category as a backup nobody has restored: a plausible document about an untested capability.

Rollbacks fail in production for one predictable reason, which is that the schema moved. The previous container image expects the previous schema, and the migration already ran. That single coupling is why so many teams conclude rollback is unsafe and reach for a maintenance window instead.

Then there is the question of whether you trust the test suite enough not to read it. If anyone on the team scans CI output looking for which failures are "the usual ones", the suite has stopped being a signal and become decoration. Flaky tests are worse than absent tests, because they train an entire team to treat red as noise, and that training generalises to the run where red meant something. Quarantine a flaky test the day it flakes rather than adding a retry, since a retry preserves the appearance of coverage while removing the information.

The last of the three is a deploy that does not drop requests, and this one produces deploy fear without anyone ever articulating it. If each release throws a visible spike of 502s, everybody learns that deploying hurts customers, and the rational response is to deploy less often in bigger chunks. The cause is usually signal handling. Kubernetes sends SIGTERM, waits out terminationGracePeriodSeconds, then sends SIGKILL, and a process that exits immediately on SIGTERM abandons every request in flight. There is a further race that catches teams who have already fixed the obvious part, because endpoint removal from the Service happens asynchronously and traffic can still arrive after the signal, which is covered properly in the mechanics of graceful shutdown on Kubernetes.

Underpinning all three is knowing within minutes whether the deploy you just ran was a bad one. Error rate and latency broken down by release version is the minimum, and getting there is a subset of logs, metrics and traces. Without that signal, the checklist is doing the job that instrumentation should be doing, which is roughly why it has forty-one steps.

Why are migrations the real blocker?

Schema changes are the only part of a deploy that redeploying the previous artifact cannot undo.

Code is reversible by definition: previous image, previous commit, done in ninety seconds. A dropped column is not reversible, because the data is gone. Teams notice this coupling, conclude correctly that the pair is risky, and then draw the wrong conclusion, which is that deploys need a window. The right conclusion is that schema changes and code changes should stop travelling together.

Expand and contract is how that works, and it is old, unglamorous and effective. Add the new column as nullable and deploy, while the running code ignores it entirely. Start writing to both old and new, then backfill historical rows in the background. Switch reads across once the backfill completes. Remove the old column in a separate deploy, days later, once you are confident nothing reads it. Every one of those steps is independently deployable and independently reversible, and at no point does the database and the code have to change in the same instant.

The reason teams skip it is honest: a column rename becomes three deploys across a week instead of one deploy in an afternoon. That trade looks bad right up until the first time a rollback fails at 11pm, after which nobody argues about it again.

Two Postgres specifics are worth knowing because they cause the outages that create the ceremony in the first place. Adding a column with a non-null default rewrote the entire table in older versions, and has been a metadata-only operation since Postgres 11 in October 2018, so the folklore about it being dangerous outlives the danger in most shops. Building an index without CONCURRENTLY, on the other hand, still blocks writes for the duration, and on a large table that is a self-inflicted outage during what everyone believed was a routine release.

Sequencing schema work so the system keeps serving traffic throughout is the same discipline required to change a system without stopping the business, at a smaller scale and with the same rules.

When is the ceremony actually correct?

Some of the time, and the distinguishing test is whether a bad change is recoverable in minutes.

Regulated medical software, payments and settlement infrastructure, firmware, and anything shipping into a customer's own data centre are all cases where the cost of a wrong deploy is unbounded or unrecoverable. A patient safety event does not roll back. A settlement run does not roll back. Software installed on a customer's premises cannot be redeployed forty times a day regardless of how good your pipeline is. In those environments the checklist is correct engineering rather than accumulated fear, and the research on deployment frequency is measuring a different game with different stakes.

The honest version for everyone else is that most teams invoking this exemption are not in one of those categories. Handling other people's money is not the same as being payments infrastructure, and a B2B tool with an enterprise logo on the homepage is still a web application where a bad deploy costs eleven minutes.

Worth declaring an interest here, since it points away from work we would enjoy selling. Deploy pipeline projects are a clean, well-scoped, satisfying engagement to quote for, and they are frequently not the constraint. The constraint is usually migration discipline and a test suite nobody trusts, which is slower and much less visible work that founders do not wake up wanting to buy. A team that fixes the pipeline while leaving those two alone gets a faster route to the same fear.

Where the deploy process is slow because the architecture cannot tolerate small changes, that is a different problem sitting inside the broader question of what your architecture is costing you commercially.

Questions engineering leads ask about release friction

Why does our deployment take so long? Usually accumulated approval steps rather than slow tooling. Time the pipeline itself, then separately time the humans waiting around it, because the second number is normally several times the first and it is the one nobody measures. Teams who do this exercise generally find that the actual compute finishes in under fifteen minutes and the release still takes most of a day.

Is it bad to deploy on a Friday? The Friday rule is a symptom rather than a policy. If Friday deploys are unsafe, deploys are unsafe every day and you have simply arranged to discover it on Monday, when the batch is larger.

How often should a startup deploy? Daily at minimum and ideally on merge, though the frequency matters far less than the batch size it implies.

How do we roll back safely when a migration has already run? Stop shipping schema and code changes together. Expand and contract lets you deploy the schema change first, keep it backward compatible, and roll code back and forth freely against it.

Should we add an approval step after an incident? Rarely, and check first whether the gate would genuinely have caught it. Gates accumulate permanently while the incidents that justified them are forgotten, so the honest test is whether you would also delete an existing one to make room.

Version 15 of that checklist will be longer than version 14. It is the only prediction in this article I would make with real confidence, and the reason has nothing to do with engineering: adding a step is visible diligence, and removing one is personal exposure, so the document can only grow until somebody decides that asymmetry is the actual problem.