Rolling back safely

Sooner or later a change goes live that should not have. What matters then is how fast you can undo it, and whether undoing it is safe.

Undo with history, not against it

The safe rollback is a new commit that reverses the old one. The history stays honest and the pipeline publishes the previous state the same way it published the change.

git revert <commit>
git push

Rewriting history to hide a mistake feels tidy, but it breaks everyone else’s copy and removes the record of what happened.

Separate the layers

On this site there are three things that can be rolled back separately: the content, the container image, and the container configuration. Knowing which layer caused the problem tells you which one to revert, and keeps the others untouched.

Pin what you roll back to

A rollback to “the previous version” only works if the previous version is still there and unambiguous. That is why images are referenced by digest: it names exactly one set of bytes.

Practise once

A rollback you have never tried is a plan, not a capability. Do it once while nothing is on fire, and note what surprised you.