Cruxly › Cloud and DevOps › Cloud & DevOps: CI/CD Pipelines

Deployment Strategies and the Rollback Plan That Survives a Real Incident

blue-green, canary, and what 'roll back' actually leaves behind

Prompt: a deploy just shipped a bug to production. Design the rollback mechanics for a service that also shipped a database migration alongside the code in this same release. "Revert the deployment" must be a precise, tested procedure — not a hope that the previous container image plus the current (already-migrated) database will simply work together.

30 minutes first. The naive plan assumes rolling back the application automatically un-does everything the release changed. It doesn't — the database doesn't roll back just because the container did.

Blue-green and canary solve different problems

Blue-green deploys a complete second environment, then switches traffic over in one step — atomically when the switch is at a load balancer, router, or Kubernetes Service selector. (A DNS cutover is not atomic: resolvers and clients cache the old answer, and existing keep-alive connections keep talking to the old environment, so a DNS switch drains gradually and unpredictably.) Rollback is switching traffic back to the still-running old environment, which is fast and simple, at the cost of running two full environments simultaneously (double the infrastructure cost during the deploy window) and being an all-or-nothing traffic switch with no gradual exposure.

Canary releases the new version to a small percentage of real traffic first, watches real production signals (error rate, latency, business metrics) on that slice, and only ramps up if those signals hold — catching a bug that only manifests under real production traffic patterns (a specific user segment, a specific data shape) that synthetic pre-production testing never exercised, at the cost of a more complex rollout mechanism and a bug that does affect some real users during the canary window, even if a small percentage of them.

They are not mutually exclusive, and picking one over the other is a question about what kind of bug you're most worried about, not a question of which is "better" in the abstract: blue-green protects against "this new version is entirely broken," fast and cleanly; canary protects against "this new version is subtly broken in a way only real traffic at scale reveals," slower but with a much smaller blast radius while that's being determined.

The database is the part that doesn't roll back for free

A schema migration that adds a column, backfills it, then a later release makes code depend on that column being populated, cannot be safely rolled back by redeploying the old container alone — the old code doesn't know about the new column and will simply ignore it (usually survivable), but if a subsequent migration in the same release dropped a column the old code still reads, rolling back the container without rolling back the schema produces a hard failure: the old code queries a column that no longer exists.

The durable pattern is expand/contract, deploy old-code-compatible first. Add the new column (expand) in one release, deploy code that writes to both old and new locations, verify, and only in a later, separate release remove the old column (contract) once you're certain no rollback to the old code path will ever be needed again. This means any single release's rollback only ever needs to revert code, never schema, because the schema change that release depended on was already made backward-compatible in a prior step.

Two details interviewers probe:

• The rollback window leaves gaps. If you roll back to old code after release 2, any rows written by the old code during that window don't have the new column filled in. Before a later release adds NOT NULL or reads only from the new column, run a backfill — or the constraint fails.
• Large tables need online schema changes. On a big table, a plain ALTER TABLE can take a lock that blocks every write for minutes. Use online tools — gh-ost or pt-online-schema-change for MySQL, CREATE INDEX CONCURRENTLY in Postgres — and set a lock timeout so a migration waiting behind a long transaction fails fast instead of queueing all traffic behind it.

A migration that is NOT naturally reversible needs an explicit, separately-tested down-migration, or an explicit decision that rollback for this specific release means "roll forward with a fix," not "roll back." Discovering mid-incident that the "rollback plan" was never actually tested against the post-migration schema is how a five-minute bug becomes a multi-hour incident — the down-migration script, if one is needed at all, has to be exercised in a rehearsal before the release ships, not written for the first time during the incident itself.

GitOps and progressive delivery: automating the rollout and the rollback

GitOps makes Git the source of truth for what's running. Instead of CI pushing to the cluster, an agent inside the cluster — Argo CD or Flux — pulls the desired state from a Git repo and continuously reconciles the cluster to match.

• CI never needs cluster credentials. CI builds the image and opens a commit or PR that bumps the image digest in the config repo; the in-cluster agent does the deploy. That removes the most powerful credential from the pipeline entirely.
• Drift heals itself. A manual kubectl edit gets reverted to what Git says (when self-heal is enabled), and the drift is visible.
• Rollback is git revert. The previous known-good state is a commit, with review and history attached.

Progressive delivery automates the canary decision. A manual canary depends on someone watching dashboards. Argo Rollouts or Flagger instead shift traffic in steps, run an analysis at each step — comparing the canary's error rate and latency against the stable version, using queries you define — and promote or roll back automatically.

Feature flags decouple deploy from release. Ship the code dark, then turn the feature on for 1% of users, an internal cohort, or one region — and turn it off in seconds without a deploy. Rollback for a logic bug becomes a flag flip; the deploy pipeline only has to handle crashes and infrastructure problems.

Measure the delivery system itself with the DORA metrics: deployment frequency, lead time for changes, change fail rate, and failed deployment recovery time — which replaced the older MTTR metric in 2023, because "time to restore" lumped deploy failures together with outages that had nothing to do with a change.

How to say it in the interview

"The hard part of a rollback is never the application container - registries keep old images, so redeploying the previous one is trivial. The hard part is that a database migration is usually not naturally reversible, and the old application code was never tested against the NEW, already-migrated schema. A rollback plan that only covers the container is a plan for half the actual problem."

Checklist

• Distinguished blue-green (fast, atomic, all-or-nothing) from canary (gradual, smaller blast radius) by what failure each catches
• Recognized a database migration is usually not naturally reversible
• Used expand/contract to spread a schema change across releases
• Tested the down-migration (if one exists) before the release ships, not during the incident
• Made an explicit decision, per release, about whether rollback means 'revert' or 'roll forward with a fix'
• Used GitOps pull-based deploys so CI holds no cluster credentials, with rollback as a git revert
• Automated canary analysis and used feature flags to separate deploy from release
• Backfilled rows written during a rollback window and used online schema-change tools on large tables

Related lessons

• Build Once, Promote Everywhere: Designing a Pipeline Without Environment Drift
• Pipeline Security: The CI System as Its Own Attack Surface