Planning a Zero-Downtime Migration: Compatibility, Cutover and Rollback

Planned downtime used to be normal: announce a maintenance window, stop the system, migrate, start it again. For many products that is no longer acceptable. Customers in different time zones use the system around the clock, integrations expect APIs to respond, and a failed migration discovered at 3 a.m. can mean lost orders and a long recovery.

A zero-downtime migration changes a running system — its infrastructure, database, schema or a whole component — while users keep working. The techniques are well known: blue-green and canary releases, backward-compatible changes, change data capture and feature flags. What teams usually lack is not the vocabulary but a plan that defines success, sequences the work and says exactly when to stop.

This guide is that plan. It uses one worked example throughout and ends with a runbook you can adapt.

The worked example

An order-management service stores orders in a single relational database. The team needs to move the orders data to a new managed PostgreSQL database and, at the same time, change the schema: the free-text delivery_address column becomes structured fields (street, city, postcode, country). The web application, a mobile app, a nightly reporting job and two partner integrations all read or write orders. Peak traffic is around 40 writes per second.

The constraints: checkout must keep working, no order may be lost or duplicated, and partner integrations must not change their requests during the migration.

Step 1: Define “zero downtime” in measurable terms

“Zero downtime” means different things to different people. For one team it means the homepage loads; for another it means no single request fails. Write a definition the business agrees with and the team can test. Google’s SRE guidance on service level objectives offers a useful vocabulary: a service level indicator (SLI) is a measured quantity such as error rate or latency, and a service level objective (SLO) is a target for it.

For the example, the migration definition is:

This table becomes the backbone of every go/no-go decision later. If you cannot measure an item today, add the monitoring before the migration starts.

Step 2: Map dependencies and compatibility

Most migration failures come from something nobody listed. Build an inventory of everything that touches the data:

• Readers and writers: services, jobs, scripts, reports, admin tools and partner integrations. Search code, connection strings and database logs; ask operations about manual scripts.
• Implicit contracts: column names used in reports, sort orders, default values, triggers, stored procedures, time zones and character encodings.
• Caches and copies: search indexes, caches, data warehouse extracts and backups that assume the old structure.
• Operational dependencies: monitoring queries, alerts, dashboards and runbooks that reference the old database.

For each dependency, record whether it reads, writes or both, how it will be migrated, and who owns the test. In the example, the inventory found a forgotten finance export that parsed delivery_address with a regular expression. Finding it in week one cost an hour; finding it after cutover would have cost a month-end close.

Step 3: Make every change backward compatible

The core technique is parallel change, also called expand and contract, described by Danilo Sato on Martin Fowler’s site. It has three phases:

1. Expand: add the new structure alongside the old one, so both work.
2. Migrate: move clients and data to the new structure, gradually.
3. Contract: remove the old structure only when nothing uses it.

In the example, that means the new database contains both the structured address fields and, temporarily, the original delivery_address text. The application writes both representations during the transition. The old column is removed only in the contract phase, weeks later.

Changes that remove your rollback path

Some changes are hard or impossible to undo while traffic flows. Schedule them only in the contract phase, after the new path is proven:

• Dropping or renaming columns or tables that any client still uses.
• Changing a column’s type or meaning in place.
• Adding constraints (NOT NULL, unique, foreign keys) before existing data has been cleaned and verified.
• Transforming data destructively, for example splitting the address text and discarding the original.

Schema operations on large tables

Some schema operations lock tables. On PostgreSQL, for example, a standard CREATE INDEX blocks writes until it finishes, while CREATE INDEX CONCURRENTLY avoids blocking writes but takes longer, cannot run inside a transaction block and can leave an invalid index behind if it fails. Plan large operations accordingly:

Check the documentation for your database engine and version: lock behaviour differs between engines and between versions of the same engine.

Step 4: Move the data — backfill, sync and reconcile

Database migration is usually the hardest part because data, unlike code, cannot simply be redeployed. The work has three parts.

Backfill

Copy existing data into the new store in batches. Batch by primary key range, throttle to protect production, record progress so the job can resume, and transform data (here, parsing addresses) in a way that is logged and repeatable. Keep the original text so you can re-run the transformation if the parser improves.

Ongoing synchronisation

While backfill runs and until cutover completes, new writes continue to arrive in the old store. There are two main options:

The example team chose CDC. Tools such as Debezium read the database’s change log — the binlog for MySQL or a logical replication stream for PostgreSQL — and publish row-level change events, commonly through Kafka, that a consumer applies to the new store. Monitor replication lag as a first-class metric; cutover should not start while lag exceeds an agreed threshold.

Reconciliation

Do not switch traffic until you can show the stores agree. Use several layers of checks, because each catches different problems:

Shadow reads

Before switching real reads, send a copy of read requests to the new store in the background and compare responses without returning them to users. Shadow reads reveal query differences, missing indexes and latency problems under real traffic, at no risk to users.

Step 5: Cut over in stages

With data in sync and reconciliation clean, switch traffic gradually. Use a feature flag or routing configuration that can be changed in seconds without a deployment.

Switching reads before writes means the riskiest change happens last, after the new store has proven itself under real read traffic. Keeping a reverse sync from the new store back to the old one after the write switch preserves a rollback path.

Deployment strategies such as blue-green, canary and rolling releases apply to the application changes along the way. Blue-green switches all traffic between two environments at once; canary releases expose a new version to a small share of traffic first; rolling updates replace instances gradually. None of them, on its own, solves the data problem — which is why data synchronisation and reconciliation come first in this plan.

Step 6: Write rollback criteria and rehearse them

Every stage needs an explicit rollback trigger and an owner who can make the call without a meeting.

Rehearsal

Run the full sequence — backfill, sync, staged cutover and rollback — in a production-like environment with realistic data volumes. Time each step. In the example, the first rehearsal showed that switching writes back required restarting a background worker that cached the connection string; the fix took ten minutes in rehearsal and would have extended a real incident by much longer.

Testing strategy before the first stage

Rehearsal tests the procedure. You also need tests that show the new path behaves correctly. From a QA perspective, the example migration used the following test plan, run in a production-like environment with a copy of anonymised production data.

Pay particular attention to idempotency. During cutover and rollback, the same change may be applied more than once — for example when a CDC consumer restarts from its last committed position. The new store should treat a repeated change as a no-op, typically by keying writes on the order’s primary key or idempotency key and applying updates only when the source version is newer.

Caches, sessions and background jobs

Databases are not the only place where state lives. These components frequently cause problems during cutover:

• Application caches. If the cache holds data in the old format, reads from the new store and cached reads can disagree. Version cache keys during the transition, or flush the relevant keys at each stage change.
• Connection pools and configuration. Long-lived processes may keep connections to the old store after a flag changes. Confirm that every service picks up the switch, including workers that only restart occasionally.
• Background and scheduled jobs. Nightly reports, exports and clean-up tasks may run between stages and read or write the wrong store. List them in the dependency inventory and decide explicitly which store each job uses at each stage.
• Search indexes and downstream copies. If a search index or data warehouse is fed from the old database, plan when its source switches and reconcile it as well.
• DNS and endpoint caching. If the switch involves changing hostnames, reduce DNS time-to-live values well in advance so clients pick up changes quickly in both directions.

Step 7: Communicate and monitor

• Before: tell support, partner managers and stakeholders what will happen, what customers might notice and how to escalate.
• During each stage: a dedicated channel, a named migration lead and a dashboard showing the SLO table from Step 1, replication lag and reconciliation status.
• After cutover: keep heightened monitoring through at least one full business cycle, including month-end or other scheduled jobs that run rarely.

Common mistakes

• Starting without a measurable definition. “No downtime” without SLIs makes every decision a debate.
• Treating deployment as migration. A blue-green switch moves application traffic; it does not move or verify data.
• Reconciling once. A single comparison before cutover misses changes made afterwards. Reconcile continuously.
• Contracting too early. Dropping the old column on cutover day removes the rollback path just when you might need it.
• Ignoring the long tail of readers. Reports, exports and manual scripts break quietly. The dependency inventory is not optional.
• Unrehearsed rollback. Most rollback problems are mundane — cached connections, DNS TTLs, missing permissions — and only rehearsal finds them.

Runbook template

Copy and adapt this outline for your migration:

1. Definition: SLI and target table signed off by product, engineering and QA.
2. Inventory: every reader, writer, cache, job, report and integration, with owner and test.
3. Expand: new schema and infrastructure deployed; application writes both representations or CDC configured.
4. Backfill: batch plan, throttling, resume procedure, transformation log.
5. Sync: CDC or dual-write design, lag monitoring and alert thresholds.
6. Reconcile: counts, checksums, sampled comparisons, invariants and edge cases, running continuously.
7. Shadow reads: comparison method and acceptable mismatch rate.
8. Cutover stages: traffic percentages, observation windows and go criteria per stage.
9. Rollback: triggers, thresholds, decision owners and rehearsed steps with timings.
10. Communication: stakeholder list, channel, status updates and escalation path.
11. Contract: removal steps, scheduled only after sign-off and a full business cycle.
12. Close-out: retrospective, updated documentation and decommissioning of old resources.

Frequently asked questions

Is true zero downtime always worth the effort?

Not always. For internal tools with predictable usage, a short announced window may be cheaper and safer. Use the techniques here when users, revenue or integrations depend on the system being available.

How long should the migration take?

The data move itself may take hours or days; the full expand-migrate-contract cycle often takes weeks, mainly because observation windows and the contract phase should not be rushed.

Should we use dual writes or change data capture?

CDC is usually safer for keeping two stores in sync because it captures every committed change in order without application changes. Dual writes can work for simple cases but need careful handling of partial failures.

Who should own the go/no-go decision?

Name individuals in advance for each type of trigger — engineering for performance, QA and the data owner for integrity, the product owner for customer and partner impact — and give them authority to roll back without a meeting.