'Backup Succeeded': Six Steps to Actually Verify Service RecoveryNote · digest for November 19
'Backup Succeeded': Six Steps to Actually Verify Service Recovery
Note3 sections≈ 4 min

'Backup Succeeded': Six Steps to Actually Verify Service Recovery

"Your report says backup — success. What do you check next?" That question, recently covered by DevOps by REBRAIN, kicks off countless DevOps, sysadmin, and DBA interviews. A green status proves only one thing: the script exited with code 0. It says nothing about whether a live service will actually come back after a real outage.

Why a Green Status and a Running Database Prove Nothing

Even if your database boots cleanly from a backup, don't celebrate yet. Start by locking down success criteria. Under NIST SP 800-34, that comes down to two metrics: RPO (Recovery Point Objective) and RTO (Recovery Time Objective). In plain terms, RPO is how many minutes of data the business can afford to lose, and RTO is how many minutes the service can stay down while you restore it.

Next, pick the failure scenario: a dead disk, a lost server, or an accidental DELETE without WHERE. Nightly dumps won't save you from the last one—you need PITR (Point-in-Time Recovery) to rewind to the minute right before the bad query.

Once the target is clear, verify the whole recovery bundle. A folder full of archives is useless if decryption keys sit in an unreachable vault or left with a former teammate. You need the backup, keys, configs, compatible software versions, and permissions. Incremental backups require the full chain—in PostgreSQL, pg_combinebackup assembles it per the PostgreSQL continuous archiving and PITR docs.

For PostgreSQL PITR, the rule is strict: one base backup plus an unbroken chain of WAL (Write-Ahead Log) segments. Missing a single WAL file stops recovery before your target timestamp. Keep configs like postgresql.conf and pg_hba.conf alongside WAL, and verify backup integrity with pg_verifybackup.

Safe Isolated Drills and Application-Level Checks

The real test is a dry run on an isolated staging environment completely cut off from production. Follow the exact runbook you'd open at 3 a.m. Start a stopwatch from the very beginning—including finding the backup, downloading it, fetching keys, and prepping the host.

Watch out for two classic staging traps. First, live integrations: forget to disable real payment gateways, emails, or webhooks, and your drill causes a real customer incident (in PostgreSQL, lock down pg_hba.conf during restore too). Second, toy data sizes: if production holds terabytes and you drill on a tiny dump, your RTO timing is fiction.

For PostgreSQL PITR, don't skip engine specifics: wipe the data directory and old pg_wal/, unpack as the database OS user (not root), set parameters in postgresql.conf, and create the recovery.signal marker. Also note that CREATE TABLESPACE logs absolute paths in WAL—on another host, replay can fail or overwrite the wrong folder, so always take a fresh base backup after tablespace changes.

Once the database is up, verify through the app itself. Log in with a test account, open a known order or document, read it, and create a new record. Check control totals (order counts and sums), foreign-key links, attachments, and error logs. Edits made in production after the restore point won't be there—that's expected.

Measuring Minutes and Making Drills Routine

Finally, compare your real timings against business targets. DevOps by REBRAIN illustrates this with an outage at 2:00 p.m. sharp:

ScenarioData restored toApp ready atData loss / downtimeResult (RPO 15m, RTO 60m)
Everything on target13:5514:425 min / 42 minPassed
Stale restore point13:3014:4230 min / 42 minFailed RPO
Slow startup13:5515:105 min / 70 minFailed RTO

Even a spotless database doesn't save you if hunting for keys blows past your RTO window. Likewise, a fast boot fails if a gap in WAL costs too much data.

To keep verification reliable, log every drill: runbook version, backup ID, restore target, actual RPO/RTO, and any snags. Assign an owner and a regular schedule, and trigger extra drills after schema changes, database upgrades, or backup tool changes. Start small: pick one service, agree on RPO and RTO, and restore a fresh backup in staging—missing keys and broken steps will show up now instead of during an outage.

Want a deep diveVotes help us choose what to break down in detail. This material has 0 votes.
Sources · 5