Stale Runbooks Kill Recovery: Testing Your Site Recovery Steps Before You Need Them

GhostHosts editorial ·

Wikimedia servers for the ULSFO caching data center being racked on June 6th, 2013 in San Francisco.
Illustrative image. Mutante · CC BY-SA 3.0 · Cropped and resized. Image source

Most small sites have some form of recovery plan, even if it lives in someone's head or in a chat thread from two years ago. The problem is rarely that nobody thought about failure. The problem is that the written or remembered steps were never exercised, so when an outage arrives they turn out to reference dashboards that moved, passwords nobody holds anymore, or a restore process that only worked on the original server. This article looks at a different angle from simply writing a runbook: how to test one, what a stale runbook looks like, and which questions reveal whether your recovery steps still match reality. The guidance here is general and applies regardless of which hosting arrangement you use.

Why recovery steps decay faster than most documentation

Recovery steps are unusual among documentation because they describe actions that are almost never performed. A deployment guide gets used weekly and gets corrected by daily friction. A recovery guide might be opened only a few times in the life of a site, and each time it is read under stress. Between those uses, the environment keeps changing. An admin panel gets redesigned, a support contact changes, a DNS provider is swapped out, or a team member who knew a shortcut leaves the company. None of these changes announce themselves to the document.

This decay is predictable, which means it can be managed. Treat every recovery step as a claim about the current system, and treat that claim as expiring. A useful habit is to attach a verification date to each runbook section, and to treat any section older than your last real change as suspect until someone confirms it. The age of a document matters less than whether anyone has performed the steps since the system last changed.

Designing a tabletop test that exposes gaps

A tabletop exercise walks through a hypothetical failure without touching production. It is the cheapest way to find documentation gaps, and it works well for small teams. Pick a realistic scenario, such as the site returning errors after a configuration change, and have one person read the runbook aloud while another follows each step on paper or in a scratch environment. The reader's job is not to speed through but to stop at every point where the instructions assume knowledge they do not have.

Record every stop. A stop might be a step that says 'restore from the latest backup' without saying where backups live, who can trigger a restore, or how long it takes. Another stop might be a credential that is referenced but not stored anywhere the on-call person can reach. Each stop becomes an action item. After the exercise, rewrite the steps so that someone who has never seen the site could complete them with only the document and the account access they are supposed to have.

A practical checklist for auditing recovery steps

Once a runbook exists, audit it against a short list of questions. Does each step name the exact screen, command, or contact it depends on? Are the locations of credentials recorded in a place that survives the departure of any single person? Does the document state what 'done' looks like for each step, so the operator knows when to proceed? Is there a stated owner for each section who is responsible for keeping it accurate?

Further questions probe harder assumptions. If the primary admin account is locked out, is there a documented path back in? If a restore is needed, is it clear whether the restore overwrites current content or sits beside it? If the fix involves a hosting plan change or a support request, does the document say how long that typically takes, based on your own past experience rather than guesses? Note that you should verify any provider-specific behavior directly with the provider, because published claims change and this article cannot tell you what any particular provider currently offers.

Comparing hosting arrangements in recovery terms

The shape of your recovery steps depends partly on how your site is hosted. On a shared plan, much of the infrastructure is managed by the provider, so your recovery steps tend to center on your application content, your own backups, and the support channels you can reach. On a VPS, you carry more of the operating system and service layer, which means your runbook has more steps that you own directly. A managed WordPress arrangement often shifts some platform concerns to the provider while leaving application-level decisions with you.

Rather than assuming any arrangement is inherently safer, write down which steps you would perform yourself and which depend on someone else. Then ask the questions that matter for your setup: what backup copies exist and who can access them, what the documented process is for restoring, and what you would do if the hosting account itself became inaccessible. Verify these answers against the current terms and documentation for the plan you actually use. Plans and their feature sets vary, and any detail you cannot confirm should be marked as unknown in the runbook instead of filled with an assumption.

Keeping the runbook alive after the test

A test that produces a rewritten document and then sits untouched has bought only a short window of accuracy. Schedule a recurring short review, perhaps after each significant change to the site or its accounts, and assign it to a named person. Make the review concrete: confirm that each listed contact still works, that each credential location is still correct, and that at least one restore or rollback step has been performed in a safe environment within the last review cycle.

Finally, make the runbook easy to find and easy to read under pressure. Short numbered steps, plain language, and links kept inside your internal documentation are more useful than thorough prose that nobody reads at 2 AM. The goal is not a perfect document but a document whose claims have been checked recently, with gaps visibly marked. A recovery plan you have tested and honestly labeled is far more valuable than a polished one that has never met a real failure.