Failure Mode Tabletops: Rehearsing Outages Before They Happen
Most small sites learn how they fail during an actual outage, usually at the worst moment and with the least information. A tabletop exercise is a low-cost way to practice that failure in a meeting room instead. You pick a plausible breakage, describe it to the group, and walk through what each person would do, what they would check, and what they would need to know. The goal is not to predict the future perfectly. It is to find the places where your team would stall, argue, or guess. This article explains how to run a useful tabletop for a site owner or a small technical team, what to put on the scenario card, and which questions reveal the most about your real reliability posture.
Why Rehearse at All
Runbooks written in calm conditions often assume knowledge that disappears under pressure. A runbook may say to check the database connection, but not say where the credentials live, who holds access, or what a healthy result looks like. A tabletop exercise exposes these assumptions quickly. When someone says they would just restart the service, the next question is whether they know which service, on which host, and whether a restart could destroy state that is not backed up.
Rehearsal also distributes knowledge. In many small teams, one person knows how the site really works. If that person is unavailable during an incident, the rest of the team needs to recognize that dependency now. A tabletop turns that hidden reliance into a visible item on a list of things to fix.
Finally, rehearsal reduces panic. People who have talked through a failure once tend to respond with more structure the second time, even when the real event looks different from the drill.
Designing the Scenario Card
Choose one failure that your site could plausibly experience. Good candidates include a login page that returns errors while the homepage still loads, a contact form that silently stops sending messages, a certificate that is close to expiry without anyone having noticed, or a deployment that changes a template and breaks checkout for part of the day. Avoid scenarios that are wildly exotic. The value of the exercise comes from realism and from the specific details of your own setup.
Write the scenario as a short timeline with a few injected facts. For example: at 9:40, a customer reports that the sign-in page shows a blank screen. At 10:05, a second report arrives about the same problem on a different browser. At 10:20, nobody on the team has yet changed anything that day. Release new information in stages, the way a real incident would unfold, and ask participants what they would do at each stage rather than revealing the cause early.
Keep a list of the artifacts participants may ask for, such as recent change logs, error messages, access logs, uptime check history, or a list of which pages depend on which services. If you do not have those artifacts, note that as a finding. Their absence is often the most important result of the exercise.
Questions That Expose Real Gaps
The most useful questions are concrete. Who is the first person notified, and how? Who has authority to roll back a change, and do they have the access to do it? What is the first thing checked, and is that check documented anywhere? If the team needs to contact a hosting provider or a domain registrar, does anyone know the account holder, the support channel, and the information the provider will ask for?
Ask what the team would tell customers and when. Consider whether a status message exists, who writes it, and whether it has to be approved by someone who might be unreachable. Ask what evidence would confirm that the fix worked, not just that the change was applied. If the answer is that the site looks fine when someone opens the homepage, the exercise has found a gap, because a login or checkout path could still be broken.
Ask how the team would decide between a fast workaround and a careful fix. A redirect or a temporary page might restore access quickly but hide the underlying problem. Deciding in advance what counts as acceptable degradation prevents arguments during the event.
Running the Session and Capturing Output
Keep the session to about sixty minutes. Assign one person to read the scenario and inject updates, another to take notes, and make sure every participant has a role even if it is simply to answer questions about their part of the system. Ask each person to describe their next action out loud before the group discusses it. This prevents the most experienced person from silently driving every decision.
At the end, write down three lists: gaps in knowledge, gaps in access, and gaps in process. Knowledge gaps are things nobody could answer. Access gaps are steps that the person who needed to act could not perform. Process gaps are places where the team had no agreed order of operations. Assign an owner and a review date to each item. A tabletop that produces no tracked changes is a pleasant meeting, not a reliability practice.
Repeat the exercise on a regular schedule with a different scenario each time. Over several rounds, you will build a set of documented responses, and the next real incident will feel more like a rehearsed procedure than an improvisation.