Dependency Failure Drills: Testing How Your Site Degrades Piece by Piece

GhostHosts editorial ·

Wikimedia servers for the ULSFO caching data center being racked on June 6th, 2013 in San Francisco.
Illustrative image. Mutante · CC BY-SA 3.0 · Cropped and resized. Image source

Most small sites are tested in only one way: someone notices the homepage is down and starts looking for the cause. That reactive pattern hides an important question. When something breaks, what else stops working, and what does a visitor experience in the meantime? A failure drill is a deliberate, low-stakes way to answer that question before a real incident answers it for you. This article describes how to run partial-failure drills for a site hosted on shared, managed, or VPS infrastructure, what to record, and how to turn the results into concrete work. The method does not depend on any particular provider's tooling. It depends on a written map of your site's parts and a willingness to watch what happens when one of them is removed.

Start With a Written Inventory of Moving Parts

A drill is only as useful as the map it starts from. Before you break anything, list every component a visitor touches: the domain and its DNS records, the web server or hosting panel that serves pages, the application layer (a CMS, a storefront, a custom app), the database, outbound email used for contact forms or password resets, third-party scripts such as analytics, payment widgets, or embedded media, and any scheduled jobs that build caches, send newsletters, or clean up sessions.

For each item, write down three things: what it does for a visitor, what depends on it, and who on your team knows how to inspect it. The third column is often the revealing one. If only one person can log into the database or the DNS registrar, that is a finding before the drill begins. Keep the inventory short enough to read in ten minutes. A dense list nobody reviews is worse than a concise one people actually use.

Choose One Dependency Per Drill

The goal of a drill is to observe cause and effect, which means changing one variable at a time. Pick a single dependency and simulate its loss. Good candidates for a first drill include blocking outbound email from the site, disabling a third-party script in a staging copy, pausing a scheduled job, or renaming a cache directory so the application must regenerate it. Avoid starting with changes that could take the live site offline for a long time, and never run an uncontrolled test against production during peak hours.

Where possible, run the drill against a staging environment that mirrors production structure. If you have no staging copy, a quieter option is to test the parts you can isolate: load a page with a third-party script blocked in your browser's developer tools, or submit a contact form while the mail configuration is intentionally pointed at an address you control and expect to fail. Record exactly what you changed, when, and how you reverted it. Reversibility is what makes a drill safe rather than reckless.

Observe Visitor Experience, Not Just Server Status

A common mistake is to judge a drill by whether the server responds. A page can return a 200 status code and still be unusable: a checkout button that spins forever, a form that accepts input and silently discards it, or a layout that loses its styling because a stylesheet came from a failed dependency. During each drill, walk through the paths a real visitor takes. Load the landing page, navigate to the main conversion point, and complete the action a visitor would care about most, whether that is signing in, submitting an inquiry, or finishing a purchase.

Note three categories of effect. Hard failures are visible errors such as blank pages or error messages. Soft failures are degraded experiences such as missing images, slow loads, or broken links. Silent failures are the most dangerous: the site appears healthy while data or messages are lost. Silent failures rarely show up in a simple up-or-down check, which is why drills that follow user paths tend to surface problems that basic status pages miss.

Ask the Questions That Turn Observations Into Decisions

After each drill, answer a fixed set of questions. Which visitor action failed, and was the failure obvious? How long did it take to notice the problem if you had not been looking for it? Was there any signal, such as a log entry, an alert, or a failed job record, that pointed toward the dependency? Who would have been responsible for diagnosing it, and did they have the access and documentation needed to do so? What was the revert step, and how long did it take?

The answer to the detection question matters most. If the only way you would have learned about a failure is a customer complaint, you have identified a monitoring gap. Be careful not to assume that a particular monitoring service or feature exists for your setup unless you have verified it. Instead, write the gap down as a question for your provider or your team: does our current plan expose logs we can review, and do we have a way to be notified when a scheduled job stops running? The honest answer may be no, and that is useful to know.

Turn Drill Results Into a Short Fix List

A drill that produces a long report and no action is wasted effort. After each session, convert findings into a short list with an owner and a deadline. Typical entries include documenting a single-person dependency, adding a note to a runbook explaining how to restore a missing cache, moving a fragile third-party script behind a fallback, or scheduling a repeat drill after a change. Limit the list to what your team can finish in the next month. Three completed fixes teach more than fifteen abandoned ones.

Repeat the drill after changes and compare results. Keep the notes from each session in one place so that patterns become visible over time. When a dependency consistently causes silent failures, that is a strong signal to redesign how the site handles it rather than simply hoping it stays healthy.

A Practical Checklist for Your First Drill

Before the drill, confirm that your written inventory is current, that at least two people know the revert steps, and that you have chosen a window when a short disruption would be acceptable. During the drill, change one dependency, record the exact time and method, walk through two or three visitor paths, and note hard, soft, and silent failures separately. Restore the dependency and confirm the site returns to its normal state, checking the same paths again. Afterward, write down who noticed what and how, identify the single most important gap, and assign one owner to address it. Review the inventory and update any entries that proved inaccurate. A drill done this way takes an afternoon, and it tends to reveal more about your site's real reliability than any amount of reading about it.