Resilience drills

Most outages are caused by something you did not test. A dependency times out, a queue backs up, a downstream service starts returning 503s. If you have never seen your code in those states, you find out at 2 AM how it behaves. Resilience drills are how you rehearse the bad day on a good day.

after/routes/debug.routes.ts
typescript
// ONLY mount in non-production environments
if (process.env.NODE_ENV !== 'production') {
  app.post('/__debug/fail', (req, res) => {
    res.status(503).json({ error: 'injected failure for drills' });
  });
}

A controlled failure endpoint that you can toggle on in non-production environments.

Kinds of drills worth running: turn off Redis mid-test and confirm the API still serves reads (your caching should degrade gracefully). Stop the Inventory Service and confirm orders still succeed but stock is flagged as pending (your events should not block the API). Flood the login endpoint and confirm rate limits kick in with 429s (your limiter should hold). Kill the database and confirm the health check flips to not-ready and the load balancer pulls the instance out.

Run these drills in staging, not production. Schedule them on a cadence: quarterly for small teams, weekly for larger ones. The first time you run them, you will find bugs. Fix them, run the drill again until it is boring, then move on to the next scenario. Boring drills mean resilient systems.

Quiz: Quiz

Loading practice…