../writing

First 15 Minutes of a Prod Incident

Naseebullah Ahmadi  Senior Software Engineer, London

Something's on fire in production and you're the one holding it. Your job for the next fifteen minutes is to shrink the impact and coordinate the response, not to find the root cause.

#incident-response
#debugging
#workflow
3 min read
#engineering
In one line

The first fifteen minutes are about impact and coordination, not cause. Take ownership, declare the incident, mitigate on the fastest lever available, and keep everyone posted; the real fix comes after.

You're here when

Signs this is your situation
  • A page fired, or a customer or colleague reports the product is down or degraded
  • Error rate, latency, or queue depth is outside its normal band on the graphs
  • A deploy, migration, feature flag, or config change went out in the last hour
  • You're first on it and nobody has said "I've got this" yet

The play

  1. 1Take ownership out loud: post "I'm incident lead for this" so there is exactly one coordinator.
  2. 2Declare it: open a dedicated channel or doc, set a severity, start timestamping. Over-declaring is cheap; you can always downgrade.
  3. 3Size the blast radius before touching anything: which users, which regions, which features, total or partial, and is it still getting worse.
  4. 4Check the last change: any deploy, migration, flag, or config push in the affected window. This is the cause most of the time.
  5. 5Mitigate, don't fix: roll back, flip the flag off, fail over, scale out, or shed load, whatever restores service fastest, even if it's ugly.
  6. 6Change one lever at a time and watch the impact metric for a minute before the next move.
  7. 7Post a status update (internal always, external if customers are hit) with impact, current action, and the time of the next update.
  8. 8Pull in a second person now: the service owner to dig, or anyone to run comms and keep the timeline while you work.
@itsnas update.md
codeupdate.md
[SEV2] Checkout failing for ~15% of users - MITIGATING
 
Impact:  card payments returning 500 since 14:32 UTC, ~15% of checkout attempts.
Not hit: existing sessions, everything outside payments.
Action:  rolled back deploy 4f2a1c (shipped 14:28); error rate falling.
Next:    update at 15:05 UTC, or sooner if status changes.
main
Nas (@itsnas)

Which path

A change went out recently

  • Roll it back first, ask why afterward.
  • If a full rollback is slow or unsafe, disable just the new path: the flag, the route, the worker.

Nothing changed on our side

  • Widen the net: upstream provider status, a dependency's incident page, a quota or rate limit, an expired cert or token, a traffic spike or abuse.
  • Buy time by shedding load or dropping to a degraded mode while you look.

Gotchas

  1. 1

    Debugging before mitigating

    Every minute spent on root cause is a minute of impact a rollback would have already stopped.

  2. 2

    Not declaring because it might pass on its own

    Under-declaring loses you responders and the timeline. Downgrading later is easy; reconstructing the first ten minutes is not.

  3. 3

    Stacking fixes

    Three changes at once and you will never know which one helped, hurt, or did nothing. One lever, observe, next.

  4. 4

    Going quiet

    With no updates, stakeholders escalate around you and add noise. A boring "still on it, next update 15:10" is enough.

  5. 5

    Being the lone hero

    A second person running comms and the timeline is worth more than diagnosing it slightly faster on your own.

Confirm you're clear

  1. 1The impact metric is back inside its normal band and has held there for several minutes.
  2. 2An update has gone out that says mitigated or resolved, and makes clear which one it is.
  3. 3The timeline is complete enough that someone else could write the postmortem from it.
  4. 4There's a named owner for the permanent fix and a retro on the calendar.