Cruxly › Cloud and DevOps › Cloud & DevOps: Incident Response
Running an Incident: Roles, Communication, and Not Making It Worse
why the Incident Commander is deliberately not the resolver
Open the interactive lessonPrompt: a production incident is declared. Multiple engineers start independently investigating and proposing fixes, someone is fielding questions from other teams, and no one is tracking the incident's overall state or timeline. Design the incident-command structure that fixes this, and explain precisely why the person coordinating the response should not also be the one debugging it.
30 minutes first. The naive fix is "get everyone on a call." A call without a defined structure and defined roles just centralizes the same chaos into one channel instead of several.
The Incident Commander coordinates; a separate role resolves
The IC's job is setting priorities, making the final call between competing options, and keeping the response coordinated — explicitly not debugging the problem themselves. This single-point-of-coordination structure exists to prevent two specific, common failure modes: multiple people independently trying to lead the response, issuing conflicting directions that responders have to reconcile themselves mid-incident; and a technically-skilled lead who's absorbed in the code, leaving no one tracking the incident's overall timeline, state, or communication needs while they focus on the fix.
A well-run incident hands the IC a tight action plan, not an open-ended mandate — measurable objectives, named owners (including backups) for each workstream, explicit time-boxes on how long to try a given approach before escalating or switching strategy, named risks, and clear exit criteria for declaring the incident resolved. This structure is what lets an IC make fast, confident decisions without personally understanding every technical detail of the problem — they don't need to know how to fix the database, they need to know whether the team currently trying has a time-box and a fallback plan if it doesn't work.
Communication is a distinct job, not a side effect of fixing the problem
A Communications Lead manages information flow to responders, the rest of the organization, and external stakeholders — as an explicit, assigned job, not something the IC or the resolvers handle in whatever spare attention they have. Drafting status updates, posting to a status page or incident channel, and fielding questions from outside the response team all take real time and focus that, without a dedicated owner, either doesn't happen (leaving everyone outside the incident uninformed and generating a flood of "any update?" interruptions directly to the people trying to fix it) or gets done by someone who should be focused on resolution instead.
The three concurrent goals of any incident response framework — coordinate, communicate, maintain control — fail independently, and a structure that only assigns one or two of them will visibly fail at whichever one it left unassigned. A team that appoints an IC and Ops Lead but no Communications Lead will coordinate the technical response fine and still generate frustration from every team waiting on an update; the fix isn't debugging faster, it's recognizing communication as its own real job with its own owner.
Maintaining control means the incident has a clear, current state at every point — not "ask whoever's currently talking." A running, visible timeline (what's been tried, what's currently being tried, what decisions have been made and why) is what lets someone joining the incident mid-way get oriented in seconds instead of interrupting active work to ask, and is what a retrospective postmortem actually depends on existing accurately, since reconstructing a timeline entirely from memory after the fact is unreliable in exactly the ways a postmortem needs to not be.
Severity levels, mitigate before you diagnose, and handing off command
Severity levels decide who gets woken up, so define them by user impact, ahead of time. A typical scheme:
| Level | Meaning | Response |
|---|---|---|
| SEV1 | Core product down or data at risk for many users | Page immediately, IC + comms, status page, exec update |
| SEV2 | Major feature degraded or a large subset of users affected | Page, IC, internal updates |
| SEV3 | Minor or contained impact, workaround exists | Business-hours ticket |
The exact labels matter less than two habits:
- Anyone can declare. An engineer who suspects a SEV2 shouldn't need permission to start one.
- Declare early and downgrade freely. A false alarm costs a few minutes. A late declaration costs the time it took to get coordinated.
Mitigate first, diagnose later. The first question isn't "why did this happen?" It's "what's the fastest safe way to stop the user impact?" Options include:
- rolling back the last deploy;
- turning off a feature flag;
- failing over to another region;
- shedding load or scaling up.
Do these even before you're sure they address the cause. If the incident started 10 minutes after a deploy, roll back first and find out why afterwards. Full root-cause analysis belongs in the postmortem, once users are no longer affected.
Add a Scribe, and hand off command explicitly. A Scribe records the timeline as it happens: what was observed, what was tried, and decisions with timestamps. The IC is then free to coordinate, and the postmortem gets an accurate record. Long incidents need IC handoffs, because nobody makes good decisions in hour nine. A handoff is explicit:
- The outgoing IC briefs the incoming IC on the current state, active workstreams and open decisions.
- The incoming IC says out loud "I am now the IC" in the incident channel.
- Everyone knows who's in charge.
Rotating responders on a schedule is part of the plan, not a sign of weakness.
How to say it in the interview
"Putting the most senior engineer in charge of an incident AND having them personally debug it is a common instinct, and it's exactly backwards - the moment they're deep in a stack trace, no one is tracking the incident's overall state, deciding priorities, or managing communication. The Incident Commander role exists specifically to be the coordination point that ISN'T also absorbed in the technical work, which is the only way both jobs get done at once instead of one starving the other."
Checklist
- Assigned an Incident Commander whose job was coordination and decisions, not personally debugging the problem
- Gave the IC a tight action plan structure — named owners, time-boxes, exit criteria — rather than an open-ended mandate
- Assigned communication as an explicit, owned role, not something handled in whoever's spare attention
- Maintained a running, visible incident timeline (kept by a Scribe) rather than relying on 'ask whoever's currently talking'
- Used pre-defined, impact-based severity levels that anyone could declare, declaring early and downgrading freely
- Mitigated first (rollback, flag off, failover) before diagnosing the root cause
- Handed off IC explicitly with a state briefing on long incidents