Teams that run machine-learning services and automated operations tooling tend to ask the same dozen questions the first time something breaks at 3 a.m. Below, each question gets a direct reply, and the running subject is IncidentOps: the organised way a group handles an unplanned outage, from the first declaration through to the last closed action item. The evidence below is deliberately sceptical: where a popular habit lacks support, it says so.

What does IncidentOps actually cover, and what does it leave out?

It covers declaring an incident, rating its severity, paging the right people, assigning roles, keeping stakeholders informed, applying a mitigation, reviewing what happened and tracking the actions that come out of the review. It does not cover designing alerts, setting reliability targets, or deep forensic work after an attack; those belong to neighbouring disciplines.

One honest caveat is worth stating early. Per the Atlas write-up of this topic, the word is a loose nickname for incident management and not an accredited field of its own. The Atlas treats the practices as solid and the branding as loose, which is a sensible way to read any new “-Ops” word.

Why do automated systems make declaration criteria more important, not less?

When a model-serving stack degrades, nobody is certain whether it is a data issue, a capacity issue or a bad release. That ambiguity is the reason to write declaration criteria down in advance. A workable rule borrowed from large-scale operations practice is to call an incident when another team is needed, when customers can see the damage, or when the problem is still unsolved after about an hour.

The economics favour calling early. Raising a flag that proves unnecessary burns a few minutes. Waiting, by contrast, burns hours: imagine three people quietly poking at production for an afternoon before anyone realises that no two of them share the same view of the situation. If your culture punishes the count of incidents, people will stop declaring them, and your numbers will look wonderful right up to the outage that nobody owned.

Should the engineer who knows the system best run the response?

Usually not. The incident commander coordinates and decides; they are not meant to be a hands-on resolver. A popular open-source runbook hands the job to the most senior participant present on the bridge, whatever their normal rank, and advises rotating it so the skill spreads across the team.

The people allowed to change production form a deliberately small group: an operations lead and the subject-matter experts they pull in. A communications lead posts updates, and a scribe keeps the timeline. In small teams one person may wear two hats, but the hats stay conceptually separate. Roles are hats, not job titles, which is why an intern can be scribe and a director can be a liaison.

Must we understand the origin before touching anything?

No, and this is the single most expensive misconception in operations. Standard relief moves (undo the latest release, drain a sick node group, flip a feature off, fail over, shed load) need only a rough fix on where the fault sits. Understanding why it happened can wait for the review.

A case from a managed Kubernetes provider illustrates the cost of the opposite instinct. A cluster-creation failure lasted six hours and forty minutes, with 41 people in the chat and seven task forces formed. The write-up concluded that rolling back the affected images as soon as the fault was located would have brought relief around 10 a.m. instead of close to noon. The team was investigating when it could have been mitigating.

How often should we update stakeholders?

Fast at the start, then steadily. The cadence the Atlas summarises: an opening note inside five minutes, a scoping follow-up about five minutes later, then no gap longer than twenty minutes during the opening two hours. The final message should say plainly whether any data was lost.

Most hosted status tools offer four stages, namely Investigating, Identified, Monitoring and Resolved. Use them as written. Inventing a fifth state such as “partially resolved” invites readers to guess.

One architecture rule deserves emphasis. Your pager route and your public status page cannot depend on whatever they are reporting about. In 2017 a large storage outage in one cloud region left the provider’s own service-health dashboard unusable for part of the morning because it depended on the failing storage. If your status page runs on the same cluster as your product, it will go down at exactly the moment people look for it.

Which metrics survive scrutiny?

This is where the analytical reader should slow down. “Mean time to resolve” has at least four meanings in circulation, covering repair, recovery, respond and resolve. Always state which phase and which population you mean.

The deeper problem is statistical. Outage lengths bunch up near the short end with a long tail, so a few marathon events drag any average upward. A study by a Google engineer ran simulations on real incident data from three companies. Even when a genuine ten percent improvement was built in, the measured mean difference came out below zero in roughly 38, 40 and 20 percent of runs. Even with nothing changed, about one run in five invented an improvement of thirty minutes. The author’s conclusion was that the default stance on MTTx figures for trend-watching should be rejection.

The Atlas sketches a lab exercise with eleven incidents. Average time to mitigate landed near 78 minutes while the middle value was just 32; one top-severity event by itself pulled the average up by roughly 45 minutes. What to report instead is the distribution of detection, acknowledgement, mitigation and resolution times, the error-budget minutes consumed, pager load as a trailing average, and the share of follow-up actions closed on schedule.

How does this connect to AIOps tooling?

Correlation engines and anomaly detectors help with the front of the funnel, where noise reduction decides whether a human is paged at all. If that topic interests you, the site’s earlier note on cutting event noise with smarter tooling covers it from the detection side. IncidentOps starts once a person has been paged and has to coordinate with others.

A survey run by a monitoring vendor in 2026 reported that 30 percent of respondents cited alert fatigue and 16 percent named painful cross-team coordination. Respondents chose to take part (1,363 of them), so read the result as a temperature check, not a census. Still, the two complaints map neatly onto two halves of the problem: fewer pointless pages, and a calmer way to work together when a page is real.

What should an automated post-incident review look like?

An owner should be named before the incident call ends. A published vendor process allows three calendar days for the gravest class and five working days for the class below it. A facilitator who was not directly involved tends to produce more candid sessions, and the output should be a short list of actions, each with a single owner and a due date.

Watch for three smells: actions that are “mostly mitigative” and so never remove the cause, vague actions with no testable finish line, and closing actions by quietly downgrading them. Another smell is the verdict of human error. Here the Atlas leans on a safety scholar who cautioned that looking back distorts how we judge human performance; dig until you reach the surrounding conditions that made the slip easy.

How do we know whether our practice is maturing?

The Atlas offers a five-step ladder as a proposal rather than a standard. Level one is heroic improvisation, level two adds a severity policy and a rota, level three requires the update rhythm to be kept in nine of ten serious incidents, level four requires four-fifths of priority actions to finish on time over two quarters, and level five is adaptive. The percentages are useful because they are checkable, which is more than most maturity models offer.

Newer to all this? A gentler entry point is the earlier primer for diagnosing IT trouble in real time; after it, the walkthrough on bringing AIOps to your IT staff shows how to judge which parts of the routine deserve automation at all.

What is the shortest useful starting point?

Write a severity policy on one page, put the rota into version control, host a status page away from your production stack, print role cards, and run a tabletop exercise within the first month. Everything else, including the metrics debate, can follow once people know who is wearing which hat when the pager goes off.