Every operations team has a person like this. When the database slows down at midnight, everyone quietly hopes that one engineer is awake, because she knows which dashboard to open, which restart is safe and which one makes things worse. That knowledge is valuable, and it is also fragile. The day she is on leave, or moves to another team, the know-how walks out with her. An AIOps runbook is the plain, written answer to that problem: a short step-by-step routine for a recurring task, kept where anyone on the team can find it and follow it.

This explainer sets out what a runbook is, why it matters even more when machine-learning tools are watching your systems, and how to write one without turning it into a forty-page manual nobody reads. The tone here is deliberately analytical: where a claim rests on common operating experience rather than a measurement, it is described that way.

What an AIOps runbook actually is

AIOps, in simple terms, means using software that learns patterns from operational data to help people notice, group and respond to problems. It might correlate alerts, flag unusual behaviour or suggest a likely cause. A runbook is not part of that software. It is the human side of the arrangement: a written procedure that says what to do when a particular situation appears, in what order, and who to call if the steps do not work.

The distinction matters because the two are often confused. A tool can tell you that a service is behaving oddly. It cannot, on its own, tell your newest colleague whether to drain a node, roll back a release or simply wait ten minutes. Someone, at some point, decided the answer for your environment. The runbook records that decision so it does not have to be rediscovered at three in the morning.

Why one person’s memory is a poor place to store procedures

Consider what happens without a written routine. The task gets done well when the expert is present and unevenly when she is not. Colleagues improvise, and improvisation under pressure produces variation: one person restarts the service first, another clears a cache first, and afterwards nobody can say which action fixed the problem. This is an observation about how teams behave, not a statistic, but anyone who has worked a few incidents will recognise it.

There are three recurring costs of keeping procedures in heads:

• Single points of failure. Sickness, holidays and resignations are normal events, not emergencies. A process that depends on one person’s availability will eventually fail on a bad day.
• Slow onboarding. New joiners learn by shadowing, which is wonderful but slow, and it consumes the expert’s time exactly when she is busiest.
• Invisible drift. When steps live only in memory, they change quietly. Six months later the “usual way” differs from what the documentation, if any, says.

Where AIOps makes the runbook more important, not less

A common assumption is that smarter tooling reduces the need for written procedures. In practice the opposite tends to hold. Automated detection raises more signals, earlier, and people still have to decide what to do with each one. If those decisions are improvised, the speed gained in detection is lost in hesitation.

A good runbook also gives automation something to point at. Many teams start by writing a manual routine, run it by hand several times, and only then automate the steps that proved stable. The written version is the specification. Skipping it and automating straight away often means automating a guess.

Anyone who wants to see how the broader DataOps community talks about keeping pipelines dependable can browse TheDataOps.org, a site aimed at data engineers and practitioners that keeps returning to the road from DevOps to DataOps. The same habit of writing routines down carries across from data work into general IT operations.

The parts of a useful runbook

Length is not the goal. Most effective runbooks fit on one screen or one printed page, because they will be read by a tired person under mild stress. A reliable skeleton looks like this:

1. Title and trigger. Name the situation in the words people actually use (“checkout is slow”, not “latency anomaly class B”), and say what makes this runbook the right one to open.
2. Owner and last-reviewed date. A runbook with no owner decays. A visible date tells the reader how far to trust it.
3. Quick checks. Two or three things to look at first, with exact locations: which dashboard, which log query, which status page.
4. Steps in order. Numbered, one action each, written as commands to a colleague: “Restart the worker pool”, not “the worker pool may need to be restarted”.
5. How to tell it worked. The specific signal that means you can stop, so nobody keeps poking at a recovered system.
6. When to escalate. A clear threshold and a named contact. This protects juniors from feeling they must solve everything alone.
7. Notes from last time. One or two lines on anything surprising. This is where hard-won knowledge accumulates.

Choosing which tasks deserve a runbook

Resist documenting everything in one go. One workable rule: when a job has come round more than twice, or a fix worked and losing it would hurt, write it down. That filter naturally picks out the recurring jobs, such as reopening a stuck queue, rotating a certificate, handling a noisy alert that turns out to be harmless, or restoring a service after a failed deploy.

It also helps to ask a blunt question: “If our most experienced person were unreachable for two weeks, which tasks would stall?” Write those first. The answers are usually uncomfortable and always useful.

A worked example in plain words

Suppose your monitoring tool groups several alerts into one incident: response times on an internal API are rising. A short runbook for it might read as follows.

1. Open the service dashboard and confirm the rise is real by checking whether it appears on two or more graphs, not just one.
2. Check whether a release went out in the last hour. If yes, follow the rollback runbook and stop here.
3. Look at the connection pool graph. If it is close to its limit, restart the pool using the standard command, then watch for five minutes.
4. If response times fall back to normal, record the time and the action in the team channel.
5. If they do not recover within ten minutes, escalate to the on-call lead with a note of what you tried.

Notice what is missing: theory, history, options. Background reading belongs elsewhere, and you can link to it. If you want to widen your understanding of the concepts behind step three, our guide to service-level indicators, objectives and agreements sets out the vocabulary that most teams use when they decide what counts as “normal”.

Keeping runbooks alive

The hardest part is not writing the first draft. It is keeping the document honest. Evidence from everyday practice suggests that runbooks go stale in predictable ways: a command changes, a dashboard is renamed, a contact leaves. Three habits prevent most of the rot.

• Update after every use. Whoever follows a runbook edits it straight afterwards if anything was unclear or wrong. Five minutes of editing while the details are fresh is worth far more than a quarterly clean-up.
• Test by handing it to someone new. The best review is a newcomer working through the steps while the author watches without speaking. Every question they ask marks a gap.
• Put the date and the owner on the page. If a runbook has not been touched for a long time, that fact should be visible, so readers treat it with appropriate caution.

Common mistakes to avoid

The first is writing for experts only: if the page assumes knowledge a new joiner lacks, it fails when it is most needed. The second is the opposite error, stuffing every branch into one page until the main path disappears. Split rare cases into their own short runbooks.

A third mistake is hiding the runbooks in a folder that nobody remembers. A procedure that cannot be found in thirty seconds might as well not exist. Link each one from the alert itself, so the person who receives a notification is one click from the routine for that notification. The beginner’s tour of live troubleshooting on this site covers the thinking behind the steps.

How to start this week

You do not need a special tool or a documentation project. Pick one recurring task that currently depends on a single person. Ask that person to do it once while a colleague writes down each action in order. Tidy the notes into numbered steps, add the trigger and the “how to tell it worked” line, and store the page next to your alerts. Then ask a third person to try it. That small loop, repeated for the next task and the next, is how a team builds a library of runbooks without ever scheduling a big initiative.

Teams introducing machine-learning tools often ask where to begin; our piece on introducing AIOps to an IT team covers the wider rollout, and the runbook habit fits neatly alongside it. The aim is modest and practical. When someone is off sick or moves on, the work should continue at the same quality, because the steps were never theirs alone.

The short version

A runbook turns private know-how into a shared, testable routine: a trigger, ordered steps, a signal that you are done and a route for escalation. Write the repeated tasks first and make the pages easy to find.