Most operations teams can tell you about a first live run that went sideways for reasons nobody had written down. The remediation script assumed a service account that only existed in staging. The alert routing pointed at a channel that had been renamed. The model scored beautifully on last quarter’s data and then met a deploy freeze it had never seen. An AIOps rehearsal is the habit that prevents most of these: you run the whole change once, end to end, in a place where nothing important can break, and you treat what surprises you as the product of the exercise.

This piece is written as an audit. Each section is a set of checks you can tick off, plus the evidence you should leave behind so later readers need not rely on anyone’s recollection. It assumes you already have a change in mind, such as a new auto-remediation rule, a retrained anomaly model or a changed escalation path, and you want to know whether it is ready for the real environment.

Why the first live run is the wrong place to learn

The cost of finding a flaw rises sharply with each step toward production. A broken assumption found at a desk costs minutes. The same assumption found during a rehearsal costs an hour and a ticket. Found live, it costs an incident, a review meeting and some trust. The point of a dry run is to move discovery to the cheap end of that curve, and the evidence for doing so is mostly your own incident history: look back at the last handful of change-related problems and ask how many would have surfaced if someone had simply performed the change once in a quiet environment.

Audit one: is the rehearsal environment honest?

• Same shape, smaller size. Confirm the staging stack has the same services, versions and permissions model as production, even if it carries far less traffic. List every difference you know about in writing.
• Realistic inputs. Feed the automation recorded or replayed events rather than a tidy hand-made sample. Real telemetry has gaps, duplicates and odd timestamps.
• Real integrations, harmless targets. Chat, ticketing and paging should be wired the way they are live, but pointed at a sandbox queue so nobody gets woken up.
• A clear boundary. Write down what the rehearsal cannot prove, such as behaviour under peak load or interactions with a vendor system you cannot copy.

The last item is the one teams skip. A rehearsal that claims to prove everything quietly teaches people to relax about the parts it never touched.

Audit two: walk the change as a person would

Automation often fails at its edges, where a human has to do something. So rehearse the human parts too. Hand the runbook to someone who did not write the change and ask them to follow it with no coaching. Watch where they pause. Those pauses mark the missing step, the unclear name or the command that needs a flag nobody mentioned.

1. Start from the trigger: what exactly fires the automation, and who is told?
2. Follow the decision: where does a person approve, override or pause it?
3. Follow the action: which systems change, in what order, with which credentials?
4. Follow the exit: how do you stop it halfway, and what state does it leave behind?
5. Follow the record: where is the outcome logged, and can someone find it a month later?

Teams that already work this way tend to be the ones that get real-time problem solving right under pressure; the walkthrough in this beginner-friendly primer on solving IT problems in real time makes a similar point about practising the response before you need it.

Audit three: evidence worth keeping

A rehearsal remembered by one person is forgotten by the next sprint. Keep a small bundle of artefacts each time:

• The exact version of what was rehearsed.
• The replay window you used and how to reproduce it.
• A timeline of what happened, including anything that took longer than expected.
• A short list of surprises, each tagged as fixed, accepted or still open.
• A named person who signs off that the open items are acceptable for go-live.

Notice that the sign-off is for open items, not for a perfect result. A rehearsal that produces zero surprises should make you suspicious rather than relieved. Either the change is trivial, or the environment is too forgiving to tell you anything.

Audit four: model-specific checks

If the change involves a learned model, add a few checks that ordinary release rehearsals ignore. Replay a period that contains a known incident and confirm the model flags it within the time you consider useful. Replay a quiet period and look at what it flags anyway, because that is the behaviour people will experience on most days. Check what happens when an upstream feed goes silent: does the system treat silence as normal, as a fault, or does it simply stop reporting? Finally, confirm you can switch the model off cleanly and that the previous behaviour returns without a redeploy.

A spreadsheet of replay windows, expected outcomes and observed outcomes is enough tooling. Teams thinking about how decisions get supported by this kind of tooling may find the site’s own discussion of machine-assisted IT decisions a useful companion read.

Audit five: the dress rehearsal for the people

The technical run can pass while the human arrangements are still vague. Before go-live, confirm the following with the people who will actually be on call that day:

• They know the change window and what normal looks like during it.
• They know where the rollback instructions live and have seen them once.
• They know who to contact if the automation does something unexpected, and that person knows they are the contact.
• They have a plain-language description of what the automation will and will not do on its own.

That last point is more important than it seems. When responders are unsure whether a system acts alone, they either interfere too early or wait too long. A single paragraph agreed in the rehearsal removes the ambiguity.

A parallel from the kitchen

Cooks have always known this. Anyone planning a large family meal tests the unfamiliar dish on a quiet evening first, because the oven that runs hot or the ingredient that substitutes badly is better discovered over a weeknight supper than in front of guests. Blendz is a global food-heritage archive of traditional recipes, rare ingredients and food stories, and visitors can contribute their own, which is a reminder that so much cooking knowledge survives precisely because people tried things, adjusted them and wrote down what actually happened.

Reading the results honestly

When the rehearsal is over, sort what you found into three piles. Things you fixed go into the change itself. Things you accept go into the release notes with an owner. Things you cannot yet explain are the dangerous pile, and they should block go-live until someone can at least describe the mechanism. It is tempting to wave through an unexplained oddity because the schedule is tight, but an oddity you cannot explain in staging will be even harder to explain at three in the morning.

Compare the rehearsal’s duration with your plan, too. If the window you booked is shorter than the run took, shorten the steps, split the change or book longer.

Keeping the habit lightweight

Teams abandon rehearsals when they feel like ceremony, so keep yours proportionate. A small configuration tweak needs a short run and a paragraph of notes; a change to how incidents are triaged deserves the full audit.

If you are building these habits as part of a wider programme, the practical sequencing in a step-by-step guide to introducing AIOps to an IT team pairs well with this audit. The rehearsal is just one practice, but it is the cheapest one to start this week.

The short version

Run the change once where it cannot hurt anyone. Make the environment honest about its limits. Let a newcomer follow the runbook. Keep evidence, name an owner for the leftovers, and refuse to ship what you cannot explain. If the real run then feels boring, the rehearsal did its job.