If your team has started weaving machine intelligence into incident response, you have probably wondered whether you are doing it well or just doing it. An AIOps maturity audit answers that question with a number instead of a feeling. It is a periodic, scored review of how your IT operation actually works — how alerts are handled, how automation is governed, how knowledge is captured — repeated on a schedule so you can watch the number move. Teams keep asking us the same things about it, so here are the honest answers, question by question.

What is an AIOps maturity audit, really?

Think of it as a formal check-up for your operations practice. At a set interval — twice a year works for most teams — you sit down with a fixed questionnaire and grade yourselves against it. The questions probe habits rather than incidents: who reviews automation rules before they go live, whether runbooks exist for the alerts your correlation engine produces, how model-driven decisions get sanity-checked by humans. Each answer maps to points, the points roll up into a score, and the score gets filed next to the previous one. That little archive of numbers becomes the truest record of whether your operation is actually maturing.

Isn’t that just what our tooling already tells us?

No, and the distinction matters. Your platforms report what happened this week; an audit examines how your organisation behaves regardless of what happened. You can have a quiet month with terrible practices or a brutal month with excellent ones. The audit deliberately looks away from the event stream and toward the discipline underneath it: the review habits, the ownership lines, the documentation, the guardrails on automated actions. If you want to shrink the event stream itself, that is a separate project — we have written before about taming alert floods with smarter correlation — but no amount of noise reduction substitutes for knowing where your practice stands.

Why does the scoring have to be repeatable?

Because a score you cannot reproduce is an opinion wearing a costume. If two engineers grading the same operation in the same week would land on different numbers, the exercise collapses into whoever argues loudest. Good audit instruments fix this by anchoring every question to observable evidence: a thing either exists or it does not, a process is either followed or it is not. Software delivery teams learned this lesson first. SCMGalaxy, a training and consulting firm that now ships a governance product called SCMGalaxy OS, built its approach around an assessment of one hundred fixed questions precisely so that identical answers always produce an identical score — which is what makes comparing this quarter to last quarter legitimate.

What areas should the questionnaire cover?

For an AIOps-flavoured audit, lean on the areas where machine assistance meets human accountability. Borrowing from the domain structure that platform uses for delivery organisations, two translate directly: one domain scores observability and reliability engineering practice — are signals defined, owned, and acted on consistently — and another scores how AI usage itself is governed, meaning who approves models, how their outputs are audited, and what happens when they misfire. Around those, add your own: alert lifecycle management, automation change control, post-incident learning, and the handoffs between tooling and people. Six to ten areas is plenty; depth beats sprawl.

Who should actually run it?

Someone close enough to know the truth and senior enough to write it down. In most IT groups that is a lead SRE or operations manager, with one rule enforced ruthlessly: answers must cite evidence. “We review automation changes” counts only if the reviewer can open last month’s review record. It helps to involve a sceptic — the engineer who thinks the whole thing is theatre will ask the questions that keep it from becoming theatre. External facilitation is optional; what is not optional is honesty, because a flattered score helps nobody and costs you two quarters of misdirected effort.

We got our score. Now what?

Now the score becomes a plan, and the plan should be staged rather than heroic. A pattern worth copying from the delivery-governance world is the three-horizon roadmap: the first thirty days go to cheap, visible fixes — naming owners, writing down the automation approval rule you already informally follow. The next stretch, out to day ninety, tackles one structural weakness properly, such as building the runbook library your audit exposed as missing. The half-year mark is for the cultural shift: making evidence-based review the default rather than the exception. SCMGalaxy OS generates exactly this kind of 30/90/180-day sequencing from assessment results, and the stages matter more than the tool — a plan without dates is a wish.

How do we know the plan worked?

You audit again. This is the step teams skip, and skipping it wastes everything that came before. Run the same questionnaire, under the same evidence rules, roughly six months later. If the number climbed, you have proof of progress you can show leadership — far more persuasive than anecdotes. If it did not climb, you have an early warning that your improvement effort is stuck, delivered while there is still time to change course. Either outcome is valuable; only silence is worthless. Over two or three cycles the trend line becomes the single most credible artefact your operations team owns.

Does a small team really need this?

Small teams arguably need it more, because they have no slack to waste on improvements that do not improve anything. A five-person operation can run a lightweight version in an afternoon: pick eight questions, score them with evidence, write a one-page plan, diary the re-score. The formality scales with you. And if your team is still early in its adoption journey, the audit pairs naturally with a deliberate rollout — our guide to bringing machine-assisted operations into a team without chaos covers the adoption side, while the audit tells you whether the adoption actually stuck.

What are the common ways teams get it wrong?

Three failure modes account for most of the wreckage. First, grading on vibes: no evidence rule, so the score reflects mood. Second, auditing once: a single score is trivia, because the value lives in the comparison. Third, scoring everything and fixing nothing: the questionnaire becomes an annual ritual detached from any roadmap. A fourth, subtler trap is letting the audit drift into a tooling review — the question is never which platform you bought, but how your people behave around it. Decision support is a good example: machine recommendations only earn trust when humans audit them, a theme running through our look at letting AI inform operational decisions without surrendering them.

Where should we look for a worked example?

Watch how the software delivery discipline does it, because they are several years ahead. Platforms there produce a scored snapshot across their governance domains, flag the riskiest gaps, and track movement between assessments — SCMGalaxy OS even publishes a sample scorecard showing how an organisation’s composite number breaks down by area, purely as an illustration of the format. You do not need to adopt their domains wholesale. You need the shape of the practice: fixed questions, evidence-anchored scores, a dated roadmap, and a re-score that proves the roadmap was more than decoration. Put your first audit on the calendar for this month, and the second one six months out, before any of the findings arrive to discourage you.