Marisol ran operations for a mid-sized logistics company, and her team was proud of the new anomaly-detection model they had shipped in the spring. Three quarters later, an executive asked a simple question in a meeting: why did the weekly capacity forecast still quote a runbook page that referred to a retired cluster? Nobody knew. The page had been linked from a dashboard for so long that everyone assumed somebody else maintained it. That small embarrassment is the reason this AIOps maintenance audit exists. Teams love launching models, pipelines and automations; they are far less enthusiastic about going back to see whether the earlier ones still behave.

Marisol’s story is a composite of the kind of thing operations teams describe at retrospectives, not a report about one named company. What makes it useful is how ordinary it is. Nothing exploded. Things just drifted, quietly, until one of them showed up in front of the wrong audience. The checklist below is what her team built afterwards, and it works for most AIOps practices whether you run three models or thirty.

Why AIOps estates decay faster than you expect

An AIOps setup is a stack of moving parts that each depend on something outside themselves. A correlation rule depends on the naming convention of the services it watches. A forecasting model depends on the shape of the data feeding it. A notification route depends on a person who still works there. When any of those inputs changes, the automation does not announce its confusion; it simply keeps producing output that looks plausible.

The honest word for this is decay. The model was accurate when it was trained and nobody has asked it to prove itself since. If you want a fuller picture of how a team brings the practice in at the start, our primer on bringing AIOps into an IT team properly covers the adoption side. This piece is about the long middle, after the launch party and before the post-incident review where you discover what rotted.

The audit checklist: whether it works, whether anyone needs it, whether it sends people somewhere real

Marisol boiled her audit down to three questions, asked about every asset in the estate. First, does it still work. Second, is it still needed. Third, does it still point to the right place. Everything on the checklist is a variation of one of those.

Part one: does it still work

• Model output against reality. Pick a recent week and compare what each model predicted with what actually happened. You are not looking for perfection, only for a gap that has widened since the last time anyone looked.
• Input freshness. Confirm that every data source feeding a model is still arriving, still complete and still named what the model expects. A silently renamed field is the most common culprit.
• Scheduled jobs. Open the scheduler and look at the last successful run of each retraining, scoring and cleanup job, not just whether it is enabled.
• Escalation paths. Send a deliberate test through each routing rule and watch who actually receives it. People change teams and mailboxes get retired.

Part two: is it still needed

• Owner. Every rule, model and dashboard gets a named owner. If you cannot name one, that is the finding.
• Last real use. Find out when a person last looked at the output or acted on it. Assets that nobody has consulted for a long stretch are candidates for retirement.
• Overlap. Two models answering the same question in slightly different ways confuse people at three in the morning. Keep the better one.
• Retirement notes. When something is switched off, write down why and when. Future colleagues will otherwise rebuild it.

Part three: does it point to the right place

This third group is the one teams forget, because it feels like housekeeping. Yet most of the embarrassing failures in Marisol’s story came from here. Runbooks link to dashboards, dashboards link to tickets, tickets link to vendor documentation and status pages. Each of those links is a promise that clicking it will help someone under pressure. When the destination moves, the promise breaks without any alarm.

• Runbook references. Walk every link in your incident documentation and confirm it opens the page it claims to.
• Dashboard annotations. Links from graphs to explanatory notes tend to outlive the notes.
• Public-facing pages. Status pages, team profiles and onboarding guides should send people to current resources.

Let something else watch the links

Checking links by hand is the sort of chore that is done twice and then forgotten. This is where a link-management platform can take the boring part. URLSnow, a link management platform, keeps a log from its scheduled health checks of which saved addresses still answer, which turns a dead destination into a flagged entry on a list rather than a surprised stakeholder. It will not tell you whether the page behind a working link is still correct, so the human read-through still matters, but it removes the guesswork about which links are simply dead.

Retire as deliberately as you launch

Marisol’s team added one rule to their process: no new model goes to production until its own retirement conditions are written next to it. What would make us switch this off? Who decides? Where does the documentation go afterwards? It sounds bureaucratic, but it took about ten minutes per project and it changed the conversation. Retiring something stopped feeling like admitting failure and started feeling like routine care.

The same mindset helps when someone proposes a clever new capability. Before saying yes, ask what existing thing it will sit next to and whether that thing is healthy. The team at one platform group put it bluntly after reading a critique of predictive maintenance applied to IT systems: a prediction about when something will fail is worth little if nobody has checked that the thing being predicted is still the thing that matters.

Make the audit small enough to repeat

An audit that takes a week will happen once. Marisol’s version takes an afternoon because it only samples. Each round, the team picks a slice of the estate, perhaps one service area, and runs the three questions across everything in it. Over a few rounds the whole estate gets covered, and the sampling itself reveals patterns, such as which team never names owners or which data source breaks most often.

A simple log helps. For each asset record the date checked, the verdict (healthy, repaired, retired) and one line of explanation. When something breaks later, you will know whether it was last examined last week or never. For wider context on the day-to-day side of diagnosis, the notes on solving IT problems as they happen pair well with this routine, because problems found early cost far less than problems found live.

What changed for Marisol’s team

After two rounds, they had retired a handful of forgotten rules, repaired several broken routes, and found that the stale runbook link was only one of many. Nothing about the exercise was clever. What shifted was the attitude: the team stopped measuring itself only by what it launched and began counting what it kept healthy. When the executive asked a similar question the following quarter, the answer was a date and a name.

Should only one idea stick, let it be the order of operations. Before you build the next detector, scan the estate you already have, fix what is cheap, retire what is dead and write down what you learned. The newest model will be much easier to trust when the older ones are no longer quietly lying beside it.