The platform team at a mid-sized logistics software company thought their operations picture was in good shape. Every change had a ticket, every automated remediation wrote a log line, and the AIOps tooling showed a healthy estate. Then a routine capacity review turned up a database replica that nobody on the current roster had provisioned, a scheduled clean-up job that had been running against a retired queue for months, and two approved changes that never reached production at all. None of it had caused an outage. All of it had been sitting there quietly. What follows is how that team learned to reconcile regularly, and what a plain weekly matching habit did for the way they ran AIOps.

Where the drift started

Nothing was careless about the team. The drift came from the normal way operations work moves. A ticket described what someone intended to do. The pipeline or an engineer did something close to that. An automated runbook then reacted to what it saw. Three records existed for one event, written at three different moments by three different actors, and nobody ever laid them side by side.

The senior engineer, Meera, described it in the review as “three diaries that all sound plausible.” The ticket said a node group would be resized on a Tuesday. The deployment log showed the resize happening on Wednesday, in a different region than first planned, after a second engineer adjusted the plan in chat. The automation then logged a scale-up event that only made sense against the new layout. If you read any single record, it looked fine. Only the comparison revealed that the ticket was now wrong, the monitoring thresholds still pointed at the old layout, and the runbook was quietly acting on stale assumptions.

The first matching session

The team did not buy anything or launch a programme. They booked one hour on a Friday and printed three lists: approved changes from the ticketing system, deployment events from the pipeline, and automated actions from the AIOps platform. Then they walked down the approved changes one at a time and asked a single question of each: what did we expect, and what actually happened?

The session produced a short list of gaps, and it is worth naming the kinds, because they repeat in almost every estate:

• Planned but never done. Two approved changes had no matching deployment. One had been superseded by a different fix, and nobody closed the ticket. The other had simply been forgotten.
• Done but never planned. A hotfix had gone out during an incident with only a chat message as its record. The replica nobody remembered belonged to the same night.
• Done differently from plan. The resize that moved regions was the clearest case. The outcome was reasonable; the paper trail was not.
• Automated actions with no owner. The clean-up job had a name, a schedule and a healthy-looking history, but pointed at a queue that no longer carried traffic.

Each gap was small when found. The forgotten change took ten minutes to schedule properly. The orphaned replica was shut down after a short check with the owning team. The clean-up job was repointed. Had the same list appeared six months later, after more drift piled on top, untangling it would have meant guessing at intent from people who had moved on.

Why the rhythm matters more than the tooling

Meera was firm about one point in the retrospective: the value came from the rhythm, not from the first session. A one-off cleanup finds a pile of gaps and feels productive, then the drift resumes. A fixed weekly slot keeps the pile from ever forming, because the gap count each week reflects only seven days of drift.

The team settled on a few rules that kept the habit light:

1. The same time every week, with the same two or three people, so it never needs scheduling.
2. Compare expectation to outcome, not tool to tool. The question is always “what did we say would happen” against “what did happen”.
3. Every gap gets an owner and a closing note before the meeting ends, even if the note is “intentional, ticket updated.”
4. Gaps that repeat become a process fix. If the same kind of mismatch shows up three weeks running, the workflow is wrong, not the people.

The fourth rule changed more than expected. Hotfix changes kept turning up without tickets, so the team added a lightweight after-the-fact ticket step to their incident closure checklist. Within a few weeks the “done but never planned” category had nearly vanished from the weekly list.

What it did for the AIOps layer

The team had been reading about using machine learning to inform operational decisions, and the exercise exposed an uncomfortable truth: models and automated runbooks are only as good as the records they consume. If the configuration data says a service lives in one region and it has moved, correlation and remediation logic will reason from fiction. Matching intent against reality was, in effect, a data quality job for their automation.

It also sharpened their view of automation scope. When they had first started introducing AIOps to the wider IT group, the conversation centred on what the platform could detect. After a month of weekly matching, the conversation shifted to what the platform had actually done, and whether each action matched an expectation someone had written down. A runbook that fires correctly but against the wrong target is worse than one that does nothing, because it looks like success.

Predictive features benefited too. A forecast of disk pressure built on a node group that had since been resized produced neat, confident, wrong numbers. Once the weekly session kept the inventory aligned with the real layout, the team’s own scepticism about those outputs softened, and the operations notes on stopping failures before they surface through predictive maintenance started to match their lived experience instead of contradicting it.

The money side of the same habit

About two months in, the finance lead joined the Friday session for a single week, and the same logic turned out to apply to the billing side. Cloud vendors, tooling vendors and contractors all send invoices that describe what was supposedly delivered, and the team’s own records describe what was actually used or received. Where the two disagree, somebody either overpays or underpays quietly. For the invoicing and settlement end of the loop, finance teams often rely on payment and invoice software; PayManagerPro sits in that space, with invoices, shareable payment links and a reconciliation step for matching what came in against what was billed. Finance runs the same check the engineers run on tickets, only with money.

A starter routine you can copy

If your own estate feels slightly out of focus, you do not need a transformation. Try this for one month:

• Pick one source of intent (approved changes are the easiest) and one source of outcome (deployment or audit events).
• Block a fixed weekly slot, short enough that nobody resists it.
• Walk the intent list top to bottom, marking each item matched, mismatched or missing.
• Assign every mismatch an owner and a closing note on the spot.
• After four weeks, look at which mismatch type recurs and fix the workflow behind it.

The logistics team still holds the session. It has shrunk from an hour to about twenty minutes, because there is simply less drift to find each Friday. Looking back over the quarter, Meera put it bluntly: the tools were never lying to them, they were just never being asked to agree with each other. Asking them to agree, a little at a time and on a fixed schedule, turned out to be the cheapest reliability work the team did all year.