Why AIOps Drift Checks Get Skipped, and a Cue That Fixes It

Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!

We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!

Learn from Guru Rajesh Kumar and double your salary in just one year.


Get Started Now!

Almost every team that runs anomaly detection or event correlation says the same thing in the first month: someone will look at the model’s behaviour every week. By the third month the calendar slot is gone, and the only time anyone examines AIOps drift checks is after a bad night. The tooling worked as designed. The habit never attached to anything.

This post treats that lapse as a problem to be diagnosed rather than a failing of discipline. The symptoms are easy to spot, the cause is usually the same, and the remedy is small enough to try tomorrow morning.

The symptom: a check that depends on memory

Picture a typical rule: “review the detector’s false-positive rate on Fridays.” It sounds responsible. In practice a Friday contains a release, a customer escalation and a half-staffed rota. The review is not rejected; it simply never wins the moment-to-moment contest for attention. Nothing signals that it is time.

Three signs usually appear together. The check has an owner on paper but no trigger. It takes longer than anyone can spare when it is finally attempted, because weeks of change have piled up. And when it is skipped, nothing visible breaks, so the skip is invisible and costs nothing until it suddenly costs a lot.

The diagnosis: no cue, too much scope

Habit research gives a tidy explanation. Actions that depend on remembering compete with everything else on your mind, so they fade. Actions that follow an event you already perform reliably borrow that event’s reliability. Researchers who study this call the pairing an implementation intention: “after X, I do Y”. The existing event is the cue, and the new action rides along.

The second flaw is scope. “Review the model” is vague and open-ended, so it feels like an hour of work and gets postponed. A step that takes under two minutes can be done even on a bad day, which is what makes it survive.

The cue most ops teams already have

Consider what happens every working day without fail: the morning stand-up. It has a clear end, usually a moment when people close the call and reach for a drink. That moment is the cue. The prescription is deliberately narrow: after the stand-up ends, open the detector’s recent-alerts view and look at the three alerts that fired most overnight. Ask one question about each: would a person have wanted to be woken for this?

That is the entire habit. It is not an audit, a report or a meeting. It is one glance, tied to a fixed event, with one question that produces a note only when the answer is surprising.

  • Cue: the stand-up call ends.
  • Action: open the recent-alerts view and read the top three overnight alert sources.
  • Output: a single sentence posted to the team channel only if something looks off; silence means all is fine.
  • Size: about ninety seconds, and never longer on a day you feel busy.

Why tiny beats thorough here

The instinct is to design the ideal check: compare score distributions, replay last week’s incidents, chart seasonality. Those are valuable, but they are poor first habits. A large routine fails on the first overloaded day and then carries the guilt of being abandoned. A tiny one is hard to refuse. After a few weeks the glance has become automatic, and that is when it is safe to let it grow by one step, such as also glancing at how many alerts were auto-suppressed.

There is an engineering parallel. Teams that run a quick smoke test after every commit rarely need a heroic regression hunt later, because small failures surface near the change that caused them. A morning glance does the same for detector behaviour: drift shows up as a slow change in what the top alert sources are, and a daily look catches it while it is still a nudge instead of a rewrite.

A worked example of the diagnosis in action

Imagine an operations group whose anomaly detector begins flagging a nightly batch job as unusual. Each flag is individually harmless. Because nobody owns a daily look, three weeks pass before someone notices that the batch job now accounts for most of the overnight pages. With the after-stand-up glance in place, the same pattern appears on day two. The note in the channel reads one line: “the batch window keeps showing up in the top three.” The fix, whether a threshold change or an exclusion, is then a ten-minute conversation instead of a post-mortem.

The checkpoint worth noticing is the sequence: cue, glance, note only on surprise. Nothing in it relies on motivation, and nothing requires the model to be fully understood by the person doing the glance.

Common failure modes and quick repairs

  • The cue is unreliable. If stand-ups are sometimes skipped, pick a steadier trigger, such as the first time you open the ticket queue each day.
  • The step grew. If the glance has become a ten-minute analysis, cut it back to the original question. Expansion is how the habit dies.
  • Nobody sees the results. A note appears only on surprise, so teammates learn that silence is healthy and a note is worth reading.
  • It belongs to one person. Rotate the glance among teammates weekly, so a holiday does not end the routine.

For teams still bringing a platform into daily work, our guide on getting an AIOps rollout established inside an existing IT team covers the surrounding adoption questions, and the walkthrough on reducing event noise with smarter tooling is a natural companion once your glance starts surfacing repeated sources.

Tying the habit to household routines as a thought experiment

It helps to notice that the same trick works outside work. Many people already attach small chores to a fixed point: filing a receipt right after unpacking the shopping, or wiping the counter right after the last plate. For people in Indian households who like to look up a brand or a service before buying, a website describing itself as a place for brand information and reviews on everyday products and services can be the thing they open right after the evening chai. The pattern is identical to the ops one: a dependable moment carries a small action that would otherwise be forgotten.

Making it stick for the first month

Write the sentence down in your team’s own words and keep it where the stand-up notes live: “After stand-up, I read the top three overnight alert sources.” The exact phrasing matters less than having one sentence that names the cue and the action together. Review it once after two weeks. If the glance is happening on at least four mornings out of five, leave it alone. If it is not, shrink it further before you blame anyone.

Also decide in advance what a skipped day means. The answer should be nothing: you resume the next morning without a catch-up session. A habit that demands makeup work punishes the very people who most need it, and it teaches the team that missing once means failing.

How to tell the habit is working

Three observable signs show that the glance has taken hold. People mention the overnight alerts unprompted during the day. The channel notes become shorter and more specific over time. And a new joiner, asked what happens after stand-up, can describe the routine without checking a document. If none of these appear after a month, the cue is probably the weak part, not the people.

What to take away

Drift checks lapse because they rely on memory and ask for too much. Pair one narrow action with an event your day already contains, keep it under two minutes, and speak up only when the glance turns up something odd. Once it is automatic, you will have a reliable early signal about your detector without ever scheduling a dedicated review.

Try it for ten working days before judging. Pick the cue today, write the sentence, and let tomorrow’s stand-up be the first trigger.

Related Posts

A Teardown Checklist for a Finished Drift-Monitoring Project

Open a small, finished model-monitoring repository and audit how it was built, from baseline and thresholds to tests and alert wording, then borrow its structure.

Read More

Eight Runbook Phrases Worth Rewriting in Plain Words

Eight bits of insider shorthand common in incident runbooks, and the everyday wording that lets a tired on-call reader act on each one without asking anyone.

Read More

AIOps With a Tiny Toolbox: Three Triage Builds Compared

A storytelling comparison of three incident triage builds made under one strict limit: a single dependency, a small codebase and one shared set of past incidents.

Read More

Spotting Recurring Incident Patterns: An AIOps Explainer

One alert is noise, three of the same kind is a habit. A plain-language look at how AIOps tooling and a simple tally reveal the recurring causes behind repeat incidents.

Read More

Why AIOps Teams Are Slowing Rollouts Down on Purpose

A friendly look at a quiet trend in AIOps: teams letting each automated change prove itself slowly, because the fast version keeps ending in a rollback.

Read More

How to Keep a Done List for AIOps Changes, Step by Step

A six-step method for recording AIOps changes that have shipped and been verified, so your team stops repeating work and can judge progress from evidence.

Read More
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x