Why Alert Rules Drift: A Decision Log Fix for AIOps Teams

Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!

We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!

Learn from Guru Rajesh Kumar and double your salary in just one year.


Get Started Now!

Open any mature monitoring stack and ask a simple question: why is this correlation window set to ten minutes, and who decided that suppression rule should exist? Most of the time the honest answer is a shrug. The settings work, or seem to, and nobody remembers the reasoning. That gap is the problem, and a decision log is the cheapest fix we know for it.

This piece diagnoses the symptom, traces it to a cause, and then walks through a repair you can apply in an afternoon. The examples come from AIOps work, but the pattern holds for any team that tunes automated systems.

The symptom: configuration with no memory

The pattern shows up in a few recognisable forms. An anomaly detector fires on a service that was retired last quarter. A grouping rule merges two alert families that nobody can explain. A newcomer proposes loosening a threshold, and the room splits between people who half-remember a bad incident and people who remember nothing.

None of these are technical failures. The tooling does exactly what it was told. What is missing is the record of why it was told that, and what the author expected to happen afterwards. Without that record every review turns into an argument from vague memory, and the loudest recollection wins.

If you have recently been working on trimming event volume, the article on quieting noisy IT events using smart technology handles the mechanics of that job. Our concern here is narrower: how to make sure the reasoning behind each tuning choice survives the person who made it.

Diagnosis: three causes worth separating

The reasoning lived in a chat thread

Most tuning decisions get made in a hurried conversation during or just after an incident. The outcome lands in the config repository as a one-line commit. The discussion stays in a channel that gets archived, searched once, and forgotten.

Expectations were never stated

Even when the reasoning is saved, the prediction usually is not. Nobody writes down “we expect paging volume on this service to fall noticeably within two weeks, and if it does not, the rule was wrong.” Without a stated expectation there is nothing to check against later, so a bad call can sit in production for years because it never visibly failed.

Hindsight edits the story

When people do look back, they unconsciously rewrite what they believed at the time. A rule that happened to work becomes obviously wise; one that misfired becomes obviously reckless. A dated entry, written before the outcome was known, stops that quiet revision.

The repair: a decision log with four fields

Keep it boring. A plain markdown file in the same repository as the configuration is enough, with one short entry per meaningful tuning or architecture choice. Each entry carries four fields.

  1. The decision. One sentence: “Group alerts from the checkout services by dependency, using a fifteen-minute window.”
  2. The options considered. Two or three alternatives, including “do nothing”, with a line on why each lost.
  3. The reasoning and the expectation. What evidence pushed you here, and what measurable change you expect, with a date to check it.
  4. The review note. Left blank at first. Filled in when the date arrives.

That last field is what turns a log from an archive into a learning tool. The review note is where you record what actually happened, in plain language, next to the prediction you made earlier.

A worked example from an event-correlation project

Suppose a platform team is choosing between two strategies for grouping related alerts: time-based windows, or grouping driven by a service dependency map. They pick the dependency map because their incidents tend to cascade along service paths rather than cluster in time.

The log entry for that choice would read something like this. Decision: adopt dependency-based grouping for the payments tier. Options: fixed time windows, which are simple but split cascades that span slow services; manual grouping rules, which rot quickly; dependency mapping, which needs an accurate topology. Reasoning: the last three incidents all propagated downstream, and each produced a burst of separate pages. Expectation: fewer separate pages per incident by the next quarterly review, and a requirement to keep the dependency map current. Review date: the first Monday of next quarter.

Three months later the review note might say that page counts did drop, but that the topology went stale after a migration and two incidents were split anyway. That is a useful result. The decision was reasonable, the follow-through on the map was not, and the next entry can address the maintenance cost directly instead of blaming the algorithm.

For readers who want the mechanics behind tools of this kind, the explainer on how automated root cause analysis uses topology and observability data gives useful background.

Where shared platforms fit into this

The same habit applies to any architectural choice, not just alert tuning. Consider how a consumer marketplace such as MotoShareGo, a motorbike and scooter rental-and-sharing website, lets people sign in through a central account service rather than running a login system of its own. Whether to centralise identity is exactly the sort of call that deserves an entry: the options, the reasons, and the expected consequences, written while everyone still remembers what the alternatives looked like.

Making the log stick

Keep entries short

Once a single entry drags past ten minutes of effort, people quietly stop writing them. Aim for five to eight lines. The goal is to capture the reasoning, not to produce a design document. Longer proposals can live elsewhere and be linked.

Write it at the moment of the decision

The entry belongs in the same pull request as the change. Reviewers can then ask “what do you expect to happen?” before approving, which improves the change itself. An entry written a month later is a reconstruction, and reconstructions drift.

Schedule the review

Put review dates on the team calendar, or add a recurring ticket. A log with no review habit is just a diary. Ten minutes at the start of each quarter, reading through entries that have come due, is enough to learn which kinds of calls tend to hold up.

Do not grade people

The point of the review note is to improve judgement about the system, not to score the author. A team that uses the log to assign blame will soon find the log empty. Treat a wrong call as data about the situation, such as a missing signal or a stale assumption.

What to look for when you review

After a few quarters of entries, patterns appear. You may find that decisions made during incidents are overturned more often than decisions made during calm periods. You may find that expectations tend to be optimistic about how quickly improvements show up. You may find that reversals cluster around a particular tool or integration.

Each pattern points at something concrete. Rushed calls suggest adding a cooling step before changes are merged. Optimistic timelines suggest lengthening review windows. Clusters around a tool suggest that the tool itself is poorly understood. None of that is visible without the written record.

Teams working on adjacent practices, such as machine learning for alert management in IT operations, tend to find the difficult step is less the model than recalling why its thresholds were set where they are. A decision log addresses that directly.

A starter template

Copy this into a file called decisions.md and adapt it.

  • Date and area: when the call was made and which system it touches.
  • Decision: one plain sentence.
  • Options weighed: two or three, with a short reason each lost.
  • Why this one: the evidence and constraints at the time.
  • Expected result and check date: something you could observe.
  • Review note: filled in later, never edited retroactively.

Resist the urge to revise old entries when you learn something new. Add a fresh entry that references the earlier one instead. The sequence of entries is the history, and the history is the whole value.

Closing thought

Alert rules drift because the reasons behind them evaporate. A short, dated, honest log keeps those reasons alive, gives every review something solid to test against, and lets a team improve its judgement about its own systems over time. Start with the next tuning change you ship: write four lines, set a date, and look again when it arrives.

Related Posts

AIOps Team Weekly Themes: Your Questions Answered

Practical answers on giving each week of an AIOps team one theme, from alert tuning to data cleanup, and what to do with the work that arrives off-theme.

Read More

A Seven-Day Changelog Trial for an AIOps Team

A platform team could not explain why their AIOps grouping changed. They tried one-line change notes for a week and then decided what to keep.

Read More

How to Sketch an AIOps Pipeline on Paper Before Any Code

A step-by-step walkthrough of drawing an AIOps pipeline by hand first, so the gaps show up while a fix still costs one eraser stroke instead of a sprint.

Read More

Findable Engineering Notes for AI Ops Beginners: A Mini-Glossary

A junior analyst lost forty minutes hunting for an old fix. A beginner-friendly glossary shows how titles, tags and a weekly tidy make engineering notes findable.

Read More

A Parking-List Audit for AIOps Teams in Release Week

Seven yes-or-no questions to test whether your team’s not-now list protects release focus or just collects dust.

Read More

Real Breaks in AIOps Teams: Where the Practice Stands Now

A look at how AIOps and platform teams treat real breaks during long incidents, what the evidence suggests, and where handovers still fall short.

Read More
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x