How an Ops Team Moved Its AIOps Experiments Into the Open

Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!

We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!

Learn from Guru Rajesh Kumar and double your salary in just one year.


Get Started Now!

The anomaly model had been “almost ready” for eleven weeks. That was the phrase the small operations team used whenever somebody asked, and for a long time nobody questioned it. What follows is how the group dropped that phrase, began working in the open and learned how much of its own trouble had been invisible to everyone, including itself.

The quiet months

The team looked after monitoring for a mid-sized product company. Their goal was modest: group the flood of alerts into a handful of incidents so the on-call engineer could read the night’s trouble in a minute instead of an hour. One engineer built a prototype on a laptop, a second tuned thresholds in a notebook, and a third kept a spreadsheet of “weird alerts” that nobody else had seen.

Each of them was productive. Together they were a closed room. The prototype’s training data was filtered in a way only its author remembered. The notebook contained a fix for a timestamp bug that the other two were still working around. The spreadsheet described a recurring false alarm that the prototype had quietly learned to treat as normal. Three people were solving the same problem three separate times, and the only thing the rest of the company saw was a weekly status line saying work continued.

The afternoon it surfaced

The turning point was small. During a post-incident chat, a database administrator mentioned that the nightly job always spiked disk latency at the same minute. The spreadsheet owner went pale: that was row 31, logged six weeks earlier, never shared. If the prototype’s author had seen it, the model would have been told about the job instead of flagging it every night as an unusual event.

Nobody was blamed. The team simply asked a different question: what would it take for anything we know to be findable by anyone who might need it? The answer they settled on needed no software budget; it was a habit of leaving work where others could look at it while it was still unfinished.

What “working in the open” meant for them

They kept the definition deliberately plain, and wrote it on one page that sat at the top of their shared tracker.

  • Every experiment gets a visible card the day it starts. The card names the question, the data used and the person who is looking at it, even when the answer is a shrug.
  • Dead ends stay on the board. A failed attempt is moved to a “tried, did not work” column with two sentences about why, rather than deleted.
  • Half-finished notebooks live in the shared repository, not on a laptop, with a line up top admitting which part is broken.
  • Odd observations go on a public notes page the moment they are noticed, not when someone is sure they matter.

None of this needed permission or budget. It needed the team to accept a mild discomfort: other people would now see the mess in the middle.

The first two weeks were awkward

Honest storytelling means admitting that the change felt worse before it felt better. The engineer with the prototype admitted that reading the shared board made every unfinished thing look embarrassing. Cards piled up with vague titles. A reviewer wondered aloud whether the board was becoming a performance rather than a tool.

The fix was to shrink the unit of sharing. Instead of posting a polished summary once a week, people posted one line at the moment of a decision: “dropping the seasonal term, it is hiding the Monday pattern” or “can’t reproduce the latency spike with last week’s data.” Short entries cost almost nothing to write, and they turned out to be exactly what a teammate needed in order to avoid repeating a mistake.

If that kind of habit is new to you, the team’s experience lines up with what a good walkthrough of smarter event noise reduction suggests: grouping alerts only works when the people who know the quirks of each system tell the model what they know.

What changed once the work was visible

Within a month, three things had happened that nobody planned.

First, the duplicate effort disappeared. When the second engineer started on a cleaning routine for log timestamps, the board showed that a card for the same fix already existed, and they paired for an hour instead of competing for a week.

Second, outsiders began to contribute. A network engineer who had never touched the project scrolled the notes page, spotted a pattern in a switch firmware rollout and left a comment that explained a whole cluster of “random” alerts. The model never needed to learn that cluster, because the rule that handled it was thirty seconds of configuration.

Third, and most surprising, trust moved. The product managers who used to ask for an estimate now simply read the board. Because they could see the dead ends, they believed the successes. The phrase “almost ready” retired itself; people could see exactly how ready it was.

A small detour into everyday life

The same instinct shows up far from server rooms. Think about how people look for something ordinary, like a replacement part or a local service. They are far happier when they can see what is on offer before they set out. One example is BuyMLocal, a site for hunting down items and services close to home, which lists what neighborhood shops say they have on hand so a shopper can plan the outing. It is the same idea: put the state of things where others can read it.

Lessons the team wrote down

After a quarter, the team added a short retrospective to the tracker. Their notes were practical rather than inspirational.

  1. Share at the moment of the decision, because memory edits the story afterward.
  2. Write the reason, not just the result. A card that says “dropped” helps nobody; a card that says “dropped, it doubled the false alarms on Mondays” helps everybody.
  3. Keep the format forgiving. If posting takes more than two minutes, people will quietly stop.
  4. Let visitors comment, but name one person who reads the comments each morning so nothing is ignored.
  5. Review the board in the weekly meeting by scrolling it together, rather than preparing slides.

Where the model ended up

The anomaly grouping did eventually go live, about five weeks after the habit changed, which was faster than the previous eleven weeks of “almost.” It was not magical. It still missed some incidents and sometimes merged two unrelated ones. The difference was that the misses were visible: a card existed for each, with a person and a next step attached, and the on-call engineers knew where to look.

That visibility also helped when the team moved on to a bigger question. Their notes from the earlier period fed straight into a later plan, much like the steps described in an easy-to-follow primer on rolling AIOps out to a team, because the lessons were already written down rather than locked in three heads.

Try it on a small scale

You do not need to reorganize a department to borrow the idea. Pick one active experiment this week, give it a visible card, and write three lines: what you are trying, what you have seen so far, and what is currently stuck. Put it somewhere your neighbors in the organization can read without asking. Then notice who stops by.

Nobody here got cleverer overnight. They just stopped paying a hidden tax: repeated work, unshared warnings and slow rumors of progress.

Related Posts

A Teardown Checklist for a Finished Drift-Monitoring Project

Open a small, finished model-monitoring repository and audit how it was built, from baseline and thresholds to tests and alert wording, then borrow its structure.

Read More

Eight Runbook Phrases Worth Rewriting in Plain Words

Eight bits of insider shorthand common in incident runbooks, and the everyday wording that lets a tired on-call reader act on each one without asking anyone.

Read More

AIOps With a Tiny Toolbox: Three Triage Builds Compared

A storytelling comparison of three incident triage builds made under one strict limit: a single dependency, a small codebase and one shared set of past incidents.

Read More

Spotting Recurring Incident Patterns: An AIOps Explainer

One alert is noise, three of the same kind is a habit. A plain-language look at how AIOps tooling and a simple tally reveal the recurring causes behind repeat incidents.

Read More

Why AIOps Teams Are Slowing Rollouts Down on Purpose

A friendly look at a quiet trend in AIOps: teams letting each automated change prove itself slowly, because the fast version keeps ending in a rollback.

Read More

How to Keep a Done List for AIOps Changes, Step by Step

A six-step method for recording AIOps changes that have shipped and been verified, so your team stops repeating work and can judge progress from evidence.

Read More
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x