Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!
We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!
Learn from Guru Rajesh Kumar and double your salary in just one year.
The anomaly model had been “almost ready” for eleven weeks. That was the phrase the small operations team used whenever somebody asked, and for a long time nobody questioned it. What follows is how the group dropped that phrase, began working in the open and learned how much of its own trouble had been invisible to everyone, including itself.
The quiet months
The team looked after monitoring for a mid-sized product company. Their goal was modest: group the flood of alerts into a handful of incidents so the on-call engineer could read the night’s trouble in a minute instead of an hour. One engineer built a prototype on a laptop, a second tuned thresholds in a notebook, and a third kept a spreadsheet of “weird alerts” that nobody else had seen.
Each of them was productive. Together they were a closed room. The prototype’s training data was filtered in a way only its author remembered. The notebook contained a fix for a timestamp bug that the other two were still working around. The spreadsheet described a recurring false alarm that the prototype had quietly learned to treat as normal. Three people were solving the same problem three separate times, and the only thing the rest of the company saw was a weekly status line saying work continued.
The afternoon it surfaced
The turning point was small. During a post-incident chat, a database administrator mentioned that the nightly job always spiked disk latency at the same minute. The spreadsheet owner went pale: that was row 31, logged six weeks earlier, never shared. If the prototype’s author had seen it, the model would have been told about the job instead of flagging it every night as an unusual event.
Nobody was blamed. The team simply asked a different question: what would it take for anything we know to be findable by anyone who might need it? The answer they settled on needed no software budget; it was a habit of leaving work where others could look at it while it was still unfinished.
What “working in the open” meant for them
They kept the definition deliberately plain, and wrote it on one page that sat at the top of their shared tracker.
- Every experiment gets a visible card the day it starts. The card names the question, the data used and the person who is looking at it, even when the answer is a shrug.
- Dead ends stay on the board. A failed attempt is moved to a “tried, did not work” column with two sentences about why, rather than deleted.
- Half-finished notebooks live in the shared repository, not on a laptop, with a line up top admitting which part is broken.
- Odd observations go on a public notes page the moment they are noticed, not when someone is sure they matter.
None of this needed permission or budget. It needed the team to accept a mild discomfort: other people would now see the mess in the middle.
The first two weeks were awkward
Honest storytelling means admitting that the change felt worse before it felt better. The engineer with the prototype admitted that reading the shared board made every unfinished thing look embarrassing. Cards piled up with vague titles. A reviewer wondered aloud whether the board was becoming a performance rather than a tool.
The fix was to shrink the unit of sharing. Instead of posting a polished summary once a week, people posted one line at the moment of a decision: “dropping the seasonal term, it is hiding the Monday pattern” or “can’t reproduce the latency spike with last week’s data.” Short entries cost almost nothing to write, and they turned out to be exactly what a teammate needed in order to avoid repeating a mistake.
If that kind of habit is new to you, the team’s experience lines up with what a good walkthrough of smarter event noise reduction suggests: grouping alerts only works when the people who know the quirks of each system tell the model what they know.
What changed once the work was visible
Within a month, three things had happened that nobody planned.
First, the duplicate effort disappeared. When the second engineer started on a cleaning routine for log timestamps, the board showed that a card for the same fix already existed, and they paired for an hour instead of competing for a week.
Second, outsiders began to contribute. A network engineer who had never touched the project scrolled the notes page, spotted a pattern in a switch firmware rollout and left a comment that explained a whole cluster of “random” alerts. The model never needed to learn that cluster, because the rule that handled it was thirty seconds of configuration.
Third, and most surprising, trust moved. The product managers who used to ask for an estimate now simply read the board. Because they could see the dead ends, they believed the successes. The phrase “almost ready” retired itself; people could see exactly how ready it was.
A small detour into everyday life
The same instinct shows up far from server rooms. Think about how people look for something ordinary, like a replacement part or a local service. They are far happier when they can see what is on offer before they set out. One example is BuyMLocal, a site for hunting down items and services close to home, which lists what neighborhood shops say they have on hand so a shopper can plan the outing. It is the same idea: put the state of things where others can read it.
Lessons the team wrote down
After a quarter, the team added a short retrospective to the tracker. Their notes were practical rather than inspirational.
- Share at the moment of the decision, because memory edits the story afterward.
- Write the reason, not just the result. A card that says “dropped” helps nobody; a card that says “dropped, it doubled the false alarms on Mondays” helps everybody.
- Keep the format forgiving. If posting takes more than two minutes, people will quietly stop.
- Let visitors comment, but name one person who reads the comments each morning so nothing is ignored.
- Review the board in the weekly meeting by scrolling it together, rather than preparing slides.
Where the model ended up
The anomaly grouping did eventually go live, about five weeks after the habit changed, which was faster than the previous eleven weeks of “almost.” It was not magical. It still missed some incidents and sometimes merged two unrelated ones. The difference was that the misses were visible: a card existed for each, with a person and a next step attached, and the on-call engineers knew where to look.
That visibility also helped when the team moved on to a bigger question. Their notes from the earlier period fed straight into a later plan, much like the steps described in an easy-to-follow primer on rolling AIOps out to a team, because the lessons were already written down rather than locked in three heads.
Try it on a small scale
You do not need to reorganize a department to borrow the idea. Pick one active experiment this week, give it a visible card, and write three lines: what you are trying, what you have seen so far, and what is currently stuck. Put it somewhere your neighbors in the organization can read without asking. Then notice who stops by.
Nobody here got cleverer overnight. They just stopped paying a hidden tax: repeated work, unshared warnings and slow rumors of progress.