Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!
We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!
Learn from Guru Rajesh Kumar and double your salary in just one year.
Every few months a platform team somewhere decides that the answer to a noisy, confusing production estate is a shiny AIOps product. This is the story of a small operations group that nearly did exactly that, and what happened when someone suggested they first write down everything they already ran. It is a composite sketch, not a real company, but anyone who has sat in an incident review will recognise the beats.
The meeting that almost ended in a purchase order
The team looked after a mid-sized web platform: a handful of services, two databases, a message queue and the usual tangle of scheduled jobs. Pages were frequent, and the on-call engineer on a bad night could see forty notifications before they understood what was actually broken. In a Thursday planning meeting, the manager said what everyone was thinking: “We need something smarter that correlates all this for us.”
Heads nodded. A shortlist of vendors was already half-formed in people’s browser tabs. Then the most junior engineer, a few months into the job, asked a question that sounded almost too simple. “Do we know what we already have that does correlation?”
Nobody did. That silence is the real beginning of the story, because it exposed that the team was about to shop for a capability without knowing whether it already owned a version of it.
Step one: a plain list, no judgement
The manager gave the junior engineer a week and one rule: list, do not evaluate. No scoring, no opinions about whether something was good. Just a spreadsheet with four columns: what it is, who set it up, what it watches, and who looks at it.
The list started where you would expect, with the monitoring stack everyone knew about. Then it kept going. Over the week, the engineer opened every corner of the estate and wrote down what turned up:
- Three separate dashboards that nobody had mentioned in a year, one of them still wired to a service that had been retired.
- A cloud provider’s built-in anomaly detection, switched on at account creation and never read by anyone.
- A handful of shell scripts in a repository called “misc” that restarted stuck workers when a queue backed up.
- Two alert-routing rules in the paging tool that already grouped related notifications, though only for one service.
- A log search tool whose saved queries held years of hard-won knowledge about how past failures looked.
None of this was glamorous. But when the spreadsheet was projected on the wall at the next meeting, the room’s mood changed. The team was not starting from an empty field. It was sitting on a half-built version of the thing it wanted to buy.
What the inventory revealed about the real gap
Here is the part that surprised people. The list did not say “you have everything you need.” It said something more useful: it showed exactly where the holes were. Correlation within a single service already existed. What nobody had was a way to connect events across the queue and the databases, because those lived in different tools owned by different people.
That was a far narrower problem than “we need AIOps.” It was closer to “two of our tools do not talk to each other, and one alert rule is missing.” Before the inventory, the team had described a swamp. After it, they could point to a specific gap on a map.
There was a second finding as well. Several of the scripts and rules duplicated each other, built at different times by people who had left. Two of them fought: one restarted a worker, the other paged a human about the restart. The engineers had been living with that conflict for so long that it had become part of the background noise.
Turning the list into a short plan
The group did not rush. They sorted the findings into three honest piles: things to switch on, things to connect and things that were genuinely missing. The first pile turned out to be the longest. Reading the provider’s anomaly features and tuning the existing routing rules cost a couple of afternoons, not a budget cycle.
If you want to see how other teams think about this kind of work, the notes on cutting down event noise using smarter tooling sit on this very site, and they pair well with an inventory like the one above. The sorting step matters because it keeps you from treating every finding as equally urgent.
The “genuinely missing” pile ended up with exactly one item: a thin layer to carry context between the queue and the database tools. That is a much smaller conversation with a vendor, or with your own engineers, than a platform-wide replacement.
A small detour for the curious
If you like reading how other people write about running complex systems, there is a blog-style site called RobotsOps, a website about robotics operations, which covers a different corner of the operations world from the one this team lived in. Reading outside your own niche is a nice way to notice habits you take for granted, including the habit of never counting what you own.
What changed in the team’s habits
The inventory did not stay a one-off. The team kept the spreadsheet alive, and they added one line to their process: whenever a new tool is proposed, someone first checks the list for anything that overlaps. Sometimes the answer is “nothing here, go ahead.” More often it is “we already have most of this, we just never turned it on.”
A few practical habits came out of the exercise, and they travel well to other teams:
- Give the list an owner, even if it is only a rotating duty, so it does not rot.
- Record who actually reads each dashboard and alert. A tool nobody looks at is a finding in itself.
- Note the date each item was last touched, which makes stale things easy to spot.
- Keep the format boring. A shared spreadsheet beats an elaborate wiki page that nobody opens.
The manager later admitted that the biggest benefit was not saving money. It was that the team stopped arguing about hypothetical tools and started talking about specific, visible gaps. Arguments about a blank page go on forever; arguments about a row in a table usually end by lunchtime.
Where this leaves the AIOps question
Did the team eventually adopt something new? A little, yes. They added a modest component to move context between the queue and the databases, and they retired two of the three forgotten dashboards. But the headline is that they arrived at that decision with a map in hand. If you are thinking about bringing AIOps into your IT team, the groundwork laid out there is far easier to follow once your own estate is written down.
To get a feel for what a live incident is like from the inside, try the primer on live troubleshooting for newcomers, which makes a handy sidekick to this story. You will notice how much of real-time troubleshooting depends on knowing what you can already see.
Try it this week
Skip the vendor call and the workshop. Open a blank sheet, give yourself a few days, and list every monitor, dashboard, script, alert rule and scheduled job you can find. Ask who set each one up and who reads it. Then look at the result and ask the junior engineer’s question: what do we already have that does this?
You might still end up buying something. But you will buy it knowing the exact shape of the hole, and that is a much better place to negotiate from than a feeling that things are noisy.