Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!
We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!
Learn from Guru Rajesh Kumar and double your salary in just one year.
Most AIOps programs stall in the same place: the planning meeting. Somebody sketches a platform that correlates every signal, predicts every outage and explains every incident, and then months pass while data sources are inventoried and nothing reaches an on-call engineer. This walkthrough takes the opposite route. You will build one small, honest first version of an AIOps capability, put it in front of the people who carry the pager, and let their reactions decide what comes next.
Why a small AIOps first version beats a grand design
An operations team learns about a new capability only when somebody uses it under real conditions. A diagram cannot tell you that engineers ignore a certain type of notification, or that the service owner you assumed was reachable changed teams in spring. A rough tool running for a week tells you both. Waiting until the model feels finished means the lessons arrive late, and by then the design has hardened around guesses.
So the goal of version one is not accuracy. It is contact with reality. You want something narrow enough to build in days, visible enough that people form opinions, and safe enough that being wrong costs nothing.
Step 1: Pick one service and one question
Choose a single service that someone already worries about. Then write down one question you wish the tooling answered for it, in plain words. Good examples are “is latency drifting upward over the week” or “which deploys were followed by a spike in errors”. Bad examples are “how healthy is production”, because nobody can check that answer against anything.
If you cannot name the person who will read the answer, pick a different question. A first version without a reader is just a science project.
Step 2: Use the plainest method that could work
Resist the urge to reach for the most sophisticated model. A rolling average with a threshold, or a comparison against the same hour last week, is often enough to produce a useful flag. The point is to get something producing output, because plain methods are easy to explain, easy to debug and easy to throw away.
Write down what the method cannot see. If your rule only watches one metric, say so in the first line of the output. Naming the blind spots early keeps the team from trusting the flag more than it deserves.
Step 3: Ship it where people already look
Do not build a new dashboard. Send the output to a channel or a ticket queue the team already reads. For a first version, a short daily message is plenty: what was flagged, the numbers behind it, and one line saying how the flag was produced. Keep the wording humble. “This looks unusual compared with last week” invites correction, while “anomaly detected” invites a shrug.
Run it in observation mode. It should never page anybody, change a setting or open an incident on its own during this stage. If you have worked on cutting down reducing the volume of IT event chatter with smarter tooling, you already know how quickly a new source of messages can annoy people, so keep the volume low and the tone calm.
Step 4: Collect reactions, not metrics
After a few days, talk to the readers. Ask three things. Which flags were useful, which were noise, and what did they expect to see that never appeared? Write the answers in a shared note and resist defending the tool. A flag that engineers call obvious is not a failure; it means your method works and the next step is a harder question. A flag they call baffling points straight at a missing input.
This is also the moment to look at how the output fits into real choices. The piece on how data-driven tooling can inform operational decisions is a useful reminder that a flag only matters if somebody can act on it, so ask what each reader did after seeing one.
Step 5: Improve one thing, then ship again
Pick the single most common complaint and fix only that. Maybe the threshold was too sensitive, or the message lacked a link to the right graph. Make the change, release it the same day, and tell the team what changed. Short loops build trust. A team that sees its feedback land within a day will keep giving feedback, and a team that waits a quarter will stop bothering.
Repeat this loop a handful of times before you widen anything. Only when the first service feels dull and dependable should you add a second service or a second metric.
Keeping the first version honest
A few habits protect you from the usual traps:
- Label it as early. Put “first version” in the message header so nobody mistakes it for a finished product.
- Keep a list of known gaps. Share it openly. It is the backlog, written by the people who will use the thing.
- Keep the off switch obvious. Anyone should be able to mute it without asking permission.
- Do not oversell forecasts. Predictive claims need a long history to test, so for now describe what happened and what looks unusual rather than what will happen.
That last point matters. Our own writing on avoiding overconfident prediction in IT systems argues that forecasts are the hardest part to earn trust for, which is another reason to start with plain observation.
A note for people who write about what they build
Shipping early applies to words as much as to code. If you want to document your first version as you go, a free place to publish a first post on a personal blogging subdomain lets you put a rough write-up live, then keep refining it as the tool and your understanding improve. By its own account, joining never asks for payment details, so it is a cheap way to test whether anyone finds your notes useful.
What done looks like for version one
You are not finished when the flags are accurate. You are finished with version one when three things are true: the readers know it exists, they have opinions about it, and you have a short list of changes ranked by how often people mentioned them. That list is worth more than a roadmap written in a vacuum, because every item on it came from somebody using the thing.
From there, the plan writes itself. Fix the loudest problem, ship again, and let the next question come from the team instead of from a slide deck. An AIOps capability grown this way ends up smaller than the grand design at first, but it ends up used, and used tools are the only ones that get better.