Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!
We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!
Learn from Guru Rajesh Kumar and double your salary in just one year.
Most teams that adopt machine-learning features for operations hit the same wall: the tool offers a dozen settings, nobody is sure which values suit their environment, and the vendor documentation describes defaults that were tuned for somebody else’s systems. The sensible way through is a small experiment. Ask a narrow question, alter a single setting, limit the damage a wrong guess could do, and record the outcome. This article explains, in plain terms, how that approach works for AIOps work.
What an AIOps experiment actually is
Strip away the jargon and an experiment is a controlled comparison. You have a behaviour you want to understand, such as how an anomaly detector reacts to a particular service. You pick a single setting you suspect matters. You alter it for a limited slice of your estate, for a limited time, and compare what you see against what you saw before or what a sibling system shows at the same moment.
A good experiment has four ingredients. First, a question phrased so the data can settle it one way or another. Second, one deliberate change. Third, a boundary: a single low-risk service, a single afternoon or a single shift. Fourth, a note of the result and the decision it leads to. If any of these is missing, you have tinkering rather than learning.
A concrete example: the lookback window
Anomaly detectors usually learn what normal looks like from a window of recent history. A short window adapts quickly but forgets the weekly rhythm of your traffic. A long window remembers the rhythm but reacts sluggishly to a real change. Teams often leave this setting at its default for months because nobody can say which value is right.
Here is how a small experiment would look. Choose one internal service whose failure would be an annoyance rather than an emergency, such as a reporting job or an internal wiki. Ask: does lengthening the lookback window make the detector flag fewer false alarms on this service without missing the incidents the on-call engineer already knows about? Change only the window length. Leave sensitivity, notification rules and everything else alone. Let it run for one working day, then compare the flagged events with the human log of what actually went wrong.
The outcome might be that the longer window hides a slow memory leak that the shorter one caught early. It might be that nothing changes at all, which is itself useful: you have learned that this setting is not the lever you thought it was. Either answer beats an argument in a meeting.
Why one change at a time matters
It is tempting to adjust the window, the sensitivity and the grouping rule in the same sitting. Resist that. When several settings move together and the output improves, you cannot say which one deserves the credit. When the output worsens, you cannot say which one to undo. A single change gives you a clean cause and a clean rollback.
The same logic applies to the data you feed the tool. If you also add a new log source during the test, the detector sees a different world and your comparison is spoiled. Freeze the inputs as far as you can, and note anything that changed outside your control, for example a deployment that landed mid-test.
Keeping it small and reversible
Small has a specific meaning here. It means a scope you could abandon in a minute without anyone outside the team noticing. Many monitoring platforms let you scope a rule to a tag, a namespace or a single host group. Use that capability. If your platform cannot scope changes that finely, run the experiment on a copy of the configuration in a sandbox or staging environment, replaying a day of recorded metrics.
Write the rollback step before you begin. It can be as short as “restore the previous window value from the saved config export”. Having it written down removes the hesitation that tempts people to leave a half-finished experiment running overnight.
Recording what you saw
You do not need a lab notebook, just a short template that you fill in the same way every time:
- The question, in one sentence.
- The single change and where it was applied.
- The start and end times.
- What the tool flagged, and what a human judged to be real.
- The decision: keep, revert or run a follow-up.
Teams tend to leave that final line blank, yet it is what converts an observation into progress. An experiment that ends with “interesting” has not finished. Pick a verdict, even a tentative one, and schedule the next small step.
Reading the results without fooling yourself
A single day is a small sample, and you should treat it that way. If the day happened to be unusually quiet, a calmer detector proves little. Ask whether the conditions were representative, and if in doubt, repeat the test on a different day. Compare like with like: weekday against weekday, and busy hours against busy hours.
Be wary of confirmation. If you hoped the longer window would help, you will notice every event that supports that view. Before looking at the results, write down what outcome would make you reject your idea. When the data lands on that side, accept it.
Where experiments fit with the rest of your tooling
Experiments are not a replacement for governance, and they should not touch customer-facing systems until you have enough confidence. They are most useful for tuning and understanding. If you are already working on reducing the volume of alerts your team receives, our piece on taming event noise with smarter tooling covers the wider picture, and an experiment such as the lookback test can be one concrete step inside that effort. Teams weighing how much to lean on automated recommendations might also read how machine-assisted analysis can support IT decisions before deciding which settings deserve a trial.
A note on looking outside your own field
Curiosity about how other hobbies and trades run their own trials pays off. As one example, DronesBee is a website devoted to drones, with articles and news, product listings, a store, a section for service providers, lookup pages for operators and pilots, and a discussion board for its community.
Common mistakes to avoid
Several traps catch newcomers. The first is choosing a service that matters too much, which makes everyone nervous and cuts the test short. The second is running the experiment without telling colleagues, so a flagged event during the test triggers confusion. Send a brief heads-up in the team channel naming the service, the window and the person to ask. The third is changing the success definition halfway through because the early results look unflattering.
The fourth mistake is the opposite of the others: never finishing. Put an end time on the calendar. When it arrives, stop, record, decide.
Building a small experiment backlog
Once the first test is done, you will find questions multiplying. Keep them in a plain list ordered by how cheap each test is and how much you would learn. Typical entries for an AIOps team include whether grouping related alerts by service changes triage time, whether a different seasonality setting suits batch workloads, or whether a seven-day baseline behaves differently from a fourteen-day one. Take the cheapest, most informative item next. Over a quarter, this list becomes a private manual of how the tool behaves in your estate, which is worth more than any generic benchmark.
A short checklist before you start
- Is the question answerable in one day or one shift?
- Is exactly one setting changing?
- Is the service low-stakes and the scope clearly bounded?
- Is the rollback written down?
- Do colleagues know?
- Do you know what result would change your mind?
If you can tick every line, you are ready. The first experiment will feel slow and a little ceremonial. By the third, it becomes a ten-minute habit, and your team starts asking “what small test could settle this?” before the debate begins. That shift in habit, more than any individual setting, is the real payoff.