Ask Why Five Times: A Step-by-Step Guide to AIOps Root Causes

Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!

We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!

Learn from Guru Rajesh Kumar and double your salary in just one year.


Get Started Now!

Every on-call engineer knows the pattern. A service falls over at 2 a.m., somebody restarts it, the graphs go green, and everyone goes back to sleep. Three weeks later the same alert fires again. The restart was never a fix; it was a pause. If you work in AIOps, this loop is the thing your tooling is supposed to break, and the cheapest method I know for breaking it is older than any machine learning model: ask “why?” again and again until the answer stops being a symptom. This is a practical walkthrough of how to run that questioning on an incident, step by step, with an AIOps workflow in mind.

Step 1: Write the problem as one plain sentence

Before you ask anything, pin down what actually happened. “The model is flaky” is not a problem statement. “The anomaly detector paged the on-call twice on Tuesday for a latency spike that was a scheduled batch job” is. Keep the sentence factual, put a time on it, and leave out blame. If two people on the team write two different sentences, you have already found your first lesson: the team does not share a picture of the incident.

A good test is whether a colleague who was not there could read the sentence and know what to look at. If they cannot, rewrite it until they can.

Step 2: Ask the first why and expect a symptom

Why did the detector page twice? Because it saw a latency jump that crossed its threshold. That answer is true, and it is nearly useless. It describes the mechanism of the alert, not the reason the alert was wrong. The first answer is almost always a restatement of the symptom in slightly more technical clothes, so do not stop here. Write it down and keep going.

Many post-incident reviews finish at this point. The team tunes the threshold, closes the ticket, and calls it learning. Nothing upstream changed, so the next odd spike will page again.

Step 3: Ask why a second and a third time

Why did a latency jump cross the threshold? Because the nightly batch job ran inside the window the detector treats as normal traffic. Why was the job inside that window? Because nobody told the detector about the schedule. Why not? Because job schedules live in one team’s scheduler and the detector’s baseline is trained by another team on whatever metrics it can see.

Notice what happened across those three answers. The first was about numbers, the second was about timing, and the third was about who knows what. You moved from the graph to the organisation. That change of scenery is what makes the method useful for problems a model cannot see on its own.

  • Keep each answer to one sentence. If an answer needs a paragraph, you are mixing two causes. Split them and follow each branch separately.
  • Answer from evidence, not memory. Open the scheduler, the config repository or the ticket history while you answer. A guess that sounds right is how a team talks itself into the wrong fix.
  • Stay on one chain at a time. Branches are fine, but finish one before starting another or the notes turn into a mind map nobody can act on.

Step 4: Push to the fourth and fifth why

Why do the two teams not share schedule information? Because there is no agreed place where a recurring job gets registered. Why not? Because when the scheduler was introduced, the review checklist covered security and cost but never asked how the job would show up in observability data. That fifth answer is a gap in a process, and process gaps are the only kind of cause you can fix once and have the fix hold.

Five is a guide, not a rule. Sometimes you hit the root at three, sometimes you need seven. The signal that you have arrived is simple: the answer names something a person could change on Monday morning, such as a checklist item, an ownership line or a required field, and it no longer names a number on a dashboard.

Step 5: Turn the root cause into one concrete change

Resist the urge to write a long action list. One incident usually deserves one real change, sized so that somebody can finish it this sprint. In the example above it might be a single new line in the scheduler review checklist: “Does this job’s run window appear in the baseline calendar the anomaly tooling reads?” Add an owner and a date, and put the change where the team will actually bump into it, not in a retrospective document that gets archived.

If you want a complementary angle on this, a related article covers taming IT event noise using smart technology, and the root-cause habit pairs well with it because noisy alerts are a classic place where the first why hides a missing process step.

Step 6: Check whether the fix reaches the original symptom

Go back to your one-sentence problem. If the change had been in place on Tuesday, would the double page have been avoided? Walk the chain forward in your head, from the checklist to the registered schedule to the baseline to the detector. If any link is shaky, you probably stopped one why too early. Add the missing question and try again.

It also helps to ask what a similar-looking failure would do under the new arrangement. If the fix still holds, it is a root-cause fix. If it only covers the exact job you examined, it is a patch wearing a root cause’s clothing.

Where AIOps tools help and where they do not

Current AIOps platforms handle the early part of the chain well. They correlate events, surface the changed deploy, and get you from a raw page to a workable hypothesis sooner. That covers roughly the first two whys. The later ones, about ownership, checklists and review habits, live in documents and conversations that no correlation engine reads. Use the tooling to arrive at question three quickly; it cannot ask the question for you. A related read on how teams use data to support choices is this overview of AI-assisted IT decision-making.

A note on running the session

Keep the session short and blameless. Fifteen to thirty minutes with the people who touched the incident is plenty. Use a shared document, write the question and answer pairs in a numbered list, and have one person act as the skeptic whose only job is to ask “is that the cause, or is that what the cause looked like?” Stop when the answer names a process, and never accept “human error” as an endpoint; it is a prompt for another why, namely why the process allowed that error to reach production at all.

The habit travels well beyond engineering. A website such as this orthopaedic hospital directory, which splits its content into hospital, doctor, forum and blog areas, faces the same puzzle when an entry goes stale: the useful question is less “who forgot?” than “why did our update routine allow the drift?”

Do this a handful of times and a pattern emerges: the repeat incidents quietly stop. Not because the models got smarter, but because the process underneath them finally got a question asked of it. That is a good return for half an hour of structured curiosity, and it costs nothing but the discipline to keep asking.

Related Posts

AIOps Hardest Task First: A Team’s Morning Case Study

A fictional AIOps team stops opening the day with small tickets and starts with the correlation rework it dreads. A case study in doing the hardest task first.

Read More

The Waiting-For List That Keeps AIOps Projects From Stalling

Most AIOps projects stall in the gaps between teams. A plain waiting-for list, with an owner and a nudge date per line, closes those gaps.

Read More

Task-List Done vs Outcome Done: An AIOps Finish Line

Two ways to close an AIOps ticket, compared side by side: ticking off steps versus proving a condition is true. Pick the finish line before the work starts.

Read More

Rubber-Duck Debugging for AIOps: A Beginner’s Primer and Glossary

Explaining a failing AIOps service aloud, line by line, exposes the assumption you stopped questioning. A worked example and a seven-term glossary for beginners.

Read More

Task Switching in AIOps: Questions and Answers on the Hidden Cost

A friendly FAQ on how jumping between alerts, chat and tickets quietly drains an AIOps day, and the small habits that make each return to a task cheaper.

Read More

Eight Small AIOps Fixes From Asking What Would Make It Easier

An AIOps team stopped tuning models for an afternoon and simply asked each other one question. Here are eight small fixes that surfaced from the answers.

Read More
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x