AIOps Explained for One On-Call Engineer at 2 a.m.

Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!

We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!

Learn from Guru Rajesh Kumar and double your salary in just one year.


Get Started Now!

Most AIOps write-ups are addressed to “the organisation”, and that is exactly why people skim them. This one is addressed to a single reader: an on-call engineer, three months into the rotation, awake at a quarter past two with a phone buzzing on the nightstand. She has about ninety seconds before she must decide whether to get out of bed. Everything below is arranged as the questions she would actually ask, because AIOps earns its keep only if it answers her, not a committee.

What does AIOps mean to someone holding a pager?

Strip away the vendor vocabulary and AIOps is the use of statistical and machine-learning methods on operational data: metrics, logs, traces, tickets and change records. The goal is narrow. It should help a person notice a real problem sooner, understand its likely cause faster, and avoid chasing signals that do not matter.

For our engineer, that translates into three small promises. The first is that when something is wrong, one grouped notification arrives instead of forty. The second is that the notification carries context: what changed recently, which services sit nearby, and which similar incidents were resolved before. The third is that it is honest about its confidence. A system that says “probable cause, moderate certainty” is more useful at that hour than one that sounds sure and is wrong.

Why does designing for “the team” produce vague tooling?

When a feature is specified for an average user, it gets averaged. The summary becomes long enough to satisfy the architect, short enough to satisfy nobody, and phrased in a register that suits no actual reader. Picture instead the one engineer above. She does not want a dashboard tour. She wants to know the blast radius, whether a customer is affected right now, and whether the fix is in a runbook she can open on a phone with one thumb.

Once you hold her in mind, decisions get easier. A correlation view that needs a laptop is out. A chart without a plain-sentence caption is out. A recommendation that cites twelve contributing factors gets trimmed to the top two, with the rest one tap away. Notice that nothing here required new technology; it required picking a person.

How do you choose the one person to design for?

Choose someone real, and choose the least comfortable plausible case rather than the most comfortable one. Good candidates are the newest member of the rotation, the engineer who covers a service she did not build, or the manager who is pulled into a call and must say something sensible to a customer within minutes.

Then write down four facts about that person, on one card:

  • The question she asks first when the alert lands.
  • The thing she is quietly afraid of getting wrong.
  • How much time and attention she realistically has.
  • What she will do next if the information is useful.

Pin the card beside the design document. When a debate starts about what to include in an incident summary, the card settles it faster than seniority does.

What should an AIOps summary say to her, in what order?

Lead with impact in plain words: who or what is affected and since when. Follow with the most probable cause and the evidence behind it, stated in a sentence or two. Then offer the first reversible action. Only after that should the summary list related alerts, historic matches and raw signals, folded away for the moments she wants to dig.

It also helps to write the summary in the voice of a colleague rather than a log line. “Checkout latency rose right after the afternoon deploy; the same pattern appeared in an earlier incident that was fixed by rolling back” is something she can act on while half asleep. A list of anomaly scores is not. If you want a grounded example of reducing the flood before it reaches a person, the post on calming a noisy stream of IT events with smarter tooling covers the mechanics; here the point is only that the filtering should be tuned to her tolerance, not to a theoretical average.

Does she need to understand the model?

She needs to understand its limits, not its mathematics. Give her three things: what data the suggestion was based on, how fresh that data is, and what the system tends to get wrong. A short note such as “this service has little history, so treat the suggestion as a hunch” respects her judgement. It also teaches her when to override the tool, which is the skill that keeps automation trustworthy over years.

Encourage feedback that costs her one tap. A simple “useful” or “off target” on each summary gives the people tuning the system a signal that originates from the person the system is meant to serve, and it tells her that her opinion is wired into the product rather than shouted into a void.

What does a good rollout look like for one reader?

Start with a single service and a single rotation. Ask the engineer in question to use the summaries for two weeks and to tell you where they wasted her time. Fix those, then widen. This is gentler than a big-bang launch, and it keeps the feedback loop short enough that the changes are visible to the person who asked for them. For a broader view on how teams introduce the practice, a practical walkthrough for adopting AIOps in an IT group lays out the organisational side, and it pairs well with the reader-first lens used here.

Measure in her terms as well. Instead of counting how many alerts were processed, ask whether she resolved the incident without paging a second person, and whether she felt equipped. Those answers are softer than a throughput figure, yet they predict whether she will still trust the system after six months.

Where can she read other people’s accounts of this?

Operational lessons travel best as stories, and engineers who have survived a rough rotation often write theirs down. One home for that kind of writing is iBlogUp, a blogging site that hands every writer a personal subdomain for their stories; follow a writer through the site feed and new entries simply turn up in your reader. Subscribing means our engineer can stash a handful of honest first-person accounts in her feed reader and read them on a quiet afternoon, not while an incident is burning.

What if the one person you picked is wrong?

Then you will find out quickly, and that is a feature. A design built for a specific person can be tested against that person. A design built for everyone can only be admired. If the engineer says the summary is still too long, you cut it. If the manager says the plain-language line is missing, you add it. Over time you may hold two or three such personas, but each stays sharp because each is one human being, not a category.

Keep a gentle check on bias, too. If every persona you pick resembles the people already in the room, widen the choice: someone in another timezone, someone who joined from a non-technical background, someone who speaks English as a second language. Their needs often expose assumptions that the rest of the team stopped noticing.

A short recap for a tired mind

  • Pick one real person and write four facts about her on a card.
  • Lead every AIOps summary with impact, likely cause, and a reversible first step.
  • State the limits of the suggestion in plain words.
  • Roll out to one service and one rotation, then widen.
  • Judge success by whether she felt equipped, not only by alert counts.

A system that serves one tired engineer well tends to serve the rest of the team acceptably; the reverse is rarely true. For a gentler starting point if the vocabulary still feels heavy, an entry-level primer on diagnosing live IT trouble is a sensible companion read.

Related Posts

Ship a Small AIOps First Version Before the Grand Design

A step-by-step way to build one narrow AIOps capability, show it to the engineers who carry the pager, and let their reactions decide what comes next.

Read More

An AIOps Maintenance Audit: Check What You Already Run

A storytelling checklist for auditing the models, rules and links already running in your AIOps estate before you ship anything new.

Read More

Write the Runbook: An AIOps Explainer in Plain Words

What an AIOps runbook is, why one expert’s memory is a risky place to keep procedures, and how to write short, findable routines anyone on the team can follow.

Read More

Why Vague IT Requests Stall and How a Short Brief Fixes Them

Most stalled AIOps efforts began as one fuzzy sentence. Treat the request like an incident: spot the symptoms, find the cause and fix it with a one-page brief.

Read More

Reconcile Regularly: How One Ops Team Matched Tickets to Reality

A platform team found quiet gaps between approved changes, deployments and automated actions. A weekly matching habit closed them while they were still small.

Read More

AIOps Cost for Beginners: A Unit-Cost Primer With Glossary

A big AIOps total tells you little. This beginner primer, with a mini-glossary, shows how dividing spend by one familiar unit reveals what really drives it.

Read More
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x