AIOps With a Tiny Toolbox: Three Triage Builds Compared

Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!

We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!

Learn from Guru Rajesh Kumar and double your salary in just one year.


Get Started Now!

An empty repository and an unlimited budget make for a strange kind of paralysis. Every AIOps team knows the feeling: the brief says “help on-call engineers decide what to look at first,” and suddenly there are forty frameworks, six streaming layers and a vendor deck for each. This comparison takes the opposite route. We set one rule up front, a single dependency and a small codebase, and then compare three ways of building incident triage inside that fence. The point is less the winner than what each limit forces you to notice.

The story below is a composite from the kind of work operations groups do all the time. A platform team inherits a pile of incident tickets and a pager rota that feels arbitrary. Nobody has the appetite for a six-month modelling project, so the team lead proposes a game: build the first useful triage helper using one library, no new infrastructure, and a code budget that fits on a couple of screens. Three small squads each pick a different approach.

Why a fence helps AIOps work in the first place

Open-ended AIOps projects tend to start with tooling and drift toward demos. When the toolbox is unlimited, the first week goes to choosing, the second to integrating, and the third to explaining why the pilot has not yet touched a real incident. A hard limit changes the order of events. With only one dependency allowed, the squad has to read actual tickets before it can justify anything, because there is no framework to hide behind.

There is a quieter benefit too. Small constraints produce small systems, and small systems are explainable. When an engineer at three in the morning asks why an incident landed at the top of the queue, an answer such as “it matched the checkout keyword and the error pattern appeared twice this hour” is far more reassuring than a score with no story behind it. The people reading a short rule can argue with it, fix it and trust it a little more each week.

The ground rules the three squads shared

Before anyone typed a line, the squads agreed on the same fence so the comparison stayed fair. Each build could depend on exactly one library of its own choosing, had to run on a laptop-sized machine, and had to fit into a codebase small enough for a reviewer to read in a single sitting. Every build also had to emit a short, human-readable reason alongside its ranking. If a squad could not explain a decision in one sentence, the decision did not count.

The test material was a folder of past incidents with the final severity that humans had assigned after the fact. A slice was held back so the final judgment could not be gamed, which keeps a tiny project honest.

Build one: the hand-written rule table

The first squad used no library beyond the standard toolkit of its language. It wrote a table of rules: service name matches, error phrases, time-of-day weighting and a short list of known-noisy components. Each rule carried a weight and a one-line reason. The ranking was nothing more than a sum.

What the constraint taught this squad was how much knowledge already lived in people’s heads. Writing rules forced a conversation with the senior on-call engineers, and the best rules came straight out of their war stories. The weakness showed up quickly, though. Rules aged badly after a release changed error wording, and the table grew a little every week. Still, for the first two weeks this approach beat both rivals on one measure that matters in operations: everyone understood it on day one.

Build two: the statistical baseline

The second squad picked a single numerical library and built a baseline from plain frequencies. How often does this error signature appear in an ordinary hour? How far above its own history is it right now? Anything far outside its own normal climbed the list. No training run was needed, only counting and a rolling window.

The fence pushed this squad into a useful discipline: define “normal” per service before doing anything clever. It discovered that two services behaved so differently by day of week that a single global baseline was misleading, so it split the history by weekday and hour. That fix took a dozen lines. The approach struggled with brand-new failure modes, since a signature with no history has no baseline, so the first sighting of a novel problem was always underweighted.

Build three: the tiny learned classifier

The third squad leaned on one machine-learning library and trained a small text classifier on ticket titles and the first lines of logs. Under the limits, no feature store, no pipeline orchestrator, no GPU. Just a script that read a folder, fit a model and wrote a ranked list.

This squad found out how much of its early accuracy came from the labels rather than the algorithm. Severity labels assigned by tired humans were inconsistent, so the first model learned the inconsistency. Cleaning a few hundred labels improved results more than any tuning. The squad also noticed that the small model was surprisingly good at spotting similar past incidents, which suggested a more modest feature: “this looks like the outage from last quarter,” attached to the ticket. That reframing, from predict severity to retrieve precedent, only emerged because the box was too small to attempt the grander plan.

Side-by-side: what the comparison actually showed

When the held-back slice was scored, none of the three squads ran away with it. The rule table was the easiest to explain and the fastest to start, the statistical baseline was the most robust to wording changes, and the learned classifier did best at finding precedent when labels were decent. Rather than crown one, the team lead wrote down where each approach was weak, which turned out to be more valuable than a trophy.

  • Time to first usable result: the rule table needed days, the baseline a little longer, the classifier longest because of label cleanup.
  • Explainability at a glance: the rule table was effectively self-documenting, the baseline needed a chart, the classifier needed a short note on its training data.
  • Behaviour on a novel failure: the rule table missed it, the baseline discounted it, the classifier guessed from vocabulary.
  • Upkeep burden: the rule table asked for constant edits, the baseline asked for occasional window tuning, the classifier asked for periodic relabelling.

Because everything lived inside the same fence, the squads could even combine ideas without a rebuild: the classifier’s precedent lookup fed a “similar past incident” line into the rule table’s reason text. The merged helper stayed under the original size limit. If you want the longer view on how a platform can support decisions of this kind, the write-up on AI-assisted choices in IT operations covers the broader ground, and it pairs well with this small-scale experiment.

A thought on creative limits from outside engineering

The game the team lead proposed is not unique to operations. Hobbyist communities have used the same trick for years, and one example is a drone-themed website called Dronesnow, which describes itself as a hub with areas for pilots, aircraft models and flight schools, plus video, articles, discussion boards and a store. Hobby communities of every kind use the same trick: a self-imposed rule, such as one theme or one build style, tends to make the result more inventive.

How to run your own fenced experiment

You do not need a quarter of planning to repeat this. Pick a Friday afternoon and hold three things steady: one dependency, one page of code per approach, and one shared set of past incidents with a hidden slice for scoring.

Pay attention to which constraint creates friction, because friction marks the interesting problem. If the one-dependency limit hurts, ask whether the dependency was truly needed or whether the team simply reached for it by habit. If the page limit hurts, ask which logic really belongs in a tool and which belongs in a runbook. Constraints are cheap probes. They reveal assumptions that an open budget would let you paper over.

Keeping the fence honest over time

A limit is only creative while it is real. After the first experiment, the squads were tempted to raise each cap “just a little,” and the lead resisted until the helper had earned its way into real on-call use. Only then did the group relax the limits, one at a time, writing down what each relaxation bought. Looking back at that log, two of the three extra dependencies that were eventually added turned out to be unnecessary, while one saved a great deal of time.

The takeaway for anyone building AIOps tooling is that restraint is a design method, not a lack of ambition. A fence gets you to something that works, reads clearly and invites criticism. After that, you can spend the larger budget on the parts that deserve it. In this case the squads shipped something useful before the third week ended, and nobody had to defend a roadmap to get there.

For another angle on small, practical wins in similar systems, the piece on using AI to anticipate maintenance problems in IT estates shows how a narrow problem statement keeps projects grounded, which suits a next fenced experiment that targets capacity or hardware health rather than incidents.

Related Posts

A Teardown Checklist for a Finished Drift-Monitoring Project

Open a small, finished model-monitoring repository and audit how it was built, from baseline and thresholds to tests and alert wording, then borrow its structure.

Read More

Eight Runbook Phrases Worth Rewriting in Plain Words

Eight bits of insider shorthand common in incident runbooks, and the everyday wording that lets a tired on-call reader act on each one without asking anyone.

Read More

Why AIOps Teams Are Slowing Rollouts Down on Purpose

A friendly look at a quiet trend in AIOps: teams letting each automated change prove itself slowly, because the fast version keeps ending in a rollback.

Read More

How to Keep a Done List for AIOps Changes, Step by Step

A six-step method for recording AIOps changes that have shipped and been verified, so your team stops repeating work and can judge progress from evidence.

Read More

AIOps Fallback Plans: Questions to Answer Before Go-Live

How to decide on a fallback for automated remediation before launch, with the trigger, the owner and the switch written down.

Read More

Why Reviewing Near Misses Builds a Stronger AIOps Career

Close calls in AIOps work are free training. Here is a light routine for turning them into fixes, interview stories and a record of sound judgment.

Read More
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x