Four thousand unread alerts greeted the analyst on Monday morning. By Wednesday the queue had grown, two genuine intrusions were still buried somewhere inside it, and the team had started calling triage “the swamp.” If any of that sounds familiar, you already understand the problem that AIOps for Security claims to treat. The label itself is an emerging editorial term — no standards body, no founding milestone — for machine learning and large language models applied inside security operations: anomaly detection and behaviour analytics, alert scoring, LLM-assisted investigation, and above all honest evaluation of whether the machinery earns its keep. In this post we will diagnose the failure properly, the way we would diagnose a noisy IT estate, and then walk through the fix in the right order.

The Problem: An Alert Queue Nobody Trusts

The visible symptom is volume. The actual disease is arithmetic. A security monitoring stack watches authentication logs, endpoint telemetry, mail flow, and cloud control planes — tens of millions of events a day, of which only a handful are genuinely hostile. Any detector sitting on top of that firehose fights base rates before it fights attackers. This is also where the field splits away from the AIOps most IT teams already know. Operations data is mostly friendly and stationary; in security the source is adversarial and adaptive, and the cost of one missed case is measured in breach-scale losses rather than a sluggish dashboard. Treating the two disciplines as interchangeable is the first diagnostic error, and it is expensive.

Why AIOps for Security Lives or Dies on Base Rates

Run the classic numbers and the swamp explains itself. Take a modern SIEM ingesting 50 million events a day that contains 50 genuinely malicious ones. A detector with a false-alarm rate of one in ten thousand still raises thousands of wrong alerts, so the share of alerts that are real — the precision — lands near 0.9%. An analyst reads roughly 112 alerts for every true attack caught. Push the false-alarm rate down to one in a million and precision climbs to about 47%, at last close to a coin flip. The detector did not change; the arithmetic did.

This is decades-old, well-documented arithmetic rather than vendor optimism. The intrusion-detection literature worked it out long ago: at a base rate of 0.002%, a false-alarm rate ten times below the base rate yields a healthy two-thirds precision, while loosening the alarm rate to 0.1% collapses precision to roughly 1.4%. Researchers have complained for years that machine learning stayed “rarely employed” in operational security precisely because of a very high cost of errors — and the complaint has aged well. Any AIOps for security programme that begins by buying a model instead of by counting its own true positives will replay that history at premium pricing.

Ranking Helps, But It Cannot Repeal the Math

Alert scoring — ordering the queue by likelihood — is genuinely useful and the least glamorous corner of the field. A decent ranker turns 112 reads per real case into 20. What it cannot do is make a weak detector trustworthy enough to act on its own. An anomaly score paging by itself is a standard anti-pattern for a reason: most anomalies are not attackers. They are configuration changes, new hires, firmware updates, or someone logging in from an airport on the way to holiday. “Anomalous equals malicious” is a myth worth killing on sight, because every false page it causes erodes the trust the entire programme depends on.

The Fix in Order: Trust Granted by Level, Not by Logo

The healthiest mental model is an autonomy ladder granted per action class, never per product:

• L0 — observe. The model writes notes. Nothing else changes.
• L1 — recommend. It deserves a glance only after at least 200 labelled alerts including 30 true positives.
• L2 — auto-close. Permitted only with a miss bound of 2% or better measured on at least 150 true positives, canary cases in place, a blind holdout, and one-click reopen for humans.
• L3 — reversible action. Requires a false-action bound of 1% or better on at least 300 would-have-acted verdicts, tested rollback, and a rate limit.
• L4 — irreversible. Never granted to AI. Not “not yet” — never.

One vendor detail makes the ladder concrete. Microsoft’s Phishing Triage Agent has closed mail automatically since its first minute in production — no observe-only mode appears anywhere in its documentation (checked October 2026), leaving a buyer who wanted an L0 phase to build one. Google’s Triage & Investigation Agent, by contrast, is classed L1 and spends about a minute per triage. Same category, very different trust posture.

Proof Comes From Bounds, Not Vibes

Suppose a triage agent closes mail and you blind-label 1,240 of its decisions, finding 71 true positives and two legitimate messages the agent wrongly closed. The raw miss rate is 2.8% — under the 2% gate, right? No. With small samples, zero observed errors bounds the true rate near three divided by n, and observed misses push the confidence bound far above the raw number: those two misses put the 95% upper bound near 8.6%, failing the gate decisively. Proving a 2% bound with a single observed miss takes 236 samples; with two misses, 313. The same evaluation might simultaneously show 99.8% precision on close decisions, 86% precision on its “true positive” verdicts, and 93.5% of analyst workload genuinely removable — all true, and none of it licensing auto-close. Each number answers a different question. Demand all of them.

Sampling method matters as much as sample size. A 5% random sample of roughly 1,845 quarterly closes that finds zero misses still bounds the miss rate near 38%. Random sampling of happy paths is weak assurance. Canaries — 50 seeded cases per quarter with a rule like “three or more closed means withdraw the privilege” — are the honest instrument.

Your Ninety-Day Prescription

The fix, in the order that actually works:

1. Export your analysts’ dispositions for the past two quarters. They already exist as clicks.
2. Pick the two alert categories with the worst toil, not the most glamour.
3. Run the base-rate lab: events in, true malicious count, false-alarm rate, resulting precision.
4. Shadow a ranker with blind labels for a month. Nobody sees its scores while it learns.
5. Set gate values per category, monthly. Pooling is how a 9.1% average phish rate conceals 42.3% of misses in cloud-IAM cases.
6. Add purple-team canaries, drift monitors, and a cost baseline before any autonomy is granted.

If you want the full map behind this prescription — definitions, the ladder, the evaluation maths — the Ops Atlas entry on AIOps for security remains the best starting point I have come across.

Fitting AIOps for Security Into an Existing IT Practice

Teams already running AI-assisted operations hold a head start, because the hygiene transfers: solid event noise reduction shrinks the queue a scorer has to walk, and the labelled-history discipline that keeps an ops ranker auditable keeps a security scorer auditable too. We covered the on-ramp in our guide to bringing AIOps into your IT team without breaking it, and the same habits carry into real-time problem solving at 3 a.m., where a wrong auto-close costs the most.

One last decision rule, because budgets are real. Score your readiness zero to three on questions like labelled-data depth, per-category metrics, rollback maturity, and adversarial testing — twenty-seven points maximum. Above fourteen, go shadow mode and start counting. Nine to fourteen, time-box a pilot with an exit date. Below nine, decline politely and spend the money on logging. The queue will not empty itself, but arithmetic — not enthusiasm — is what drains it.