Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!
We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!
Learn from Guru Rajesh Kumar and double your salary in just one year.
Something interesting is happening in AIOps shops, and it runs against the usual story. For years the pitch was speed: faster detection, faster correlation, faster remediation. Lately, though, the teams with the fewest bad weekends are those who purposely ease off the moment automation is about to touch production. This post is a quick tour of that trend in AIOps, why it keeps showing up, and how it plays out in practice.
Nobody is arguing for sluggishness. The idea is narrower: at a handful of points, an unhurried pace is the fast option, because the rushed version produces a rollback, an apology and a second attempt that costs more than the careful one ever would have.
The pattern behind the trend in AIOps
Walk through a typical story. A team trains a model to suggest fixes for a recurring class of incidents. The demo is great. Someone says, why not let it act on its own, starting Monday? It goes live everywhere at once, handles the easy cases nicely, then does something odd with an edge case nobody had written down. By Tuesday the automation is switched off, a review meeting is booked, and the project loses a month of goodwill.
The version of that story with a happy ending looks different. The same model is allowed to suggest only, and humans compare its suggestions with what they would have done. Then it acts on one low-stakes service. Then two. Each stage lasts as long as it needs to, and nobody feels embarrassed about that. The total calendar time to full rollout ends up shorter, because there is no month of recovery in the middle.
That is the whole trend in one picture: the slow step is placed exactly where mistakes would be expensive, and everything else stays fast.
Where the slow steps actually go
Teams that do this well are surprisingly specific about where they hit the brakes. A few places come up again and again.
- Reading the whole change before approving it. Skimming a summary is quick; reading the full diff of a runbook or automation policy catches the one line that says “restart everything”.
- The first live action of any new automation. A human watches it happen, start to finish, before it is allowed to repeat.
- Widening the blast radius. Going from one service to many is the moment small flaws turn into big ones, so it gets the longest pause.
- Turning suggestions into actions. Advice and action are different permissions, and promoting one to the other is treated as its own decision.
Why rushing keeps backfiring
There is a simple arithmetic to it. A rushed release saves, say, a few days. A failed one costs an incident, a rollback, a postmortem, a round of rebuilding trust, and the original work again. Even if only one rushed release in four goes wrong, the expected cost is higher than the slow path every single time. People rarely do that sum in the moment, because the speed is visible immediately and the rework is visible later.
There is also a human side. Engineers who are told to hurry tend to skip the questions that make them look slow: what happens if this misfires at three in the morning, who gets paged, how do we turn it off? Those questions take ten minutes when asked calmly and ten hours when asked during an outage.
For related reading, the write-up on how teams lean on AI to support IT decisions looks at the decision-making angle, while the article on predictive maintenance claims in IT systems shows why confident predictions deserve a patient look before anyone acts on them.
What it looks like in a normal week
The slow-down habit is less dramatic than it sounds. In practice it shows up as small rituals.
A team keeps a short written list of what an automation is allowed to do and updates it before the automation changes, not after. A reviewer reads the list out loud once, which sounds silly and catches a surprising number of gaps. The first run of any new remediation happens while two people are watching the screen, with the manual fix ready in another window. And the promotion from “suggest” to “act” is scheduled with a gap long enough for odd cases to appear, rather than the next sprint by default.
A small detour: how people borrow the idea elsewhere
The habit travels well outside operations. People who plan a trip to a new city often find that an unhurried first afternoon beats a packed schedule, since it avoids the rebooking and backtracking that hurry creates. For anyone sketching a visit, an events-and-city website about Hyderabad with event listings, a blog and a forum lets people browse around before they commit to a plan. The comparison is loose, but the instinct is the same: look first, then move.
Signs your own pace is too fast
A team rushing the wrong steps usually gives itself away without any scorecard. Watch for these symptoms.
- Rollbacks are more common than clean releases, and nobody is surprised by them.
- Postmortems keep finding that the information was available, just unread.
- New automation is described as “basically done” for weeks while small fixes keep arriving.
- People say “we will tidy that up after launch” about safety checks.
Should this describe your team, the cheapest experiment is to pick a single upcoming change and give its riskiest step double the usual attention. Do not announce a new process. Just do that one step slowly, note whether anything surprising turned up, and compare the result with how the last rushed change went.
Where this is heading
This habit will probably spread as automation gets more capable. The more a system can do on its own, the more it matters that someone chose, deliberately and without hurry, what it may do. Teams that treat that choice as a real piece of engineering work, rather than paperwork standing between them and the launch, are the ones spending less time on rollbacks and more on building.
The takeaway is friendly rather than strict. Move quickly on everything that is easy to undo. When the next step is the hard-to-undo one, read the whole thing, watch the first run, widen slowly, and let the calendar catch up. In AIOps, as in most work, the shortest route to done is often the one where you stop to look properly the first time.