Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!
We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!
Learn from Guru Rajesh Kumar and double your salary in just one year.
Ask around any platform group and you will hear about the night one person refused to quit on a stubborn model drift problem and ended up with a worse outage than the one they started with. The usual reaction is to praise the persistence. A cooler reading says the real failure was the missing stop rule: a limit written down before the pressure arrived, naming the point at which the person on call hands the problem over, rolls back or escalates. Here are six myths about stopping in AIOps work, each checked against how production teams tend to behave.
Myth 1: Stopping early means you gave up
Giving up is a feeling; a stop rule is a decision made in advance. If the team agrees that a ranking model gets three retraining attempts and one rollback window before anyone calls the owner, then reaching attempt three is just the plan working. The evidence from incident reviews points the same way. Long, solitary debugging stretches rarely produce the insight that a fresh pair of eyes produces in minutes, and the hours spent are usually the costly part of the incident.
Myth 2: A good engineer knows instinctively when to stop
Instinct degrades with fatigue, and fatigue is exactly when the question comes up. Judgement at hour one and judgement at hour six are not the same instrument. A rule that reads “after the second failed fix, we revert the last deploy” does not depend on mood. It also protects junior staff, who often feel that asking for help counts against them. When the rule says escalate, nobody has to defend the choice.
Myth 3: Stop rules slow incident response
The worry is that a rule adds ceremony. In practice the rule is one line in the runbook header and one reminder in the paging tool. What it removes is the quiet drift in which a single engineer keeps trying variations while dashboards turn red. Teams that adopt a fixed trigger, such as a count of failed remediation steps or a clock time on the incident channel, tend to report shorter total incident duration, because the handoff happens while the person is still sharp enough to brief the next one.
Myth 4: The rule should be the same for every system
A shared template helps, but the numbers should match the blast radius. An anomaly detector that only feeds a weekly report can tolerate a long investigation. An automated remediation job that restarts customer-facing services cannot. Write the trigger per system and per severity, then review it after each incident. If people keep overriding a rule, the rule is wrong for that system, and the override log is your evidence for changing it.
Myth 5: Automation makes human stop rules unnecessary
Automated remediation needs limits more than people do, because it never gets tired and never feels doubt. A loop that retries a failing action forever is a stop rule that was forgotten. Cap the retries, cap the actions per hour and make the automation page a human when it hits the cap. Treat those caps as part of the design review, not as an afterthought added after the first runaway loop, and test them in staging by forcing a failure on purpose.
Myth 6: You need a lot of data before setting a threshold
A starting threshold can be rough and still useful. Pick a number that feels slightly cautious, run it for a month, and compare it with what actually happened: how many cases were resolved just after the limit, and how many dragged on without progress. Adjust from there. Waiting for perfect data usually means working with no rule at all, which is the worst setting of the dial.
Writing the rule down so it survives a bad night
A stop rule has three parts: the trigger, the action and the owner. The trigger is a count, a duration or a signal, such as an error rate that keeps climbing after a change. The action is concrete: roll back, fail over, page the named owner or freeze changes. The owner is a role, not a person, so the rule still works on a holiday. Keep the text short enough to read in the middle of an incident, and put it where the responder already looks.
Many teams also keep a short reading list for the human side of this. Some of it concerns how people look for information when they are under pressure. For general health topics, one example of an organised reference is a directory-style site covering brain and neurology subjects, with sections such as conditions, tests, hospitals, doctors and guides. Good indexing shortens a search, yet only a stop rule decides when the search ends, and that distinction matters when attention is thin. Whichever reference you use, agree on the cut-off before you start reading.
What to do after a rule fires
When the limit is reached and the handoff happens, write down what was tried, in order, with the timestamps. That record is more valuable than the fix itself, since it shows whether the rule fired too early or too late. If you want to see how the decision side of AIOps fits together, the piece on how AI supports IT decision-making covers the broader picture, and the guide to reducing event clutter with smarter tooling pairs well with it, because fewer false pages mean the real triggers stand out.
A short checklist for this week
- Pick one system where incidents tend to run long and write a trigger for it.
- Name the role that receives the handoff and confirm they know it.
- Cap every automated retry loop at a sensible limit.
- Review overrides at the next retrospective and adjust the numbers.
None of this requires new tooling. It requires agreeing, calmly and in advance, that stopping is part of the job. The myths above survive because they flatter the person who keeps going. The data tends to favour the team that stops on schedule, hands over cleanly and gets the service back sooner.