Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!
We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!
Learn from Guru Rajesh Kumar and double your salary in just one year.
An operations team switched on automated remediation for a flaky queue service on a Thursday afternoon. The model had been trained on months of incident history, the demo looked wonderful, and nobody wrote down what would happen if the automation started doing the wrong thing. By evening it was restarting healthy workers in a loop, and the room spent two hours arguing about how to turn it off. This post answers the questions that story usually raises, with one idea running underneath: in AIOps, a pre-decided plan B is what separates a calm switch from a scramble.
Why does an AIOps rollout need a fallback at all?
Because every automated decision rests on something you do not fully control. It might be the quality of the training data, the freshness of a feature feed, the accuracy of a topology map, or the assumption that tomorrow’s traffic resembles last quarter’s. When any of those slips, the system does not announce itself. It keeps producing confident output, and the output is simply wrong.
A fallback is the path you walk when that happens. It is not a prediction of failure and it is not pessimism. It is one short statement, settled early, that begins with “if this, then we go back to that.” Teams that have such a sentence tend to use it quickly and without drama. Teams that lack one tend to debate, and debate is expensive at three in the morning.
What does a good plan B actually contain?
Three things, and they fit on an index card.
- The alternative: the specific manual or older automated route you will use instead, named plainly, such as “page the on-call engineer with the raw alert” or “fall back to the static threshold rules.”
- The trigger: an observable signal that tells you to switch, such as a spike in auto-actions that had to be reversed, or the model’s input feed going stale beyond an agreed window.
- The owner: the role, not a person, who has the authority to flip the switch without asking for permission.
Leave out any one of these and the card stops working. An alternative without a trigger sits unused while things degrade. A trigger without an owner produces a long chat thread. An owner without an alternative can only say “stop,” which is not a plan.
Where should the fallback live?
Close to the thing it protects. If the automation lives in a pipeline, the fallback belongs in the same repository, with the same review process. If it lives in a vendor console, put a short note in the on-call handbook that says where the off switch is and what it changes. The worst place for a plan B is a slide from the project kickoff that nobody opens again.
One practical habit helps a great deal: give every automated action a “manual equivalent” label in your runbook. When the auto-remediation closes a ticket or scales a service, the label tells a tired human exactly how to do the same job by hand. If you cannot write the label, you do not yet understand the automation well enough to trust it.
How is a fallback different from simply having good monitoring?
Monitoring tells you something is off. The fallback tells you what to do about it. Many teams have excellent dashboards and still lose an hour at the decision point, because the dashboard cannot say “now revert.” Pairing the two is the whole trick: each alert that could imply a bad model output should link to the page that describes the switch.
If you are still tuning how your alerts are grouped, an earlier piece on this site about tidying up event noise with smarter tooling deals with how alerts get bundled before anyone sees them. A fallback sits one step after that: once the right alert reaches the right person, they need somewhere to go.
What does the switch look like in practice?
The cleanest version is a feature flag or a configuration toggle that moves a service from “model decides” to “rules decide” without a deployment. The less clean version is a documented sequence of commands. Either is fine, provided that someone has confirmed it works on a quiet day. A switch that has never been flipped is a theory.
For the story above, the team eventually added a line to their change template: “Name the rule set we return to, and the signal that sends us there.” The next time the automation misbehaved, an engineer flipped the toggle within minutes, noted the time, and went back to the investigation. Nothing heroic happened, which was the point.
Does every automation deserve this treatment?
No, and pretending otherwise burns effort. Ask one question: if this component produced bad output for an hour, who would notice, and what would it cost? An automation that suggests a dashboard colour needs no card. One that restarts services, reroutes traffic or closes incidents on its own absolutely does. Match the care to the blast radius.
That reasoning also helps with prioritisation. Teams new to AIOps often want to wrap everything in fallbacks on day one, then abandon the whole habit because the paperwork feels heavy. Start with the two or three actions that can hurt the most and expand slowly.
What if the fallback itself fails?
It can. Manual processes have their own weak points, such as an on-call person who is unreachable or a runbook that references a retired system. The honest answer is to keep the fallback boring and old. The more modern and clever the backup is, the more likely it shares a dependency with the thing it replaces. A plain threshold rule and a phone call beat a second machine-learning model every time.
It also helps to keep the plan B independent of the same data feed. If the model fails because its inputs are stale, a fallback that reads the same inputs will fail in the same way. Choose a route that draws on something different, even if it is cruder.
How do we know when to retire the fallback?
You do not retire it so much as promote it. After the automation has run through several real events without needing a switch, the trigger thresholds may be loosened, and the card can be reviewed less often. But keep the card. Conditions change, models are retrained, vendors alter behaviour, and the day you most need the old path is the day you forgot it existed.
A related thought comes from the earlier article on how AI supports IT decision-making: decision support works best when people stay able to override it. A plan B is that ability written down.
Is there a human-sized version of this outside IT?
Everyone already does it. Anyone who has organised a community evening knows to name an indoor option in case the outdoor one is rained off. Events-listing websites such as MyEventNow, a listings site that pairs event pages and venue pages with a blog and a community forum, make the same idea easy to see in a daily setting: browse a few alternatives before you commit, so a cancelled plan turns into a switch rather than a lost evening. Operations teams are simply doing the same thing with services instead of seats.
What should we do this week?
Pick one automated action in your environment and run through a short checklist.
- Write the manual equivalent in two or three lines.
- Choose one observable trigger and a threshold you can measure today.
- Name the role that may flip the switch without asking.
- Confirm on a calm day that the switch works, and record how long it took.
- Put the card where the on-call person will actually find it.
If that exercise takes more than an hour, the action is probably too tangled to automate without a better exit. That finding is valuable in itself.
What about predictive features specifically?
Predictions are the trickiest, because a wrong prediction looks the same as a right one until the moment it matters. If you rely on forecasts to schedule maintenance or capacity, settle beforehand how you will respond when the forecast turns out to be off, for instance “fall back to the calendar-based schedule for the next cycle.” The earlier note on stopping predictive maintenance from going wrong in IT systems is a useful companion here.
The Thursday team did eventually bring the automation back, with a smaller scope, a named switch and a trigger written on a card taped near the on-call desk. It has been quiet since, not because the model got perfect, but because nobody is afraid of it any more. That calm is what a plan B buys you.