Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!
We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!
Learn from Guru Rajesh Kumar and double your salary in just one year.
Every operations team has a moment that goes the same way. An alert fires, three people look at it, and for several minutes nobody is sure who should do what. The technology is fine; the missing piece is a decision made earlier. That is what if-then rules are for, and in this walkthrough we will build a small set for an AIOps setup, step by step, in a way you can finish over a couple of coffee breaks.
An if-then rule is just a trigger paired with a response: if the deploy fails, then roll back first. If an alert fires twice, then page the owner. The magic is not in the wording but in the timing. You write the rule while you are calm, and the rule does the thinking when you are not.
Step 1: Collect the moments that keep repeating
Open your ticket history or chat scrollback and look for situations that come back. Do not start with theory. Start with the last month of real incidents and jot down the ones that made people hesitate. You are hunting for patterns such as a model score that drifts, a collector that stops sending data, a correlation that groups unrelated events, or a remediation script that half-completes.
Aim for five to eight situations. More than that and the list becomes a manual nobody reads. If you want background on how machine learning changes the alert stream you are about to write rules for, the piece on how learning models reshape alert handling in IT operations is a handy companion.
Step 2: Turn each situation into a trigger you can observe
A trigger must be something a person or a system can see without interpretation. “The system feels slow” is not a trigger. “Median response time stays above the agreed limit for ten minutes” is. In AIOps work this is especially important, because the whole point of the platform is to surface signals, and your rule should lean on a signal the platform already produces.
Try writing each trigger as a single clause that starts with “if”. Then read it aloud to a colleague. If they ask “what do you mean by that?”, tighten it. Good triggers in this field tend to be countable: a number of repeats, a duration, a status change, a missing heartbeat.
Step 3: Attach one small response
Now add the “then”. The response should be one action, not a project. Rolling back, switching a feature flag off, restarting a single collector, pausing an automated remediation, or sending a prepared message to the owner are all good examples. If the response needs a meeting, break it down until the first move is something one person can do in a few minutes.
A few sample pairs to borrow from:
- If an automated remediation runs twice without clearing the alert, then pause it and hand the case to a human.
- If a data feed goes quiet, then check the collector before touching any model.
- If two alerts share the same service and start within a minute of each other, then treat them as one incident and open one ticket.
- If a new model version raises the false-alarm rate, then switch back to the previous version while you investigate.
Step 4: Name the owner and the fallback
A rule without a name beside it is a wish. For each pair, write who acts first, and who acts if the first person does not respond within a set time. Use roles instead of individuals, such as “the on-call engineer” and “the platform lead”, so the rule survives holidays and reshuffles. Teams that are bringing AIOps into daily work for the first time will find the framing in a simple guide to getting AIOps adopted by your IT team useful when deciding who owns what.
Step 5: Write it down where people will actually look
The best rule in the world fails if it lives in someone’s head. Put your list at the top of the on-call channel, in the alert annotation, or inside the dashboard panel that fires the notification. Keep each rule to a single line. Bold the trigger so a tired reader can scan to it. A short, visible list beats a long, perfect document every time.
Step 6: Test one rule on purpose
Pick the rule you are least sure about and trigger it in a safe environment. Does the owner get the message? Does the rollback complete? Is the wording clear at three in the morning? Testing one rule per week is far more realistic than testing all at once, and each test usually exposes a missing detail: a wrong channel, an expired credential, a step that assumed a permission nobody had.
Step 7: Trim and tune after each real incident
When a rule fires in anger, spend two minutes afterward asking whether it helped. Did the trigger arrive too late? Was the response too big? Did a case turn up that no rule covered? Adjust the line, or add a new one. Rules are living things. Many teams find the list stabilises after a couple of months, and from that point the changes are small and calm.
If your rules touch data quality, you may also enjoy reading about how pipeline automation cuts down daily data errors, since many AIOps alerts start life as a problem with inputs rather than with models.
A word on rules that grow too clever
It is tempting to build branches: if this and that but not the other, then do one of three things. Resist. The more conditions you add, the more a rule resembles the in-the-moment judgement you were trying to avoid. If a rule needs a flowchart, split it into two rules or leave the edge case to human judgement. Simplicity is a feature here.
Borrowing the habit from other places
The idea is not unique to operations. Travellers use it all the time: if the flight is delayed then take the later train; if the phone dies then use the saved offline copy of the booking. Before a trip, people often browse a web resource that sets physical SIM and eSIM choices for travelers side by side and organises them by country, provider, plan and guide, so the phone question is closed before the airport. Your alert queue deserves the same early decision.
Your first if-then rules, in one sitting
Here is a gentle exercise to finish. Set a timer for twenty minutes. List the three alerts that most often caused hesitation last month. For each one, write a trigger you can observe, one small response, and a named owner. Paste the three lines into your on-call channel and ask the team to challenge them. You will probably change at least one. That is the system working.
Within a few weeks you should notice that handovers get shorter, newer teammates ask fewer “what do I do now?” questions, and the first five minutes of an incident feel less like improvisation. None of this requires new tooling. It only asks you to settle the response today, while nothing is on fire.