Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!
We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!
Learn from Guru Rajesh Kumar and double your salary in just one year.
Every AIOps team eventually ships a change that is hard to take back: an auto-remediation rule that restarts services on its own, a new anomaly model that pages people, a routing change that silences a whole class of alerts. The question worth asking is not whether the change is clever. It is how the team makes sure nothing obvious was forgotten in the minutes before it goes live. In practice there are two approaches, and a pre-flight checklist is the one that tends to survive a busy afternoon.
Here the two approaches meet head to head: the launch that relies on experienced people remembering everything, and the launch that runs the same short list in the same order every time. Neither is free, and neither is perfect, so the details matter.
The memory-driven launch
The first approach is the familiar one. Senior engineers know the system, they have shipped dozens of similar changes, and they carry a mental list of things to check. On a calm day this works well. The person doing the release glances at the dashboards, recalls that the last rule misbehaved during a batch window, and decides the timing is fine.
The weakness is that memory is selective. It favours the failure you saw most recently and quietly drops the dull items. A rule that was tested against last month’s alert volume might meet a very different pattern today, and nobody thinks to ask because the last three releases went smoothly. Fatigue, interruptions and a hurried chat message from a stakeholder all erode the list that lives in someone’s head.
There is also a transfer problem. When the experienced engineer is away, the person covering inherits the responsibility without the mental list. The release still happens, because the calendar says so, and the missing checks are discovered afterwards.
The checklist-driven launch
The second approach writes the checks down once and runs them the same way each time. It is the idea aviation made famous: before an irreversible step, pause and read the short list aloud. Nothing about it requires brilliance. The value is that the list does not get tired, does not skip the boring item, and does not change depending on who is on shift.
For an AIOps release the list stays short on purpose. A useful version fits on one screen and covers only the launch, not the whole project. A reasonable set looks like this:
- The rule or model change has been reviewed by someone other than its author.
- A rollback path exists and has been located, not merely assumed.
- The alert channels are quiet enough that a new page would be noticed.
- An owner is on call for the next several hours and knows what changed.
- The scope of the change is as narrow as the first release allows.
Notice that none of these items tests the model itself. Model quality is evaluated long before launch day. The checklist is about the moment of release, where the cost of a missed detail is highest and the ability to undo it is lowest.
Side-by-side: where each approach holds up
Consider four situations that come up regularly and see how each approach copes.
**A quiet release on a calm week.** Both approaches succeed. The memory-driven launch is faster, and the checklist adds a couple of minutes. If every release looked like this, nobody would bother with lists.
**A release squeezed between incidents.** Here the approaches diverge. The person rushing between two fires will skip the item that feels least relevant, and in AIOps that is often the check that alert channels are calm. A fixed list makes the skip visible, because someone has to decide to leave an item unticked rather than simply forgetting it.
**A hand-off to a colleague.** The checklist carries the knowledge. A new on-call engineer can read the same five lines the veteran uses and understand what the team considers essential. The memory-driven approach depends on a conversation that may not happen.
**A change nobody has made before.** This is where checklists are weaker, and it is worth being honest about it. A list built for ordinary rule changes may not cover a genuinely new kind of automation. The fix is not to abandon the list but to add one line when a gap is discovered, and to keep the list short enough that adding a line does not bury the rest.
What a good AIOps pre-flight list looks like
Comparing the two approaches also shows what makes a list worth keeping. Good lists share a few traits.
First, they are written in plain verbs. “Confirm the rollback command runs against a staging copy” is better than “ensure rollback readiness”, because the first can be ticked and the second can be argued about.
Second, they are ordered by when the information is available. Reviewing the change comes before checking the on-call roster, which comes before choosing the release hour. Reading in the same order every time turns the list into a rhythm.
Third, they distinguish between a read-aloud step and a do step. Some items are confirmations, such as the owner is named and reachable. Others require an action, such as opening the rollback path and looking at it. Mixing the two without marking them leads to ticks that mean nothing.
Fourth, they get pruned. A list that grows with every postmortem eventually becomes a document nobody finishes. When an item has never caught anything and nobody can say what it protects against, remove it.
The cost side of the comparison
Skeptics are right that a checklist has a price. It takes minutes, and those minutes feel wasted on the nine out of ten releases where nothing was wrong. The comparison is only fair if the cost of the tenth release is counted too.
In AIOps the downside of a bad launch is unusual. A faulty remediation rule can act at machine speed, repeating the mistake across many hosts before a human notices. A noisy detection change can flood the on-call phone and train people to ignore pages. These are not failures you can quietly fix an hour later, which is exactly the category of step where a pre-flight list earns its place.
The memory-driven approach also has a hidden cost: it concentrates skill in a few people and makes releases dependent on their availability. A checklist spreads the baseline across the whole rota, so the quality of a launch stops depending on who happens to hold the pager.
Combining the two
The most workable arrangement is not a contest. Experienced judgement decides what to ship and when. The checklist decides whether the team is ready to press the button. Judgement is flexible and checklists are rigid, and each covers the other’s weakness.
Teams that do this well tend to keep the list close to the tools. It sits in the release ticket template or at the top of the runbook page, so the person launching sees it without hunting. They read it aloud in a short call or a chat thread, so a second person hears the answers. And they review it on a regular cadence, trimming items that never fire and adding the occasional one that a near miss exposed.
For readers who like to see how other technical communities think about careful, repeatable procedures, it can be interesting to look beyond operations. CryptoBlockCoins is a blog about cryptocurrency and blockchain, with coin, tutorial and daily-log sections, and its tutorial style shows how step-by-step discipline is explained to a very different audience.
A short worked example
Imagine a team about to enable an automatic restart for a service that tends to hang. The memory-driven version goes like this: the engineer who wrote the rule is confident, the dashboards look healthy, and the rule is enabled late in the day. The checklist version takes a few more minutes. A colleague reads the rule and asks what happens if the restart itself fails. The rollback toggle is found and tested. The team notices that a different alert channel is already noisy from an unrelated issue, so they postpone the launch until it settles. None of this required cleverness. It required a list that insisted on being read.
If your organisation already tracks related decisions, it may help to connect this to existing guidance on the team’s site, such as the write-up on how AI can support IT decisions or the piece covering reducing event noise with smarter tooling. Both are useful context for deciding which checks belong on your own list.
Verdict
If releases are rare, low-stakes and always handled by the same person, memory may be enough. For anything that acts automatically, pages humans, or cannot be reversed in a few seconds, the fixed list wins on consistency, on hand-offs and on the quality of the worst day, not the average one. Keep it short, keep it in the same order, and keep it about the launch itself. The goal is not to prove the team is careful. The goal is to make carelessness harder to fall into when everyone is busy.