Start, Stop, Continue: An AIOps Post-Release Retrospective FAQ

Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!

We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!

Learn from Guru Rajesh Kumar and double your salary in just one year.


Get Started Now!

Every team that runs machine-assisted operations eventually hits the same moment: a rollout of a new anomaly model or a revised alert policy has finished, the dust has settled, and nobody is quite sure what to take from it. A start, stop and continue retrospective is one of the simplest ways to turn that fog into decisions. This FAQ-style guide explains how an AIOps team can run one after a release, what belongs in each list, and how to keep the result from becoming another forgotten document.

What is a start, stop and continue retrospective for an AIOps team?

It is a short meeting, usually under an hour, held after a defined stretch of work such as a model update, a migration of telemetry sources or a quarter of on-call duty. The group writes three plain lists. Start holds practices the team does not do yet but believes would help. Stop holds habits that cost time or attention without paying anything back. Continue holds the things that worked and deserve protection from the next round of tinkering.

The format works well for AIOps because the field produces plenty of vague impressions: the model felt noisy, the runbooks felt stale, the handoffs felt slow. Three lists force each impression to become a specific action with an owner, which is the only form in which it can change next month’s results.

When should the team hold it?

Hold it soon enough that people remember the details, and late enough that the outcome is visible. For a release of a new correlation rule set, that usually means after a full on-call rotation has lived with it. Holding it the day after deployment gathers opinions about the deployment itself; holding it after a rotation gathers evidence about the behavior. Put the date on the calendar when the release is planned, so the retrospective is part of the work rather than an afterthought that gets dropped when things get busy.

Who should be in the room?

Invite everyone who touched the work: the engineers who tuned the models, the people who answered pages, whoever maintains the data pipelines feeding the platform, and a representative from any team that consumed the alerts. Cap attendance so that each person has real airtime. A facilitator who did not own the release helps, because that person can ask the awkward question without defending a decision.

What goes under Start?

Start items are additions, and they should be small enough to begin within a week. Good examples from AIOps work include the following.

  • Labeling closed incidents with whether the platform surfaced the problem early, late or not at all, so the next model review has real outcomes to compare against.
  • Reviewing a sample of suppressed alerts each week to confirm that nothing important was quietly filtered out.
  • Writing a one-paragraph change note whenever a threshold or feature is altered, so on-call staff can connect a change in behavior to its cause.
  • Inviting a service owner to the next tuning session, so context about expected traffic patterns reaches the people adjusting the detectors.

Notice that none of these says “improve the model”. A start item names a behavior, not an aspiration.

What goes under Stop?

Stop items are the hardest to write because they admit that effort was spent badly. They are also the most valuable, since a team that only adds practices eventually drowns in them. Typical entries include pausing the daily manual export of dashboards that nobody opened, ending the habit of retraining a model on a fixed schedule when its inputs have not shifted, and dropping approval steps that duplicate an automated check already in the pipeline.

One quick check: imagine the practice disappearing overnight and picture who would complain. If the honest answer is that nobody would notice, it belongs on the stop list. If someone would notice but the benefit is small, consider shrinking it instead of abolishing it, and record that decision explicitly.

What goes under Continue?

Continue is not a courtesy list. It protects practices that are easy to lose when the team gets excited about changes: the blameless review of every major page, the short pairing sessions between a data engineer and an operator, the habit of keeping a rollback path ready for every model change. Writing them down gives the team permission to say no when a new proposal would crowd them out.

If the team notes a practice it keeps skipping, that usually belongs under Start with a smaller scope, not under Continue as a wish.

How do we keep the lists honest?

Three habits help. First, require each item to cite an observation: a ticket, a graph, a moment from a handover. “Alerts were annoying” becomes “the same storage warning paged us repeatedly on a quiet weekend”. Second, limit the lists, for example to three items each, so the group has to choose. Third, assign one owner per item, and write the owner next to it before the meeting ends.

It also helps to separate people from process. If a Stop item sounds like criticism of an individual, rephrase it as a description of the process that allowed the problem. The goal is for the next engineer who joins to read the notes and find them useful instead of defensive.

What does a short example look like?

Imagine a team that rolled out a new forecasting component for capacity warnings. After one rotation, its lists might read as follows. Start: tag each capacity page with whether the forecast was close or far off. Stop: sending the weekly summary to a distribution list that had grown far beyond the people who acted on it. Continue: the brief Friday check where the on-call engineer walks the next person through anything unresolved. Three lines, three owners, and a clear review date. That is the whole artifact.

How does this relate to other places people look for ideas?

Teams often borrow retrospective habits from outside their field, and that can be a healthy source of fresh framing. Aviation is a popular example because its culture of debriefing is so well known. One aviation discovery website, BestPilotsNow, is organized into sections for pilots, flight schools, courses and resources, with a forum and a blog alongside. It is a reminder that debriefing is a shared habit across very different kinds of work, and a quick read of such material can spark ideas for your own meeting format.

How do we follow up so the lists matter?

At the start of the following planning session, read the previous three lists out and mark every entry as finished, abandoned or pushed forward. Carried-over items get one more cycle, then must be done or dropped. If the same Start item survives three rounds, the team has learned that it is not actually a priority, and removing it is an honest outcome.

Keep the lists somewhere visible to anyone who joins the on-call rotation. Newer engineers often learn more about how the team really works from a few retrospective notes than from formal documentation, because the notes show what the team tried and discarded. If you want inspiration for how others describe the data side of this craft, our earlier piece on how AI-supported analysis feeds IT decisions covers the decision context, and the discussion of using smarter tooling to quiet event floods is a good companion for the Stop list.

What are the common mistakes?

  • Letting the Continue list stay empty, which leaves the team treating everything as up for change.
  • Turning Start into a wish list of large projects instead of small experiments.
  • Skipping the review date, so items never come back for a decision.
  • Recording so many items that none gets an owner.
  • Holding the meeting only after failures, which teaches people to associate it with blame.

Final thoughts

A start, stop and continue retrospective costs an hour and a handful of sticky notes, yet it gives an AIOps team something that dashboards alone rarely supply: a shared, written decision about how to work differently next time. Keep it short, tie every line to an observation, give each item an owner, and revisit the lists at the next planning session. Over a few cycles the habit compounds, and the team spends less time arguing about impressions and more time acting on what it actually saw.

Related Posts

AIOps Hardest Task First: A Team’s Morning Case Study

A fictional AIOps team stops opening the day with small tickets and starts with the correlation rework it dreads. A case study in doing the hardest task first.

Read More

The Waiting-For List That Keeps AIOps Projects From Stalling

Most AIOps projects stall in the gaps between teams. A plain waiting-for list, with an owner and a nudge date per line, closes those gaps.

Read More

Task-List Done vs Outcome Done: An AIOps Finish Line

Two ways to close an AIOps ticket, compared side by side: ticking off steps versus proving a condition is true. Pick the finish line before the work starts.

Read More

Eight Ways to Calibrate Your AIOps Task Estimates

Hours guessed versus hours spent, tracked as a simple ratio, can turn vague AIOps task estimates into numbers you can plan around. Here are eight practical moves.

Read More

Ask Why Five Times: A Step-by-Step Guide to AIOps Root Causes

A practical walkthrough for turning a repeat alert into a process fix by asking why until the answer names something your team can change on Monday.

Read More

Rubber-Duck Debugging for AIOps: A Beginner’s Primer and Glossary

Explaining a failing AIOps service aloud, line by line, exposes the assumption you stopped questioning. A worked example and a seven-term glossary for beginners.

Read More
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x