Picture a platform group inside a logistics firm, an invented team whose troubles will ring true for anyone who tunes anomaly detection. For months their AIOps detection settings had been improving, and then one Tuesday they got worse, and nobody could say why. This is a story about keeping a version history of your detection configuration, and about how little it takes to do it well.

The Tuesday the models got worse

The group maintained a single configuration file that told their monitoring stack how sensitive to be: seasonal baselines, the window used to learn normal behavior, the grouping rules that decided which signals belonged to one incident. Over spring, an engineer named Dara had tuned it patiently. Missed detections fell. Responders trusted the output. Then a colleague, Mateus, tightened a baseline window to react faster to a new payment service, saved the file over the old one, and moved on.

Two weeks later, weekend traffic began triggering incident groups that looked nothing like the weekday ones. Nobody connected the two events. The file had been overwritten, the edit lived only in someone’s memory, and the memory was vague about whether the change had been the window, the grouping rule or both. The team spent most of a week arguing from hunches.

What an analytical review found

When the group finally sat down to reconstruct events, they used the only evidence available: the platform’s own audit trail, which recorded that the file had changed and nothing else. They could see timestamps but no reasons. Here is how they summarized what they could and could not establish:

• Known: the file changed on a specific date, and weekend behavior shifted about a fortnight later.
• Unknown: which of three edits made that day was responsible, because the three edits had been saved as one.
• Unknown: what the earlier, better-performing values had been, since only the final numbers survived.
• Unknowable: whether the original tuning had been as good as people remembered, or whether recollection had flattered it.

That last point mattered more than anyone expected. Without a record, even the good old days could not be verified. A team that cannot reproduce its best state cannot be sure it ever had one.

The habit they adopted

The fix was undramatic. Every meaningful change to the detection configuration now produces a new dated version instead of replacing the old one. Each version carries a single line stating what changed and what the author expected to happen. Not an essay, one line: “Shortened baseline window for the payment service, expecting faster reaction to deploys.” Dara called it the cheapest piece of discipline she had ever introduced, because it cost thirty seconds per change and removed arguments that had cost days.

Three conventions kept the habit light enough to survive busy weeks:

1. One change per version. If you altered the window and a grouping rule, that is two versions, even when you did both in one sitting.
2. Dates in the version name, written the same way every time, so a sorted list reads as a timeline.
3. The expectation goes in the note, so a later reader can tell whether the change did what its author hoped.

Comparing versions turns opinions into evidence

Within a month the habit paid for itself. A new grouping rule looked promising in a demo and produced oddly fragmented incidents in production. Because the previous version still existed, the team placed the two side by side, replayed a week of recorded signals through each and saw the difference directly. The rule was reverted in an afternoon, and the note on the reverted version recorded why, so the idea would not be re-proposed in six months by someone who had never heard the story.

That second benefit is easy to overlook. A version history does not only let you go back; it keeps the reasoning for the choices you declined. Teams that lose their past tend to repeat their experiments, and they pay for them again each time.

Where this connects to the broader practice

The same approach applies to anything a team tunes by hand. For teams that have not yet brought this kind of tooling in, our walkthrough on bringing automated operations into an IT group covers the groundwork, and the habit described here fits comfortably on top of it. For readers wondering how recorded reasoning supports choices under pressure, there is also a related piece on how analytics can inform IT decisions.

The same logic holds for any reusable instruction people refine over time. A library like Promptosia, built around keeping reusable prompt instructions with their versions and a public page for sharing, rests on the same observation: a first draft is seldom the best one, and the draft you overwrite is often the one you wanted later.

Practical guidance if you want to start

You do not need new tooling to begin. A repository that stores the file, or even a folder of dated copies with a short notes file beside them, will do the job. What matters is the sequence: copy first, edit second, note third. Reverse that order and the habit collapses the moment someone is in a hurry.

Review the history occasionally, too. Once a quarter, read the notes from top to bottom. Patterns appear: which kinds of changes tended to help, which tended to backfire, which engineers wrote notes worth reading. The platform group found that their riskiest edits were nearly always the ones made late on Fridays, a lesson that would have stayed invisible in an overwritten file.

What changed for the team

By the end of the year the group could answer questions that had once started arguments. When did the weekend behavior change? Which edit was responsible? What did the settings look like before? Each answer took minutes. More importantly, engineers stopped fearing the file. Tuning had felt risky because mistakes were permanent; with every state preserved, experimenting became an ordinary part of the work, and the file improved faster than it ever had under the cautious regime.

None of it demanded cleverness, only steadiness. It required treating each good state as worth keeping and each change as worth a sentence. If your detection settings are overwritten in place today, start with the next edit: copy, date, note, then change.