Ask an AIOps engineer why a correlation rule was changed three months ago and the honest answer is often a shrug. Somebody tuned a threshold, somebody retrained an anomaly model, somebody added a suppression window during a busy week, and the reasoning went home with them. An AIOps working log closes that gap. Kept by whoever makes the change, each dated entry notes what they tried, what the system did in response, and what they plan to try next. Below, the method is laid out in seven stages, with attention to the evidence it needs to hold so that later decisions rest on something firmer than recollection.

Why an AIOps working log beats memory

Operations work driven by machine learning has an awkward property: cause and effect are separated by time. You adjust a baseline on Tuesday, and the consequence shows up as a missed incident two weeks later, or as a quiet improvement that nobody can attribute. Human memory is poor at linking those two moments. People recall the dramatic outage, not the small configuration change that preceded it.

Research on expertise has long pointed in the same direction. Practitioners improve fastest when they get feedback on specific decisions, and feedback is only usable if the decision was captured when it was made. A log turns each adjustment into a small experiment with a written hypothesis. After a month, the entries form a dataset about your own environment, one that no vendor documentation can supply.

It also changes handovers. When an engineer rotates off a platform, a colleague reading their log inherits context, not just credentials. That is the practical payoff most teams notice first.

Step 1: Decide what counts as an entry

Before choosing a tool, settle the scope. An entry is warranted whenever you change something that affects detection, correlation, routing or automated remediation, or whenever you learn something unexpected about how the platform behaves. Routine status checks with no finding do not need recording. Keep the bar low enough that people actually write, yet high enough that the log is not padded with filler.

A useful test: would a colleague taking over tomorrow need to know this to avoid repeating your work? If yes, it goes in.

Step 2: Use a fixed, four-line format

Structure matters more than prose quality. A compact template keeps entries comparable across people and weeks:

• Date and scope — the day, the service or model affected, and your initials.
• What I tried — the specific change or investigation, with the old and new value where relevant.
• What happened — observed effect, including “no visible change yet”.
• Next — the follow-up, and when you intend to look again.

Four lines take a couple of minutes. The discipline is in filling the “what happened” line honestly, even when the answer is disappointing. Negative results are some of the most valuable entries, because they stop the next person from walking the same dead end.

Step 3: Put the log where the work already happens

A log that lives in a separate application tends to be forgotten. Better options are a markdown file in the same repository as your detection rules, a page in the team wiki linked from the platform’s runbook index, or a dedicated channel where each message follows the template. What matters is proximity: writing the entry should be one step away from making the change.

Version control has a particular advantage. When rule changes and log entries sit in the same commit history, you can line up a configuration diff with the reasoning behind it years later.

Step 4: Record evidence, not impressions

This is where an analytical habit pays off. “Seemed to help” is an impression. “Correlated groups for the payments cluster dropped from several per shift to one over three shifts; sample size small” is evidence with an honest caveat. You do not need statistical rigour for every entry, but you should name what you looked at and over what window.

Practical evidence for AIOps entries includes:

• Links or identifiers for the incidents you reviewed.
• The time window of data the model was trained or evaluated on.
• Which signals were included or excluded, and why.
• Any human override you made, so nobody later mistakes it for model behaviour.

If you are still building the case for the discipline itself, the site’s practical walkthrough on introducing AIOps to an IT team without missteps pairs well with this habit, since early adoption is exactly when undocumented changes pile up.

Step 5: Close the loop with a weekly read-back

Writing is half the value; reading is the other half. Once a week, spend fifteen minutes scanning the last seven days of entries. Look for “next” lines that never got a follow-up, experiments whose results are still pending, and patterns such as the same service appearing repeatedly. Add a short summary entry noting what was confirmed, what was abandoned and what remains open.

This read-back is personal, not a team ceremony. It is the moment when scattered observations turn into a working theory about your environment.

Step 6: Tag entries so they can be found later

After a few months, a flat log gets hard to search. Lightweight tags solve most of that: the service name, the model or rule identifier, and a type such as tuning, investigation or rollback. Resist building a taxonomy; three or four consistent tags are enough. When someone asks why the storage anomaly detector behaves the way it does, a tag search should surface every relevant entry in seconds.

Step 7: Make handovers read from the log

When work changes hands, whether for leave, a rotation or a new role, the handover note should be a short digest of the log rather than a fresh write-up from memory. Point to the open “next” items, the experiments still running and the changes most likely to surprise someone. The incoming engineer can then read the underlying entries for detail.

Teams that do this find that live troubleshooting improves too. During an incident, being able to check what changed recently, and why, shortens the path to a hypothesis; the site’s primer on working through IT problems as they happen describes that investigative flow from the responder’s side.

Learning from how other practitioners log

It helps to see how others structure their notes before settling on your own style. AI Universe, a blog covering AIOps, machine learning and AI security, has a Stories area where people in the field post their day-by-day notes, a handy way to compare entry formats and the level of detail people find worth keeping.

Common mistakes to avoid

• Writing only after failures. If entries appear only when something breaks, the log becomes a list of grievances with no baseline of normal behaviour.
• Logging in retrospect. Entries written days later drift toward a tidy narrative. Capture the moment, then add corrections as new dated entries.
• Editing old entries. Append instead. The value lies partly in seeing what you believed at the time.
• Turning it into procedure. A working log is a personal record of attempts and findings. When something proves reliable, promote it into proper documentation and link back to the entries that justified it.

What to expect after a quarter

The early weeks feel like overhead. By the end of a quarter, most engineers notice three shifts. Repeated investigations become rare, because a quick search shows that a colleague already ruled out the obvious cause. Model and rule changes become easier to defend, because the reasoning is on record. And onboarding new team members speeds up, since the log doubles as an honest history of how the platform reached its current shape.

None of this requires new tooling or a mandate. It needs a template, a place close to the work, and the willingness to write four plain lines after each meaningful change. Start with your next tuning session and see what the record shows you a month from now.