The alert landed at 02:47 and named the wrong culprit. Checkout latency, paginated dashboards, a red line climbing on the ingestion service — everything on screen said the runtime was sick. What the screens could not say was that the sickness had been injected at 17:12 that afternoon, when an engineer with a support ticket and ten minutes of patience edited a replica count by hand and went home. The monitoring stack did its job. The problem was that nobody could see the wound, only the bleeding. This is the blind spot that GitOps was designed to close, and for teams that already run AIOps, it is the most underdiagnosed condition in the estate.

Running a Diagnosis on the Deploy Path

When an AIOps platform flags an anomaly, it answers one question: what changed in behaviour? The follow-up question — what changed in the system? — usually requires archaeology. Someone opens a wiki page, asks around in chat, diffs a config from memory. The reason is structural. In most operations, the production environment is not written down anywhere trustworthy. It is the sediment of hundreds of console sessions, half of them undocumented, any of them capable of undoing last week’s fix.

GitOps attacks that structure directly. The desired state of the environment lives as declared files in a version-controlled repository. A reconciliation agent runs inside or beside the cluster, reads that declaration on a loop, compares it to what is actually running, and corrects the difference. Change the files through a reviewed pull request and the environment follows. Touch the environment directly and it drifts — briefly — before the agent pulls it back. The repository becomes the single answer to “what changed, when, and who agreed to it.”

Anyone who wants the full written model will find it in the Ops Atlas GitOps entry, which is the reference I hand to engineers who ask where this practice is formally described.

Note what this does and does not replace. Your CI pipeline still builds the image, runs the tests, and pushes artifacts to a registry. That work stays. What moves is the final hop: instead of the pipeline holding credentials and reaching into production to apply them, the agent inside production reaches out to the repository and pulls. Security guidance for service-based systems, including NIST SP 800-204C, has been blunt about why this ordering matters — a push model hands cluster credentials to whatever system drives the deploy, and once that system leaks or is abused, the attacker owns the estate. A pull model keeps those credentials inside the perimeter, where the agent needs nothing more than read access to one repo path.

Why GitOps and AIOps Belong on the Same Rota

The two practices are frequently confused and rarely combined well. AIOps is sensory: it watches behaviour, correlates events, and tells you something is off before customers do. GitOps is motor control: it decides what the system should look like and quietly enforces that decision forever. One without the other leaves a familiar gap. Monitoring without a declared baseline can only describe deviations from an unknown normal. Declared state without monitoring can only enforce the shape you wrote down, not flag the behaviour it produces.

Put them together and the diagnosis chain shortens dramatically. An anomaly fires. The responder opens the repository instead of a shell. The git log for the affected manifests shows exactly which merged pull request touched the workload, who reviewed it, and what the previous version looked like. Rollback is a revert commit, applied by the same reconciler that applied the mistake. No snowballing of manual “temporary” fixes that later become permanent. Teams that already invest in reducing event noise with smart tooling will recognise the pattern: every alert should be traceable to a cause, and cause should be a queryable fact, not a memory exercise.

The Habits That Make the Loop Trustworthy

A reconciliation agent is only as honest as the declaration you feed it. Four working rules keep the loop clean:

• Declare the state, don’t script the steps. Describe what should exist — deployments, config, ingress, quotas — rather than the sequence of commands that produces it. A scripted apply that someone must remember to run is not reconciliation; it is the old world with extra syntax.
• Keep history immutable. Tags move, branches get rebased, and suddenly “yesterday’s config” is a matter of opinion. Pin deployments to digests or immutable revisions so the agent can verify rather than assume.
• Review everything that reaches production. The pull request is the change-control gate. If a person, a bot, or an AI assistant can push to a watched path, that actor effectively has deploy rights and should be scoped and reviewed like any deployer.
• Expect drift and decide its treatment in advance. Autoscalers rewrite replica counts; webhooks fill in fields. Pre-agree which differences the agent reverts, which it tolerates, and which page a human.

That last rule matters more than people expect. Self-healing is a feature until three in the morning, when an incident responder needs to shut something down quickly and the reconciler keeps undoing the fix. Mature setups give responders a documented break-glass route — suspend the application, make the emergency change, restore control afterwards. Write the procedure before the incident, rehearse it once, and the agent stays a colleague instead of becoming an opponent.

Adopting GitOps Without Boiling the Estate

The migration pattern that works is boring on purpose. Choose the store and the layout first: one repository or several, directories per environment rather than long-lived branches, and a naming scheme a newcomer can read unaided. Bootstrap the reconciler so that it manages its own installation — the tool that enforces the rules should itself be declared by them. Then onboard one namespace at a time, with pruning switched off while trust is being built. Nobody should ever lose a workload to an over-eager first week.

Secrets deserve a deliberate answer before the first workload lands. Plain credentials have no business in a repository; the common patterns encrypt them into the repo (SOPS with an age key, Sealed Secrets) or keep them in an external vault and let an operator sync them in. Pick one, document it, and resist the urge to run three schemes in parallel. For the people side, a phased plan borrowed from bringing AIOps into an IT team without friction transfers almost unchanged: small pilot, visible win, then widen.

What Good Looks Like Ninety Days In

Measure the loop, not the tooling. Three numbers tell most of the story. First, how long a merged change takes to become a healthy, serving workload — minutes, not days. Second, how often the live state disagrees with the declared state, and whether each disagreement had a ticket. Third, how long the environment stays out of agreement before the agent or a human closes the gap. Publish them where the team can see. Decision-support platforms that turn operational data into choices are exactly the right consumer for these figures, and the combination turns deployment hygiene into a managed metric instead of a vibe.

The one-line diagnosis: if an alert can fire for a change that no repository records, your monitoring is watching a system that nobody fully controls. Declare the state, let an agent enforce it, and spend your on-call energy on genuine anomalies instead of archaeology.

None of this demotes the AIOps investment — it finishes it. The sensors stay, the correlation stays, the anomaly models stay. What changes is that the system they watch finally has an authoritative description of itself, and every deviation from that description is either corrected automatically or escalated with a clean paper trail. Fewer 02:47 mysteries. Faster answers. And an estate where the fix you shipped last Tuesday is still in place on Friday, because something tireless is holding the line.