If you run a model-backed service, you probably know the feeling: everything looks green on the dashboard, and yet someone writes in to say the answers are timing out. That gap between “the servers are fine” and “people are happy” is exactly what SRE, short for site reliability engineering, was built to close. Let’s go through it together, one friendly step at a time, using an imaginary AI search assistant as our example.

Before we start, a quick bit of background. The Ops Atlas describes SRE as treating operations as a software problem, with targets drawn from what users actually experience. If you want the long version, the Ops Atlas entry on SRE is a thorough read, and I’ll lean on it for the facts below.

Step 1: Decide what “working” means to a user

Start with a sentence, not a tool. For our assistant it might be: “A person asks a question and gets a useful reply quickly.” Now turn it into something countable. SRE people call these counters service level indicators, and the pattern is always the same: good events divided by valid events.

For the assistant you could count two things. First, the share of requests that returned a normal answer instead of an error. Second, the share of requests answered faster than a threshold you pick, say three seconds. Notice that the second one is a proportion, not an average. Averages hide the unlucky tail, and the unlucky tail is where your complaints live.

Measure as close to the person as you can. A probe that sends a real question from outside your network tells you more than the inference server’s own CPU graph.

Step 2: Choose a target short of perfection

This is the part that surprises newcomers. A sensible reliability target is never perfect. The Atlas is blunt about it: chasing every extra nine can cost vastly more than the one before, and users cannot usually tell the difference anyway, because their own phone or Wi-Fi fails more often than that.

Choose a number you can defend, such as 99.9% of requests succeeding over a rolling four-week window. Four weeks suits this well, since every weekday appears equally often and a hectic Monday cannot skew the result.

One more rule of thumb: if your assistant calls another service, that service needs to be more reliable than you promise, by roughly one extra nine. You cannot be steadier than the things you stand on.

Step 3: Turn the leftover into an error budget

The error budget is simply one minus your target. At 99.9% over twenty-eight days, that works out to roughly forty minutes of trouble, or one thousand failures per million requests. Think of it as an allowance you are allowed to spend on shipping new things, running experiments and swapping model versions.

Here is where AI teams get a real benefit. Releases of a model, a prompt template or a retrieval index are all risky changes. When the budget is healthy, ship boldly. When it is nearly gone, slow down and fix reliability first. Writing that agreement down, and getting a manager to sign it, turns an argument into a rule. The Atlas stresses that targets without a signed policy tend to be ignored.

The opposite problem exists too. If your budget is always nearly full, you are either over-delivering or your target is too loose. Both are worth a conversation.

Step 4: Alert on how fast you are burning, not on every blip

Many teams drown in pages. If you have ever read about cutting down the flood of IT event noise with smarter tooling, you already know the cost. SRE offers a calmer approach called burn-rate alerting.

Burn rate compares how fast the allowance is disappearing with an even drip across the whole period. Burn rate 1 means you reach the finish line exactly empty. Burn rate 6 means the money runs out in a sixth of that time.

• Page a human when a fast burn shows up over a short and a long window together, so brief spikes do not wake anyone.
• Open a ticket, not a page, when a slow burn drags on across a day or three.
• Do not alert on raw metrics that no user would ever notice.

The Atlas notes a worked example where a short, sharp incident was caught in minutes while a long, quiet one only produced a ticket. That is the behaviour you want: loud for emergencies, polite for slow leaks.

Step 5: Keep an eye on toil

Toil is manual, repetitive work that a script could do and that grows with the service. Restarting a stuck embedding worker by hand every Tuesday is classic toil. The usual guideline says it should stay below half of anyone’s working hours, and the Atlas mentions that survey averages land lower still, near one third.

Make it visible. Keep a simple tally of hours spent on repetitive tasks per month, divide by total hours available, and watch the percentage. If it creeps over the line, a cap only helps if it has teeth, for example by redirecting tickets back to the developers who built the feature.

Automation also pairs nicely with the idea behind predictive maintenance applied to IT systems, though remember that SRE itself is about the discipline and agreements, not about machine-learning correlation.

Step 6: Learn from every bad day without blaming anyone

Finally, build a postmortem habit. Pick a trigger up front, such as any lone incident that swallows over twenty percent of the allowance, and always write the review when it fires. A good postmortem is blameless but not toothless: it is relaxed about people and strict about actions and owners.

Close your loops. The anti-pattern the Atlas flags is the postmortem whose action items sit forever in a document. Assign each one to a named team, give it a date, and check it at your next review.

A tiny worked example

Suppose your assistant handled fifty million valid requests this month with a 99.9% target. Your budget is fifty thousand failures. If thirty-one thousand requests failed, you hit 99.938%, the target was met, and about 62% of the budget is gone. That is a perfectly healthy month, and it tells you there is still room to try that new model version.

Now imagine a rollout pushes the error ratio to 0.6%. That is six times the allowed rate, so the burn rate is 6, and your remaining budget would vanish in under two days. This is precisely when a fast-burn page earns its keep.

Common stumbles to avoid

• Renaming your existing operations team and calling it SRE without changing how targets or priorities work.
• Setting five-nines targets for every internal tool, which makes every change feel dangerous.
• Averaging percentages from different services instead of adding up the good and total counts first.
• Forgetting services that report no data, which can quietly disappear from your calculations.

Your next move

Nobody needs a big department to start. Pick one user journey, write one indicator, agree one target and watch the budget for a month. If you want help deciding when to back your own judgement, there is a useful piece on how AI can support IT decision-making that complements the budget-based approach nicely.

The real prize is a shared language. When product, engineering and operations all talk about the same budget, releases stop being a tug of war. Start small, keep it honest, and let the numbers do the arguing for you.