Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!
We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!
Learn from Guru Rajesh Kumar and double your salary in just one year.
The invoice arrived on a quiet Monday, and nobody on the platform team could explain it. A spending ceiling would have prevented the whole episode, but we did not have one, so what followed was a week of detective work. This is the story of that week, written as a diagnosis: the symptom, the false leads, the real cause, and the habit that now keeps our experiments honest.
If you run machine learning or AIOps experiments on rented infrastructure, some of this will sound uncomfortably familiar.
The symptom: a bill that grew while nothing shipped
Our team had been trialling an anomaly-detection model against a stream of operational telemetry. It was a modest research task. A couple of engineers, a few notebooks, a cluster that was supposed to spin up for training runs and disappear afterwards. The bill, however, climbed week after week, and the climb did not track with anything we could point to. No release had gone out. No new customer data had arrived. The only thing that had changed was that people felt comfortable, and comfort, it turns out, is expensive.
The first instinct in a situation like this is to open the cost explorer and stare at the largest bar. We did exactly that. The largest bar was compute, which told us nothing, because compute was always going to be the largest bar. What we needed was a question the dashboard could not answer: who decided that this much was acceptable?
Nobody had. That was the diagnosis in one line, and it took us three days to reach it.
False lead one: blaming the model
The loudest theory in the room was that the model was simply hungry. Anomaly detection over high-volume telemetry is heavy, the argument went, so of course it costs a lot. That theory was comfortable because it implied there was nothing to fix. We would just have to pay.
It fell apart when we compared what the training jobs actually needed against what they had been given. Several jobs had been launched with generous instance types because the engineer launching them did not want to be the person whose run crashed for lack of memory. Padding the request felt like diligence. Multiplied across many small runs, it became the real source of the growth. The model was not hungry; the people feeding it were anxious.
This is the part of the story that rarely makes it into postmortems. Overspending on experiments is usually not recklessness. It is caution with no upper bound. Every individual decision is defensible, and the sum is not.
False lead two: blaming forgotten resources
The second theory was the classic one. Somebody must have left something running. We did find a few orphaned volumes and one idle notebook server, and we felt a brief, satisfying rush of having caught the culprit. We shut them down and waited for the bill to flatten.
It did not flatten. The forgotten resources were real, but they were a rounding error next to the pattern underneath. Cleaning up strays is hygiene, and hygiene matters, but it treats the symptom. If the underlying habit stays the same, new strays appear the following month, wearing different names.
The real cause: no ceiling, so no stop signal
Once we stopped hunting for a villain, the structure of the problem became visible. We had never written down a maximum. Without a maximum, there was no moment at which anyone was entitled to say “that is enough.” Each run was judged only against its own usefulness, never against the total, and usefulness is a famously elastic standard when you are excited about a project.
A spending ceiling is the opposite of an elastic standard. It is a number, agreed before the experiment begins, that represents the most the team is willing to pay to find out whether the idea works. Crucially, it is chosen while nobody is attached to the result. Once the work is underway, every extra run feels justified, so the limit has to come from a calmer moment.
Our team had talked about how to introduce AIOps into an IT group without chaos, and in that conversation we had covered tooling, people, and process. We had not covered money. That gap was ours, and it is common. Teams plan the technical rollout in careful detail and leave the budget as an afterthought, which means the budget gets decided by default, by whoever clicks the launch button.
The fix: write the number down first
Here is the routine we adopted, in the order we follow it now.
- Decide the ceiling before opening a notebook. We ask what the experiment is worth if it succeeds, and what we would be content to lose if it fails. The answer to the second question becomes the ceiling. It goes into the experiment’s description page, in plain text, where everyone can see it.
- Choose the shape of the run to fit under it. If the ceiling is modest, we use smaller instance types, a sampled slice of the telemetry, and fewer training epochs. The limit drives the design, instead of the design producing a bill that we then try to excuse.
- Set an alert at a fraction of the ceiling. The cloud console lets you define a budget with notifications. We set one at around three quarters of the cap, and another at the cap itself. The first is a nudge; the second is a stop signal.
- Treat the stop signal as a stop. When the second alert fires, the experiment pauses. People can ask for more, but they have to make the case in writing, and the extension becomes a new, smaller ceiling with its own alert.
- Review after the experiment, not during it. We compare the actual spend to the ceiling and note which choices stretched it. That note feeds the next ceiling.
None of this is sophisticated. The sophistication is in doing it before the work starts, when the temptation is to skip it because the real task feels more interesting.
What changed in the second round
The next experiment on the platform was a forecasting model for capacity planning. Before anyone touched a cluster, we wrote the ceiling in the project page and argued about it for twenty minutes, which was the most productive twenty minutes of the whole effort. One engineer wanted a higher number to leave room for exploration. Another wanted lower to force discipline. We settled in the middle and agreed that the first alert would trigger a short conversation instead of a panic.
Something surprising happened. The experiment finished sooner than the previous one, and it produced a clearer answer. Having a limit forced us to define what “done” looked like. We stopped chasing marginal accuracy gains that cost a great deal and meant very little. The team described it afterwards as the difference between exploring a forest with a compass and wandering in it with a snack budget.
The ceiling also changed the tone of the conversations. Before, a large run was a quiet act of private worry. Now it was a visible choice against a shared number, and people began to ask each other things like, “Can we get the same signal from a tenth of the data?” Often the answer was yes.
Diagnosing your own version of this
If your cloud or tooling bill feels like it has a mind of its own, ask yourself a few plain questions before investing in any new dashboard.
- Is there a written number anywhere that says how much this piece of work may cost?
- Who would be entitled to stop it if the number were reached?
- Do alerts reach a person who can act, or a channel nobody reads?
- Are runs sized to the question, or sized so nobody gets blamed for a crash?
If the honest answer to the first question is no, you have your diagnosis. Everything else is tuning. A beginner-friendly companion to this habit is learning to solve operational problems as they happen, because a ceiling and a calm troubleshooting routine reinforce each other: the limit gives you time to think, and the routine tells you what to do with it.
The same idea shows up far from engineering. A restaurant table-reservation platform such as DineGlobe, which lets guests book directly with a venue, reminds me that a dinner out also benefits from a number chosen at the start of the evening rather than the end. The principle is not special to cloud work. It is just easiest to see when the invoice is the thing that hurts.
The short version
We spent a week looking for a culprit and found a missing habit instead. The bill grew because no limit existed, so no moment existed at which anyone could say stop. Write the number down before the first run. Shape the experiment to fit beneath it. Set an alert short of the line, and another at it, and let the second one mean what it says. Then review, adjust, and do it again with a little more evidence in hand.