The Quarter a Small Ops Team Learned to Count Disk Days

Upgrade & Secure Your Future with DevOps, SRE, DevSecOps, MLOps!

We spend hours scrolling social media and waste money on things we forget, but won’t spend 30 minutes a day earning certifications that can change our lives.
Master in DevOps, SRE, DevSecOps & MLOps by DevOps School!

Learn from Guru Rajesh Kumar and double your salary in just one year.


Get Started Now!

The story below is invented, and any team that has run a handful of Linux servers will recognise its outline. A small platform group, six people and about forty hosts, spent a quarter proving to itself that SysOps work is mostly about boring arithmetic and written-down habits. Nothing in it needed a clever model or a new product. It needed someone to ask what the machine was actually doing.

The group ran an event-correlation stack, and the stack lived on ordinary servers. When a data volume on one of those servers filled up on a Friday evening, the correlation engine stopped ingesting, and for ninety minutes nobody could tell whether the silence meant calm or breakage. The post-mortem was short. The disk had not “suddenly” filled. It had been filling for weeks, and no one had done the sum.

The sum nobody had done

The Ops Atlas entry on SysOps as a discipline of keeping servers healthy describes systems operations as the upkeep of services, users, logs, scheduled jobs, packages, filesystems, clocks and performance triage on a host. The team read it during the retrospective mostly for its worked example on disk runway, because it matched their incident almost exactly.

The Atlas example uses a 500 GiB ext4 volume with 410 GiB used and 63 GiB of growth over seven days. That is nine gibibytes a day. A naive reading says ninety gibibytes of room divided by nine is ten days. The entry points out that the root-reserved five percent, roughly 25 GiB, is not available to ordinary applications, so applications hit a wall near 475 GiB and the real runway is about seven days. Seven, not ten.

The team applied the same arithmetic to their own volume and found they had been wrong by almost exactly that margin. The alert threshold sat at 90 percent, which on that volume left under two days of headroom at the observed growth, and the pager fired at 02:00 on a weekend. The threshold was reasonable in the abstract and useless in practice, because it said nothing about the rate.

What they changed in the first thirty days

They stopped debating tooling and wrote a short list. Each item was something a person could finish in an afternoon.

  • An inventory of every host, with an owner column that could not be blank.
  • A weekly job that records used space and growth per filesystem, and a dashboard cell that shows days of runway instead of a percentage.
  • A one-page triage script that prints load, memory, disk, network and recent kernel messages in a fixed order, so that the person on call starts from the same facts every time.
  • Removal of one shared administrator login that half the team had been using for years.

The Atlas sheet suggests a similar first month: inventory, retire shared root, install host statistics collection, write a triage script and wire a small set of alerts. The team did not copy the sequence deliberately. They arrived there independently and then noticed the overlap, which is a decent sign that the sequence is sensible rather than fashionable.

The restart that almost destroyed the evidence

The second incident came three weeks later. A host reported high load and a nervous engineer reached for a restart, which is the reflex the Atlas lists first among its anti-patterns because a restart wipes the very state you need to see. A colleague asked for ten minutes first. They ran the triage script and found that load was high while the processors were nearly idle. The load figure on Linux counts tasks stuck waiting on storage as well as tasks that are running, so a sick network mount had been inflating it.

Had they restarted, the mount would have come back, the load would have dropped, and the cause would have survived to hit another host the following week. Instead they wrote down what they saw, fixed the mount options and added a line to the triage script that lists tasks in uninterruptible sleep. The notes went into the team wiki in plain words: what looked wrong, what was actually wrong, how to tell the two apart.

If you are building a similar habit on an operations team, the article on a newcomer’s approach to solving incidents as they happen covers the mindset side, and it pairs well with the host-level routine described here.

Days 30 to 90: turning habits into records

Month two was less exciting and more valuable. The team moved their scheduled work from old cron entries to systemd timers, partly because a timer can retry a missed run after a reboot and can trigger a follow-up unit when it fails. They capped the size of their system journals explicitly rather than leaning on defaults. They also agreed that a reboot would never be done on a host until its mount table had been checked, because a stale entry in the filesystem table is a classic reason a machine does not come back.

Month three was about proving things worked. They restored a random sample of backup data onto a scratch machine, which turned up one job that had been writing empty archives for a while. They rehearsed logging into a dead machine via its remote management console. They put the end-of-support date of each operating system release on a shared calendar. The Atlas maturity ladder tops out at a rung it calls replaceable, and it defines that rung by exactly this kind of evidence: drill records, console records, restore records.

Where event noise fits

Because this team sells correlation to others, they were tempted to say smarter alerting would have caught the disk problem. They concluded the opposite. Their write-up on taming a flood of alerts with better tooling still stands, but noise reduction only helps when the underlying signals are sound. A growth-rate figure computed correctly from a single host beat any amount of statistical cleverness applied to the wrong threshold.

That is not a case against analytics. It is a reminder of sequence. Fix the signal at the host, then aggregate, then correlate. Skipping the first step produces dashboards that look sophisticated and answer nothing.

Lessons worth keeping

  1. Express capacity in time remaining, not percent used. A number of days can be compared against how long a purchase or a clean-up takes.
  2. Collect evidence before changing anything. The restart can wait ten minutes.
  3. Record every manual edit somewhere a colleague can find it, even if the record is only a line in a shared log.
  4. Do not call something finished until a restore, a console login and a reboot have each been tried at least once.
  5. Treat cloud-hosted virtual machines like any other server. The guest operating system remains yours to patch.

None of these points belong to one vendor or one year. The team’s quarter ended with fewer pages, a calmer rota and a wiki that new joiners actually read. The correlation engine was never the star of the story. A spreadsheet row labelled “days of runway” was.

Related Posts

KubernetesOps for Beginners: One Month, One Glossary

Follow a new engineer through her first month of KubernetesOps, with a plain-English glossary covering requests, drains, upgrades and the version calendar.

Read More

InfraOps in 2026: Where Infrastructure-as-Code Practice Stands

A state-of-the-field look at InfraOps: licensing splits, secrets in state, drift measurement and the pipeline as an attack surface.

Read More

CloudOps for AIOps Folks: Eight Friendly Habits for Calmer Clouds

Eight easygoing habits that help AIOps-minded teams run the provider-facing layer of a cloud estate with fewer surprises and quicker diagnosis.

Read More

A DisasterRecoveryOps Audit Checklist for Production Teams

Seven sections, one working session: a practical audit of recovery objectives, decision deadlines, failover order, access paths and drill evidence.

Read More

Six BackupOps Myths That Quietly Break Recovery Plans

Green dashboards, replicas and locked vaults feel safe. Six common BackupOps myths, why they fail, and what to do instead.

Read More

The ObservabilityOps Career Path, Told as a Short Story

A generic engineer moves from dashboard reader to owner of the telemetry system itself. Here is the path, the skills and the first ninety days.

Read More
0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted
0
Would love your thoughts, please comment.x
()
x