Priya had been on the rota for three weeks when she noticed that her nights all started the same way. A page would arrive, she would open a terminal, and she would type the same six commands in the same order: check the queue depth, look at the last fifty log lines, restart the stuck worker, wait for the queue to drain, confirm the dashboard went green, and write a note. Nothing about it was hard. It was simply repetitive, and the repetition was eating her sleep. This is the story of how she turned that chore into a small script without breaking anything, and how you can follow the same steps for your own manual fix.

Step one: do it by hand, on purpose, a few more times

Her first instinct was to automate everything that very night. Her team lead talked her out of it. His advice was to keep doing the fix manually for another handful of incidents, but this time to treat each one like a field observation. Which commands never changed? Which ones depended on what she saw? Where did she pause to think?

That is the part most people skip. A script written after one incident captures that incident, not the pattern. After four or five, the pattern shows itself, and so do the exceptions. Priya kept a scrap file open during every page and simply pasted each command as she ran it, with a one-line note beside any moment where she had to decide something.

Step two: sort the commands into two piles

When she laid the notes side by side, the picture was clearer than she expected. Four of the six steps were identical every time: reading the queue depth, collecting recent log lines, restarting the worker, and checking the health endpoint. Two steps were never identical. Deciding whether the logs pointed to a bad deploy, and deciding whether a restart was even the right move, took real judgment and a look at context.

This split is the heart of sensible automation. The unchanging steps become a script. The judgment steps stay with a person, and the script is built so that it hands control back to her at exactly those points. A tool that tries to decide everything is how teams end up with automation nobody trusts.

Step three: write the dullest possible first version

Priya’s first script did almost nothing clever. It printed the queue depth, saved the last fifty log lines to a timestamped file, and then stopped and asked her a plain question: restart the worker, yes or no. Only if she typed yes did it restart, wait, and poll the health endpoint. Then it appended a note to the incident file with the time and what had been done.

• It gathered facts and left the verdict to her.
• It wrote down what it did, so the next person could read the trail.
• It refused to run twice at the same moment, so two people could not collide.
• It had a dry-run flag that printed the commands without running them.

That dry-run flag turned out to be the most useful feature. Colleagues who were nervous about running anything new could read exactly what it would do before trusting it. If you want a grounding in how teams think about real-time diagnosis before you script it, the walkthrough on tackling live IT problems as a newcomer covers the habits that make a script like this worth having.

Step four: test it where nothing can be hurt

Before it touched production, Priya ran the script against a staging copy of the worker, then against a worker she had deliberately jammed. She also tried the awkward cases: what happens if the health endpoint never goes green, what happens if the log file cannot be written, what happens if the queue is already empty. In two of those cases the script did something silly, and she fixed it before anyone else ever saw it.

A useful rule here is to make the failure behaviour boring. If anything unexpected happens, the script should stop, say what it saw, and leave the system exactly as it found it. A noisy, early stop is far better than a confident wrong move at three in the morning.

Step five: put a name and a place on it

A script living on one laptop is a liability. Priya moved hers into the team repository, gave it a name that described the situation rather than the mechanism, and wrote four sentences at the top explaining when to use it and when not to. She also listed the two judgment questions from her notes, so that a new person would know the script was a helper and not a replacement for thinking.

Naming matters more than it sounds. Someone half asleep will search for the symptom, not the implementation. A name such as the one she chose, which spelled out the stuck-queue symptom, meant people actually found it.

Step six: review what the script changed, not just what it did

After a month she looked back at the incident file. The script had run on most pages, and the average time spent on the mechanical part had dropped sharply, which freed her attention for the interesting question of why the worker kept sticking in the first place. That second question eventually led to a real fix in the code, and the script was needed less and less.

That is a good outcome and worth remembering: the best automation sometimes exposes the problem it was written to hide. If the same script fires every night for months, treat that as a signal, not a success. Her team now reviews the log monthly with exactly that question in mind. Related thinking on cutting the churn that wakes people up appears in this piece on calming noisy IT events.

Where the same habit shows up outside IT

The pattern is not unique to operations. Anyone who handles a recurring process, from a travel coordinator to a patient-liaison desk, benefits from doing it manually first and then templating what never changes. A good example is the enquiry flow of HipHospitals, a medical tourism site that, according to its own pages, gives overseas patients a way to browse hospitals and doctors per country and request either a no-charge quote or a consultation slot. Behind a flow like that sits plenty of routine admin where a reusable reply template and document checklist can save time. You can see how it presents things on its overseas patient enquiry site, though the lesson stays the same wherever you work: standardise the routine part and keep a human on the judgment calls.

A short checklist you can copy

1. Run the manual fix at least four or five times, pasting every command into a scratch file.
2. Mark each step as identical every time or dependent on judgment.
3. Script only the identical steps, and make the script pause at each judgment point.
4. Add a dry-run mode and a plain written log of what was done.
5. Test on staging, on a deliberately broken case, and on an empty case.
6. Store it with the team, name it after the symptom, and write when not to use it.
7. Review the log monthly and ask what the repeated firing is trying to tell you.

Priya still keeps a scrap file open during unusual pages. Her rule is simple: the first few times, do it by hand and watch yourself. Only when the pattern has stopped surprising you does it deserve to become a script. Automating too early locks in a guess; automating too late wastes your nights. The sweet spot sits in between, and it is found by paying attention rather than by tooling.