The Cost of Turning It Off and On Again

The restart fixes the outage and destroys the only copy of the crime scene

The most reliable trick in computing is also the most dishonest. When something misbehaves, you turn it off and on again. The frozen laptop comes back. The stuck phone app reopens clean. The production service that had been throwing errors for twenty minutes goes quiet, the dashboards return to green, and the people watching them breathe out. It works often enough that we’ve stopped thinking of it as a guess. We treat it as a cure.

It isn’t a cure. A restart erases volatile state, resets counters, closes connections, reallocates memory, changes timing, rotates leadership, and clears the exact arrangement of conditions that produced the failure. It does not prove the system is healthy. It proves only that the system is no longer in the state you needed to examine.

That distinction sounds academic until you’ve lived through its consequences. A system that breaks for reasons you don’t understand will break again. The restart buys you time, and it charges for that time in a currency you don’t notice you’re spending: the evidence.

What a restart actually does

Picture the moment before the restart. A process has been running for nine days. Inside it, memory has filled with the accumulated residue of everything it has done. Connection pools hold sockets in various stages of health, some open, some half-closed, some waiting on a reply that will never come. Caches are warm with data shaped by the last week of real traffic. Counters have climbed to numbers nobody planned for. Timers are mid-flight. A queue is backed up in a particular way, and the order of items in that queue encodes the precise sequence of events that tipped the system over.

This is the crime scene. Every detail in it is a clue. The strange thing about software failures, especially in older and more complicated systems, is that the cause is rarely sitting where the symptom appears. The symptom is downstream. The cause is some unlikely arrangement of conditions that had to line up just so, and the only place that arrangement still exists is in the live, broken process in front of you.

Now restart it. Memory is wiped and handed back fresh. The pools rebuild from nothing, every connection new and healthy. The caches empty and slowly refill. The counters drop to zero. The timers vanish. The queue drains. The system that comes back up is not the system that failed. It’s a different system that happens to run the same code, and it has no memory of what went wrong because the thing that went wrong lived in the memory you just threw away.

You haven’t fixed anything. You’ve reset the dice and rolled again, and this time they came up fine. The conditions that produced snake-eyes are gone, so of course the next roll looks healthy. It tells you nothing about whether the dice are loaded.

Why we do it anyway

None of this means restarting is wrong. Most of the time it’s the right call, and pretending otherwise would be a kind of engineering vanity that customers don’t owe you.

When a service is down, real people are affected. Orders aren’t going through. Someone’s payment is hanging. A doctor can’t pull up a chart. The clock on the outage is running, and every minute you spend poking at the broken process to satisfy your curiosity is a minute the people who depend on the system stay stranded. Recovery has a moral weight that investigation doesn’t. The fastest path back to working is usually the restart, and choosing it is often the responsible thing to do.

So the tension is real, and it doesn’t resolve cleanly. You want the system back now. You also want to know why it broke, because a cause you don’t understand is a cause that will return, possibly at a worse hour, possibly in a way the restart won’t fix next time. Both wants are legitimate. They point in opposite directions, and the restart serves only one of them.

What goes wrong is when we let the first want quietly erase the second. The incident gets marked resolved. The write-up says the service was restored by restarting the affected nodes. The field for root cause stays empty, or fills with a phrase that sounds like an answer while saying nothing: “transient issue,” “intermittent fault,” “self-corrected.” Those phrases are how an organization admits it has no idea what happened without admitting it out loud. The outage is over. The mystery is intact. And the next time it strikes, you’ll be exactly as ignorant as you were the first time, having learned the one lesson that the restart so generously teaches: that you can make the symptom go away without ever meeting its cause.

The discipline of the pause

The way out is not heroics. It’s a small habit, practiced before the pressure hits, of capturing evidence before you destroy it.

A restart takes seconds. Capturing the state of a process before you restart it can take seconds too, if you’ve set yourself up to do it. A thread dump. A snapshot of memory. The current contents of the queue. The connection table. The recent log lines that haven’t yet been flushed. The output of whatever diagnostic the system exposes about its own internals. None of this brings the service back any slower if you’ve made it routine, and all of it preserves the thing the restart is about to erase.

This is what people mean, at bottom, when they talk about observability, though the word has been worn smooth by marketing. Observability isn’t a wall of dashboards. A dashboard answers a question you already knew to ask. The failures that matter are the ones nobody anticipated, and they produce questions you’ve never asked before. What you need in that moment isn’t a prettier chart. It’s the ability to ask a brand-new question of evidence you had the foresight to keep. A system you can actually debug is one that can testify about its own state before that state is wiped.

The same logic applies to the machinery that restarts things for us. Modern infrastructure restarts constantly and automatically. An orchestrator notices an unhealthy node and replaces it. A watchdog kills a process that stops responding. A health check fails and traffic reroutes. Each of these is a tiny restart performed by a robot, and each one, by default, throws away the evidence just as thoroughly as a human hitting the button, only faster and more often. A system that heals itself without recording why it needed healing is a system optimizing itself into permanent ignorance. If automation is going to restart things on your behalf, the least it can do is take a snapshot on the way through. Make the recovery leave a trail.

Recovery is not understanding

There’s a cultural piece to this that’s easy to miss. Organizations reward the person who brings the system back. At two in the morning, when everything’s on fire, the engineer who restarts the cluster and ends the outage is the one who gets thanked. That instinct is fair as far as it goes. Ending the outage matters.

But the person who, three days later, works out that one particular hardware model corrupts data after a specific sequence of operations, and proves it, and gets it fixed so it never happens again, has done something more durable, and that work is usually invisible. It doesn’t come with a dramatic moment. There’s no green dashboard to point at. There’s just a failure that stopped recurring for reasons most people will never know to appreciate. A team that only celebrates the restart is training its best people to value the rescue over the explanation, which is a slow way of guaranteeing the same fires keep starting.

“It went away” is an outcome. It is not a root cause, and the gap between the two is where the next outage is already forming.

So the next time something breaks and your hand drifts toward the restart, pause for the length of a held breath. Ask what state is about to vanish, and whether you can keep a copy of it before it does. Then restart, with a clear conscience, having taken the evidence with you. The system will come back either way. The difference is whether it comes back as a mystery you’ve solved or a mystery you’ve merely postponed.