A site called in because its pay-at-pump terminals had stopped taking cards. The terminals said their link to our service was up. The pumps were stuck, no card sales went through, and the only thing that fixed it was someone restarting our service by hand.
Both sides were telling the truth as they saw it. The terminal had reconnected after a network blip and was perfectly healthy. Our service was still holding the old connections, which had died without a word. This post is about half-open TCP connections, why a dead socket looks exactly like a quiet one, and the small, switched-off-by-default monitor a colleague and I built so the service stops trusting the other end to say goodbye.
This is Part 4 of Forecourt Engineering. Part 3 was about every client reconnecting at once. This one is the opposite problem: a client reconnects, and the service never notices it left.
A dead peer doesn't throw
The weak assumption, and I've written plenty of code that makes it, is that when the other end goes away, you'll find out. The connection closes, a read returns zero, an exception fires, and your cleanup code runs.
That's true when the other end closes politely. A graceful close sends a message saying so, and your socket hears it.
On a forecourt, nobody closes politely. A network switch reboots, a cable gets knocked, a site router drops its lease, and the packets simply stop. The terminal notices, because it's actively trying to talk, so it reconnects on a fresh socket and reports its link as healthy. Our side was listening, not talking, so it heard nothing at all.
A socket in that state is half-open. Your end still believes the connection exists. Reads block waiting for data that will never come, and nothing errors, because from TCP's point of view nothing has happened yet. Silence from a dead peer and silence from an idle peer look identical.
TCP keepalive exists for exactly this, but the operating system defaults are far too slow to help. On Windows, the default idle time before the first keepalive probe is measured in hours. By then the site has rung support, and someone has already restarted the service.
What we already had, and the gap in it
Our service already detected silence from one direction. The terminal sends regular messages, and each one pushes out an expiry time for that terminal. If the expiry passes with nothing received, the service drops the terminal's connections and signs it out.
That's an application-level heartbeat, and it's the right tool for "the terminal has gone quiet". It catches a terminal that stops talking to us.
It didn't catch this. Here, the terminal hadn't gone quiet. It had reconnected and was chatting happily on its new connection. The problem was that the workers on our side that listen for the terminal's connections were still holding the old, dead sockets, and nothing made them start listening properly again.
So the terminal thought the link was fine, our heartbeat thought the terminal was fine, and card sales still didn't work. Every component's health check passed, and the system as a whole was broken. That's the kind of failure no single health check will ever find.
The fix: track the timestamps, restart what's stale
The fix was co-delivered, and I want to be precise about who did what.
I started it. I added a switch to turn it on, and a background task that looks at when each of a terminal's connections last connected and last disconnected, decides whether one has been down too long, and restarts it if so. I also added the logging you need to see why it made each decision.
My colleague then reworked the check to restart the whole listener worker rather than just its connection thread, so the worker comes back in a clean state instead of a half-repaired one. They made the staleness threshold tunable per site, and they handled a case that looked broken but wasn't, which I'll come to.
Here's the shape of the decision, as generic C#:
public sealed record LinkVerdict(bool Restart, TimeSpan DownFor, string Reason);
public LinkVerdict Assess(DateTime? lastConnected, DateTime? lastDisconnected, DateTime now)
{
if (lastDisconnected is null)
return new(false, TimeSpan.Zero, "never disconnected");
if (lastConnected > lastDisconnected)
return new(false, TimeSpan.Zero, "reconnected since last drop");
var down = now - lastDisconnected.Value;
if (down < _graceWindow)
return new(false, down, "inside grace window");
return new(true, down, "down longer than grace window");
}The verdict carries the reason and how long the link has been down, not just a yes or no. That came out of review, and I agreed with it straight away. When a monitor restarts something on a live site, the first question anyone asks is "why", and the log line should answer it without anyone reading the code.
Review also pushed me to move the "is this switched on" check out of start-up and into the task itself. Start-up reads config once. Reading it on every tick means the switch and the interval can be changed at runtime, without a release or a restart.
Why restart the whole worker
My first version restarted the connection. My colleague's version restarts the whole listener worker, and I think that's the more important of their changes.
A listener that's been holding a dead socket may have other state that's gone wrong with it: buffers, pending reads, whatever it believed about the terminal on the other end. Restarting just the connection leaves all of that in place and hopes it's fine. Restarting the worker throws it away and starts from a known state, which is the only state you can actually reason about.
Why not just tune keepalive
You can turn TCP keepalive on per socket in .NET and shorten its timings, and for some services that's the right answer. It would tell us sooner that a socket was dead.
It wouldn't restart the listener, though. Knowing a connection is dead and doing the right thing about it are separate problems, and the second one needs application code either way. The timestamps approach gives us the decision, the reason and the restart in one place, and it works the same whatever the operating system's keepalive settings happen to be on a given site.
The case that looked broken but wasn't
This is my favourite bit, and it's my colleague's work.
When a terminal reconnects quickly, the events can land out of the order you'd expect. The new connection can be recorded before the old connection's disconnect is. On timestamps alone, that looks exactly like a stale link: the most recent event is a disconnect.
But the link is perfectly healthy. The terminal is connected and talking; we just heard about the old socket's death a moment after we heard about the new socket's birth.
Restart it, and you've broken a working connection to fix a problem that doesn't exist. So there's a short grace window after a disconnect inside which the monitor leaves the link alone and lets the picture settle. A monitor that restarts healthy links is worse than no monitor, because it creates the outage it's meant to prevent.
Why it shipped switched off
The monitor went out behind a switch that's off by default. Each site gets it only when someone turns it on.
That was deliberate. Anything that restarts connections on a live forecourt could cause the very outage it's meant to prevent if it's wrong, and "wrong" includes a threshold that suits one site's network and not another's. Off by default means we turn it on where the problem actually is, watch it, and roll back with a config change instead of a release.
Later, another site showed the same pattern on an older version of the service. The advice there was three things: upgrade, turn the monitor on, and fix the site's network, which was the real cause.
Rolling it out site by site has a second benefit: each site that turns it on gives you evidence about the threshold, so the defaults get better from real networks instead of guesses.
That last part matters. The monitor doesn't fix a bad network. It stops a bad network from turning into an outage that needs someone to drive out, and that's all it claims to do.
The general pattern
Take away the forecourt and this is a pattern for any long-lived connection in a .NET service, whether it's a raw socket, a message broker consumer or a SignalR hub.
Don't trust the other end to tell you it's gone. Track when each connection last connected and last disconnected, and treat "no news" as a state you have to evaluate, not a state that means "fine". Restart what's been stale for longer than a threshold, and make the threshold tunable, because networks differ.
Most frameworks already have some version of this. SignalR, for example, has server-side settings for how long a client can stay silent before it's treated as gone. The lesson isn't to rebuild that, but to know what those settings are, which side of the connection they protect, and whether anything restarts the component that was holding the dead connection.
Leave a grace window so fast reconnects aren't punished. Log every decision with its reason. And ship anything that restarts things on a live estate switched off, so you can turn it on where it's needed and off again without a release.
Where it falls over
Restarting a link doesn't restore everything the terminal lost. At the second site, the terminal came back still stuck in the pay-at-kiosk mode it had been left in, because the service doesn't re-send the pump mode after a link recovers. The connection was fixed and the forecourt still wasn't quite right. That's a separate gap and it's still open.
Off by default was the right call for a live estate. It also means the sites that need it don't get it until someone asks. I'd like the monitor to raise an alert whenever it restarts something, so we find the sites that need it before they ring us.
And the shared credit is real. I started it and shaped the restart decision and its logging. My colleague made it robust, and the quick-reconnect handling is theirs. If I told this story as mine alone, it would be a better story and a worse one.
Assume the silence is lying
My position: on any network you don't control, a quiet connection is an unknown, not a healthy one. Your code should evaluate it on a schedule, with timestamps and a threshold, instead of waiting for an error that a dead peer will never send.
If you run long-lived connections, look at what your service does when the other end vanishes without closing. If the honest answer is "nothing until someone restarts it", track the timestamps, restart what's stale, leave a grace window, and ship it switched off until you've watched it work.
Next in the series: two timers for the same timeout, one in our service and one in the terminal, and the rule that the shorter one always wins. The rest of my writing is on the blog.