Forecourt Engineering, Part 6: The Empty Window at Start-Up

After a restart, the service I work on briefly had no pumps at all. Not an error, not a crash, just an empty list. The code that builds the pump list cleared it first and then tried to refill it, and on the first call at start-up, the data source it refills from hadn't connected yet.

So for a short window, everything that read the pump list quietly worked on nothing. The admin page showed no pumps. Mode changes were skipped. And the step that applies a site's scheduled mode, the one that closes some pumps overnight, ran against an empty list and did nothing.

This post is about start-up ordering bugs: why "clear, then refill" is a trap when the refill depends on something slow, and why the useful question is what every reader sees during start-up, not just after it. It's Part 6 of Forecourt Engineering.

How a small UI ticket led here

I found this because of a much smaller piece of work. Just before it, I'd added icons to the pump cards in our admin page, showing when a pump was in a scheduled mode. Sites can put chosen pumps into a different mode on a timetable, usually to close some overnight or make them pay-at-pump only.

Doing that meant reading how scheduled modes actually get applied. So when a colleague raised a tech-debt ticket saying the pump list was allocated more than once at start-up with different results, I already knew which parts of the start-up sequence cared about pumps.

That's a pattern I've noticed more than once. The small ticket teaches you the map, and the map is what makes the next ticket make sense. It's how the twin timer in Part 5 turned up too.

Clear, then refill, then hope

The method that builds the pump list has three callers. One runs early in start-up. One runs when the connection to the payment configuration data comes up, which is where the list of pump terminal ids lives. One runs during a worker restart.

The early one runs before that connection exists. So it asks for pump ids and gets nothing back. Here's the shape of what it did next, as generic C#:

public void AllocatePumps()
{
    var ids = _source.FetchPumpIds();   // empty if the source isn't connected yet
    _pumps.Clear();                     // wipes everything, whatever came back
    foreach (var id in ids)
        _pumps[id] = new Pump(id);
}

It clears the collection unconditionally and then loops over an empty result. And the pump collection is a singleton in the dependency container, so every worker in the service sees the empty dictionary immediately.

The weak method, in one sentence: clear the state first and trust the replacement to arrive. It's fine when the replacement is local and instant. It's a trap when the replacement depends on something that might not be ready yet.

The singleton makes this much worse than it looks. If each worker built its own copy of the pump list, one early, empty copy would only hurt one worker. Because they all share one collection, a single early clear reaches every worker in the service at once, and none of them has any way to know the list it's reading is only temporarily empty.

What every reader saw in the window

The useful part of the analysis wasn't the bug.

The method was plain. I read the start-up method top to bottom and wrote a numbered list: each step, what it starts, and whether pumps exist yet at that point. Then I found every consumer of the pump collection and marked where it first runs in that list. Anything marked before the data source connects is a reader of the empty window.

That list turned out to be the most useful thing I produced. It was the list of everyone affected by it. I walked the start-up sequence step by step and wrote down every consumer that reads the pump collection, in the order they run.

The admin page's pumps endpoint showed nothing. A worker that flags pumps for kiosk payment found none to flag. The worker that handles mode changes skipped them. And the step that applies the site's scheduled mode ran in a background task against an empty collection, so it applied nothing.

One consumer worried me more than the rest. It treats a count of zero as a signal to create placeholder entries of its own. An empty window that only lasts a few moments can still leave something behind if a reader acts on it.

Ask what every reader sees during start-up, not just after it. Most code is reasoned about in its steady state, and start-up is the one time the steady state doesn't hold.

The recovery that didn't quite recover

The service does recover on its own. Once the data source connects, a second allocation fills the list properly. On a normal day restart, nobody would notice anything.

But that recovery path only set each pump's default mode, not its scheduled mode. So if the service restarted overnight, pumps that should have been closed or pay-at-pump only could come back in their daytime mode and stay that way until the next scheduled change.

It also explains why nobody had seen it. During the day, the default mode and the scheduled mode are usually the same thing, so a restart looks perfect. The gap only opens if the service restarts inside a scheduled window, and restarts are rare enough that a timing coincidence like that can hide for a long time.

That's the real customer impact, and it's operational, not cosmetic. A site that closes pumps overnight does it for a reason, and a restart shouldn't quietly open them again.

The failure that looked like "no pumps"

There was a second, smaller problem upstream, and it's worth showing because the pattern is everywhere.

If the fetch from the data source failed, it returned null. The next line called a LINQ method on that null, which threw. The exception was caught, logged once, and turned into an empty list:

try
{
    var rows = _db.FetchTerminalRows();          // null when the fetch fails
    return rows.Any() ? Map(rows) : new List<PumpId>();
}
catch (Exception ex)
{
    _log.Error(ex, "Exception fetching pump ids");
    return new List<PumpId>();                   // failure now looks like "no pumps"
}

So "the data source is broken" and "this site has no pumps" came back as exactly the same value.

The clue had been in the existing start-up logs all along, as that one error line. It looked like ordinary start-up noise, so nobody had read it as the cause of anything. Every caller had to treat them the same way, because it had no way to tell them apart. The honest version returns something that distinguishes them, a result type or at least a null the caller must handle, so failure stays a failure.

Four ways to fix it

I wrote down four approaches before choosing one, because the choice had real trade-offs.

Defer the first call until the data source is ready. It's the simplest, but it risks breaking code further down start-up that assumes pumps exist. Guard the clear so it doesn't wipe the list when nothing comes back. It's low risk, but it could hide a genuine "this site has no pumps" case.

Allocate only when the data source reports it's connected, or split start-up into two explicit phases. Both are more robust. Both are much bigger refactors of a start-up sequence a lot of code depends on.

I chose the guard because it was the smallest change that closed the window without reordering start-up for every worker. Deferring the first call looked simpler, but I couldn't prove nothing later in start-up assumed pumps existed, and a start-up sequence is a bad place to find out.

I shipped the guard, together with the missing step on recovery:

public bool AllocatePumps(string caller)
{
    var ids = _source.FetchPumpIds();
    _log.Info("{Caller} allocated pumps: {Count} found", caller, ids.Count);
 
    if (ids.Count == 0)
        return false;                               // keep what we have
 
    _pumps.Clear();
    foreach (var id in ids)
        _pumps[id] = new Pump(id);
 
    _modes.ApplyScheduledMode(_pumps.Values);       // not just the default mode
    return true;
}

Here's how the four options compared when I wrote them down:

| Option | Risk | Size | |---|---|---| | Defer the first call | Breaks code that assumes pumps exist | Small | | Guard the clear | Could hide a genuine "no pumps" case | Small | | Allocate only on connect | Changes start-up order for every worker | Large | | Two-phase start-up | Robust, but touches most of start-up | Largest |

The collection is no longer wiped when nothing comes back, and the first successful allocation now applies the scheduled mode, not just the default. Those two changes close the empty window and the overnight gap together. The call-chain logging means the logs now say which caller allocated and how many pumps it found, so the next person debugging start-up doesn't have to reconstruct the order from scratch.

If I were writing it fresh, I'd go one step further and build the new collection separately, then swap it in with a single reference assignment. Clear-then-fill leaves a gap even on the happy path, however brief, for any reader on another thread. Swapping a fully built collection means readers see the old list or the new one, never half of either.

Where it falls over

The guard trades one risk for another. If a site genuinely comes back with no pumps, the service now keeps the old list rather than clearing it. That's the right default at start-up, where "nothing came back" almost always means "not connected yet". But it's a case that needs to be loud in the logs, not silent, and I'd want an alert on it rather than just a log line.

I found this by reading code, not from a site. That's good, because nobody lost an overnight closure to it that I know of. It also means I haven't seen the fix tested against a real overnight restart on a live site.

And a separate overnight investigation showed the same shape in a different place. Pumps stayed in their overnight mode after the window had ended, because the mode was set once from the clock and nothing re-checked it when the window ended. Both problems point the same way: a scheduled mode needs a timer that re-evaluates it, not a one-off call at start-up and a hope that nothing restarts.

Keep the old state until the new state arrives

My position: never clear state before you know you have its replacement. It's the kind of code that works every day in the steady state and fails in the one window where the steady state doesn't hold, and start-up is exactly that window.

The same shape turns up in plenty of .NET services: a cache warmed from a database that isn't reachable yet, feature flags loaded from remote config, a hosted service that starts before the thing it depends on. Each one works after start-up and misbehaves during it.

Pick one collection your service builds at start-up and trace every reader of it through the start-up sequence, in order. If any of them can see it empty, or half-filled, guard the clear or swap in a fully built replacement. And make sure your fetches can tell "failed" apart from "empty", because a caller can't handle a difference it can't see.

Next in the series: a payment-host simulator I designed to be polite, and real captures that showed real hosts aren't. The rest of my writing is on the blog.