There was a bug where a failed pump authorisation left the terminal's cancel button dead. The customer's payment was approved, the pump refused to start, and they couldn't back out cleanly. To reproduce it, QA had to install a special build of our service where the pump authorisation was hard-coded to fail.
That bothered me more than the bug did. A special build proves the special build. QA were testing something that would never ship, and every retest meant another bespoke install. This post is about fault injection in production code: why I'd rather put a narrow, default-off failure switch in the shipping binary than keep a broken build on the side, and what that costs.
This is Part 5 of Engineering Habits. Part 4 was about a design caught before any code. This one is about how a failure path gets tested once the code exists.
Why a special build is the weak option
The special build had the pump authorisation hard-coded to fail. It reproduced the bug. It also differed from the real build in every way that matters for confidence.
It was built separately, installed separately and versioned separately. Any fix tested on it was tested on a binary that would never reach a site. And every time QA needed it, someone had to remember how it was made, build it again against the latest code, and hand it over.
Worst of all, it failed every authorisation. QA couldn't run a normal sale and a failing sale on the same pump in the same session, which is exactly how a real site behaves when one authorisation goes wrong among many that don't.
The better option: a switch in the real build
The ask was to fold that behaviour into normal configuration. I added two settings, both off by default: an on switch, and a chosen amount.
When the switch is on and the customer picks that exact amount at the pump, the service skips the real pump authorisation and goes down the same failure path it takes when the pump controller genuinely says no. Everything downstream looks exactly like a real failure, which was the whole point.
Here's the shape of it, with neutral names:
private bool AuthorisePump(Pump pump, Sale sale)
{
if (FailAuthorisationEnabled.GetValue()
&& sale.ChosenAmountPence == FailAuthorisationAmountPence.GetValue())
{
LogDeferred(LogLevel.Warn, "TestHook",
() => new[] { $"Failing pump authorisation on purpose for pump {pump.Id}" });
return false; // the same failure path as a real refusal from the controller
}
return _controller.AuthoriseWithLimit(pump.Id, sale.LimitPence);
}The injection sits just before the real call, and reuses the failure handling that was already there. There's no new failure plumbing, which means QA test the real failure plumbing, not a copy of it.
Test your failure paths on the binary customers actually run. That's the habit, and a small switch is usually all it takes.
The details that made it usable
A few small decisions made this something QA could rely on rather than just tolerate.
The trigger is one amount, not "fail everything". QA can run normal sales on the same pump and only fail the one they choose. It also makes the trigger obvious in the logs: that amount, that sale, and a warning line saying it fired on purpose.
I compared against the amount the customer chose, not the adjusted figure the service actually sends to the pump. QA need to predict the trigger from the front of the pump, without knowing anything about our internals.
The amount is in whole pence, the same unit as everything else in that part of the code, so the check is a plain equality. The ticket first gave it in pounds with a decimal point, and I'd started designing around parsing text into pence before the ticket was corrected. I told that part of the story in the Spec-Driven series, and it's why I now ask about units before I design.
The settings appear in the admin UI automatically, because the UI builds its settings screens by reflection over the config classes. There was no UI work and no migration. It merged as one small change to the pump worker, plus tests.
I made one deliberate non-decision. If someone switched it on and set the amount to zero, a zero-value request would fail. I didn't add a special case. It's a QA toggle that ships off, someone has to turn it on and choose the amount, and a special case for a situation that doesn't arise in normal use is just more code to read.
What a good failure switch has
Looking back, the switch works because of a few properties I'd want in any fault-injection hook:
- Off by default, so shipping it changes nothing until someone acts
- A narrow trigger, so it fails one chosen thing and nothing else
- The real failure path, so what's tested is what runs on a bad day
- A loud log line, so anyone reading the logs knows the failure was deliberate
- No UI or deployment work, so QA can use it the same day it ships
Leave any of those out and the switch becomes either dangerous or useless.
When the obvious test home is broken
The tests were the awkward part, and they taught me a habit of their own.
The obvious home was an integration test class that already covered this handler. It was red on the main branch: its constructor call was missing an argument that had since become required, so every test threw before asserting anything. I proved that by stashing my change and running the class on a clean tree.
Then I checked whether a one-line fix would rescue it. It didn't. With the argument supplied, nearly all the tests still failed, because the fixture never wired up a pump controller at all. That's a fixture rewrite, not a quick repair, and it had nothing to do with this ticket.
So I put my tests in the maintained unit fixture instead, and wrote up the broken class in my notes. They cover the three cases that matter: matching amount while enabled fails and never calls the real authorise method; disabled proceeds normally; a different amount proceeds normally.
Prove a failure isn't yours, then decide whether fixing it is your job. Sometimes it is. This time it wasn't, and the right move was to put working tests somewhere healthy and leave a clear note.
Where it falls over
This is a switch in production code that makes real sales fail. It ships off, it fires on one amount, and it logs when it does. But anyone with admin access to config on a live site could turn it on, and a customer who happened to choose that amount would get a failed authorisation. I'd be happier if it were limited to test builds, or to an engineer-only role. That's the honest cost of testing on the real build, and it's worth saying out loud.
The unit tests set a private field by reflection, because the handler dereferences a controller that's only wired up when the worker starts, and starting it drags in most of the configuration and connection logic. That's a smell. It tells me the class is hard to construct, and I worked around it rather than fixing it.
And I left the broken integration class broken. It's written up, but it's still there for the next person, who'll hit the same wall I did.
Ship the switch, not the special build
My position: a failure path you can only test on a special build is a failure path you haven't really tested. A narrow, default-off switch in the shipping binary, with a trigger QA can choose and a log line that shouts when it fires, is usually the cheaper and more honest option.
Look for any special build, patched binary or local hack your team uses to reproduce a failure. Ask whether a config switch in the real build could do the same job. If it can, keep it off by default, make the trigger narrow, log it loudly, and decide up front who's allowed to turn it on.
Next in the series: an error message that kept being reported as a bug, and the investigation that showed it was telling the truth. The rest of my writing is on the blog.