A phase-two pull request turned up carrying a commit from a completely different ticket. It had come in through a merge, nobody had meant it to be there, and it was sitting in a diff a reviewer was about to read as "this ticket's change". My first instinct was to add another line to the assistant's instructions. Then I noticed how many lines like that I'd already written, and I started building deterministic checks for AI-assisted development instead.
This post is about that rule: if a mistake happens twice, it becomes a script, not a reminder. I'll show the weak fix I kept reaching for, the three checks that replaced it, and how I decide what the model should judge and what a script should just count. The honest section at the end is mostly about where these checks don't run.
Part 5 was about keeping the assistant's context lean. This is the other half: the mistakes no amount of context will fix.
The weak fix: one more line in the instructions
Here's what I used to do when an AI-assisted change went wrong in a mechanical way. I'd open the assistant's instructions and add a sentence. "Before raising a PR, check that every commit belongs to this ticket." Done, lesson learnt, move on.
It works most of the time. That's the trap. A note in the instructions is advice, and advice gets weighed against everything else the model is reading. On a long session with a lot of context, "most of the time" becomes "usually", and nobody can tell you in advance which session will be the exception.
It also doesn't scale in the direction you want. Every reminder you add makes every other reminder a little lighter, which is exactly the dilution I wrote about last time. You end up with a list of rules that each work most of the time, and a combined failure rate nobody has measured.
A mechanical check shouldn't depend on anyone remembering, human or model. If two people running it would always get the same answer, it isn't a judgement call, and it doesn't belong in a prompt.
The better fix: judgement to the model, the grep to a script
The split I use now is simple. Anything that needs judgement stays with the model and the reviewer: is this log line a fault, is this design too clever, does this test actually test the behaviour. Anything mechanical becomes a small script that the assistant's workflow calls, and the model's job is just to run it and report what it says.
That keeps the model doing what it's good at. It's good at reading an unfamiliar method and explaining what it does. It's not reliably good at listing thirty commits and checking each subject line for a ticket number, especially an hour into a session.
Each check's instructions open with why it exists, usually the incident that created it. When someone asks why a check is there, the answer is in the file, not in my memory.
There's a second benefit that's easy to miss. When the model runs a script and reports its output, the report is checkable: I can see the command, the exit code and the lines it printed. When the model does the same check in its head, all I get is its summary. "I checked every commit and they're all fine" reads the same whether it checked thirty commits or skimmed three.
That changes how I review. A script's output is evidence I can rerun. A model's assurance is a claim I'd have to verify myself, which rather defeats the point of asking it to check.
Check one: every commit belongs to this ticket
The stray commit came first, so its check came first. Before any PR, the script lists every commit on the branch that isn't on the main development branch, and confirms each subject line names the branch's ticket.
The detail that matters is matching the ticket as a whole token. A shorter ticket number that happens to be the start of a longer one isn't a match. Here's the shape, made generic:
#!/usr/bin/env bash
# Every commit on this branch must name the branch's ticket as a whole token.
ticket="$1" # e.g. 4217, taken from the branch name
base="${2:-develop}"
strays=$(git log --format='%h %s' "$base"..HEAD \
| grep -vE "(^|[^0-9])${ticket}([^0-9]|$)")
if [ -z "$strays" ]; then
echo "Branch clean: every commit names ticket $ticket."
else
echo "Commits that don't name ticket $ticket:"
echo "$strays"
exit 1
fiThe script doesn't decide anything. It reports the branch clean or lists the strays, and then the workflow asks me whether each stray is expected: a merge commit, or a companion fix that genuinely belongs with this change. Only then does anyone cherry-pick anything out. The highest-risk case is a combined branch for two phases, because a merged-in branch can bring anything with it.
Check two: generated code that overwrites itself
This is the check I value most, because the mistake it catches is silent.
Some of our message models are generated from XML schemas through T4 templates. The template project produces a source file, and a post-build step copies it into the project that consumes it. In Visual Studio, the template runs first, so the copy is fresh.
From the command line, the template doesn't run, but the post-build copy still does. So it quietly overwrites the consuming file with the stale generated one, and any hand edit to that file is gone on the next build.
The tests can't see it. The round-trip test compares two objects that are both missing the same field, so they match, and the build stays green. That's the kind of failure I most want a script for: every signal you'd normally trust says everything's fine.
The check diffs every generated and copied pair and fails if they differ. It reads the pairs from the build targets rather than guessing them from file names, and it covers every template project, not just the one where we first got bitten. A new template project means reading its build target and adding the pair explicitly.
The real fix is still to edit the schema and regenerate. The check is the safety net for when someone doesn't, and "someone" includes an assistant that sees a generated class, spots the missing field, and helpfully edits the output instead of the source.
Check three: config that drifts from its copy
My assistant config lives in a folder git ignores, so I keep a tracked copy in a docs repo where colleagues can read and review it. The rule is that any change to the live config gets mirrored in the same session.
Two rule files drifted anyway, and nobody noticed until an unrelated audit happened to look. That's a textbook "happens twice" case: a rule everyone agrees with, broken quietly, found by accident.
Now a diff runs at the end of any session that touched the config. It normalises line endings and one documented difference in link paths, then compares the rules, settings, commands and every mirrored skill file against the live copy. The project's top-level brief is left out on purpose, because its paths differ by design, so it only gets flagged if its section headings differ.
The one that came from a rule, not an incident
A fourth check didn't come from a mistake at all. Our team writes arrange, act and assert comments in every test, and spotting a missing one by eye across a large test file is slow and dull.
So a short awk script prints one line per test that's missing a marker. It's honest about its limits. It checks the markers are there, not that the test is arranged well, and a test with three perfect comments can still be a bad test.
That's the pattern across all of them. The script owns the part with one right answer. The reviewer still owns the part that needs taste.
Where a workflow skill has a mechanical step, like validating a spec's structure or creating the linked document skeletons from the interview-first workflow, I've pushed that step into a script too. The model spends its effort on content, and the structure comes out the same every time.
How I decide what becomes a script
Three questions, in order.
Is it mechanical? If two careful people would always reach the same answer, yes. Commit subjects, file diffs and missing comments are mechanical. "Is this warning a fault" isn't.
Has it bitten us twice? Once is a note. The first time something goes wrong, I write it down where the assistant will read it. The second time, the note has proved it isn't enough.
Can the model reason its way past it? A note can be outweighed, misread or forgotten on a long session. A script's exit code can't. If the cost of the mistake is silent data loss, like the overwritten generated file, I don't wait for a second time.
The opposite mistake is just as easy to make. Imagine a script that flags any log warning appearing more than a set number of times. It would be deterministic and still wrong: the same answer every run, and the answer would be about volume, not about faults. Judgement stays with the model and the reviewer because it needs context a script can't have.
None of this replaces review. The assistant still stages and never commits, and every change still goes through a normal PR with human review and CI. The checks just mean the reviewer's attention goes on the design, not on spotting a commit that shouldn't be there.
Where it falls over
These checks only run if someone runs them. They live in the assistant's workflow as pre-PR steps, not in the pipeline. A developer who doesn't use the skills gets no protection at all, and neither do I on a day I skip the workflow. Some of them belong in CI, and the commit-scope check is the obvious first one to move. It isn't there yet.
I haven't counted what they've caught since, so I can't put a number on the value. I know what each one was built to stop. That's not the same as a measured catch rate, and I'm not going to present it as one.
The test-comment lint is deliberately shallow. It catches a missing comment, not a badly arranged test.
And scripts rot too. The generated-code check depends on someone adding the pair when a new template project appears. If they don't, the check passes, confidently, on a project it doesn't know exists.
Write the script on the second time
My position: a reminder in an assistant's instructions is the right response to a mistake the first time, and the wrong one the second. Keep judgement with the model and the reviewer. Turn the mechanical failures into small scripts that give the same answer every run, and make each one explain the incident that created it.
Start with the mistake that's already bitten you twice. For most teams using coding agents, it's commits that don't belong on the branch, and the check is a dozen lines of shell. Run it before every PR, then move it into CI once it's proved itself, and let the PR go through the same human review as everything else.
Next in the series, I'm going back to the Ralph TUI loop from Part 3 with the result of running a real feature through it, and why a green suite didn't mean the feature was right. The earlier parts are on the blog if you're catching up.