Messages That Survive
7 parts, in reading order.
Messages That Survive, Part 1: Kafka Was Down and Nobody Noticed
If our message broker goes down, no user notices. That isn't luck. It's one design decision: write the change and the message in the same database transaction, and publish later. This is the outbox pattern, what it costs, and where it still falls short.
Messages That Survive, Part 2: Every Message Arrives Twice
Kafka promises your message will arrive. It doesn't promise it'll arrive once. We made the second delivery harmless with an inbox table keyed by message ID. The post covers why the order of two lines matters more than it looks, and the kind of duplicate an inbox can't catch.
Messages That Survive, Part 3: Retry, Back Off, Then Stop
Some messages will fail every time you process them. Retry them forever and one bad message becomes a permanent load. The retry policy we settled on, the exception that lets a handler say 'never', the watchdog that restarts a hung worker, and the part we still don't handle well.
Messages That Survive, Part 4: Version Your Events, Don't Edit Them
A producer can change its payload in an afternoon. Its consumers can't all change that fast. We treat a changed event as a new version that lives next to the old one. This covers how long both stay, and what happens when nobody remembers to retire the old one.
Messages That Survive, Part 5: The Dead-Letter Topic We Didn't Have
Marking a message 'terminated' in a table stops it hurting anything. It doesn't get it processed. What happens to a failed message in our setup today, why that isn't a recovery plan, and what a dead-letter topic, a replay tool and a runbook would change.
Messages That Survive, Part 6: The Consumer That Waited Forever for a Topic
Services don't start in the order you drew them on the whiteboard. A consumer that subscribes to a topic before any producer has created it can sit there doing nothing, and look perfectly healthy while it does. How that happens, and how to make it stop.
Messages That Survive, Part 7: The Back-Off Setting Nobody Used
A retry schedule was defined, documented in code, and never read. The publisher used a different one. This is a pattern worth looking for in your own configuration: the setting that exists, looks right, and changes nothing.