Case study · 2026
Transaction Ledger Service
An event-driven ledger that never loses or double-applies a transaction, even when it crashes mid-write.
The problem
A ledger has to apply transactions in order, even when they arrive out of order and the service crashes mid-write. This one accepts writes over REST, returns 202 instantly, and applies balances asynchronously through Kafka. Nothing gets lost, nothing gets applied twice.
Crash the consumer mid-write
Browser modelKafka guarantees at-least-once delivery, so a crash means the message comes back. The ledger guarantees each transaction lands exactly once. Arm a crash and watch the redelivered message get skipped instead of double-applied.
This is a simplified model running in your browser, not the live service. It reproduces the behaviour asserted by redeliveredEvent_isAppliedExactlyOnce, a real Testcontainers test that republishes the same event twice against Postgres and Kafka and asserts the balance never moves.
Waiting for transactions...
Invariant
Balances match the ledger: $0
Sum of balances always equals the sum of writes that reached the ledger, no matter how many times a message is redelivered.
Decisions, and what they cost
Every architecture is a set of trade-offs. These are the ones I made, the alternatives I rejected, and why.
Partition Kafka by account ID
vs. one global ordering keyPer-account ordering is all a ledger needs, and it lets consumers scale out. The cost: no global order, and a hot account can skew a partition.
Manual offset acknowledgement
vs. auto-commitOffsets commit only after the balance is durably applied, so a crash means redelivery, not loss. The consumer is idempotent, and the tests prove exactly-once effects.
Backoff retries + dead-letter topic with replay
vs. infinite retry or drop-on-failurePoison messages can't block a partition, and nothing silently disappears. Failures land in the DLT and replay after a fix. The cost is one more thing to monitor.
Integer money + optimistic locking
vs. floats and last-write-winsCents-as-integers kills float rounding. Optimistic locking turns concurrent balance updates into a retry instead of a silent overwrite.
Evidence it works
31 tests, including Testcontainers integration tests on real Postgres and Kafka, proving ordering under redelivery, exactly-once application, and DLT recovery.
Correlation IDs trace every transaction end to end: HTTP thread → Kafka headers → consumer.
Grafana dashboard tracking throughput, p99 latency, rejection rate, and consumer lag.
Live SSE dashboard with a failure simulator, DLT replay, and a 100-transaction burst generator.
What I'd do differently
I'd add the Kubernetes deployment earlier. The interesting question a ledger has to answer is what breaks during a rolling restart, and how the consumer group rebalance is handled. That's the difference between running a system and operating it. In progress now.