Idempotency: "Timeout Is Not Failure — It's an Unknown Outcome"

Every payment, payout, or approval endpoint that an external system calls needs to answer one question correctly: what happens when the caller retries the exact same request? Networks deliver "at least once" in practice — a caller's timeout does not mean your server failed to process the request. It may have succeeded after the caller gave up, and the caller will retry, because it has to.

// external caller times out and retries -> processed twice
@PostMapping("/pay/auto")
public Response autoPay(@RequestBody AutoPayRequest req) {
    paymentService.process(req); // no duplicate-transaction check
    return Response.ok();
}

Why the exists() check alone isn't enough

The intuitive fix is to check whether the external transaction ID has already been processed before doing the work. That helps the common case, but it does not close the actual race: two identical requests can arrive at nearly the same moment. Both run existsByExternalTxId(), both see "not found", both insert — duplicate payout anyway. This is a TOCTOU race: time-of-check to time-of-use.

The real fix: a UNIQUE constraint

@PostMapping("/pay/auto")
public Response autoPay(@RequestBody AutoPayRequest req) {
    // 1. fast path: check the external transaction id first
    if (paymentService.existsByExternalTxId(req.getExternalTxId())) {
        return Response.duplicate(req.getExternalTxId()); // return the EXISTING result
    }
    // 2. UNIQUE constraint + INSERT closes the race window
    paymentService.processWithIdempotency(req);
    return Response.ok();
}

The UNIQUE index on the external transaction ID column is the guarantee, not the exists() check — the second INSERT fails at the database level regardless of timing. The application catches that unique-violation and returns the existing result instead of erroring, so a retried request is idempotent from the caller's point of view.

The second half of the same incident: unbounded queues

A related failure mode from the same incident class: async jobs (Kafka consumers, @Async work, notification retries) that accumulate failed items with no retry limit, no TTL, and no cleanup batch. Each failed item is tiny — day one, fifty stuck rows, nobody notices. By day ninety it's two million rows, the polling query that re-reads them slows down, memory for the batch grows, and the system dies. Often the system that dies is not the one that caused the failures.

• Cap retries with a retry_count column and move exhausted items to a dead-letter queue.
• Move items older than N days to a separate table so the live queue stays small.
• Alarm on an accumulation threshold (e.g. pending-send count > N) — don't wait to notice it manually.