kunalships.dev
Tutorial 2026-08-18 · 15 min read · #backend-engineering#aws#django#postgresql#distributed-systems

Idempotency: The Claim-First Pattern That Stops Double Charges

Retries are guaranteed in distributed systems. Duplicate side effects are not. A deep dive into idempotency keys, the claim-first ledger pattern, and how to implement it in Django and PostgreSQL.

Your Lambda ran twice. Your customer got charged twice. Nobody wrote a bug, and yet here you are, refunding a payment at 2 AM and wondering what happened.

What happened is that you built on infrastructure that retries, without building the thing that makes retries safe. This article is about that thing: idempotency, the key that identifies an operation, and the claim-first ledger pattern that ties them together. By the end you’ll have a Django implementation you can lift into production, and more importantly, the reasoning to defend it in a design review.

What idempotency actually means

An operation is idempotent when performing it once or performing it five times produces the same result. The word comes from mathematics: f(f(x)) = f(x). Absolute value is idempotent. abs(abs(-4)) is still 4, no matter how many times you apply it.

Some HTTP semantics come with this property built in. GET doesn’t change anything, so repeating it is trivially safe. PUT /users/42 {name: "Kunal"} sets a value, and setting it again lands in the same state. DELETE /orders/8 deletes the order; the second delete finds nothing and that’s fine.

Then there are the operations that matter: POST /orders, charge_card(), send_email(). Run twice, they do the thing twice. These aren’t naturally idempotent and no amount of careful coding inside the function changes that. The function can’t tell a retry from a fresh request, because from the inside they look identical.

So we don’t fix the function. We wrap it in a memory layer that can tell the difference. That layer is the subject of the rest of this article.

Retries are not an edge case

The instinct to resist here is “we’ll just make sure nothing retries.” You can’t, because retries are a feature of every layer you build on, and mostly not one you can turn off:

Where duplicates come from: clients, Lambda's async retries, and SQS redelivery all converge on your handler

The conclusion worth internalizing: the network will replay your requests. This isn’t a failure mode to prevent, it’s a contract to design for. The goal is never “no retries.” The goal is “retries are harmless.”

The key: naming the operation, not the attempt

To make a retry harmless, the system needs to recognize that two physical requests are the same logical operation. That recognition token is the idempotency key, and everything about the pattern follows from one sentence:

Every retry of the same logical operation must carry the same key, and different operations must carry different keys.

This sounds obvious and is violated constantly, almost always the same way: by generating the key in the wrong place.

A key minted at intent time survives every retry; a key minted inside the handler is fresh per attempt and defends against nothing

A key generated inside the handler, a uuid4() at the top of the function, a timestamp, a request ID, is born fresh on every attempt. Each retry mints a new key, looks like a new operation, and sails past your protection. The format was never the problem; a UUID is a perfectly good key. The birthplace is the problem. The key must be minted upstream of the operation it protects, at the moment the intent is formed, and then carried unchanged through every attempt.

Upstream sources, in rough order of preference:

  1. A natural business identity. If the operation already has an ID in your domain, use it plus the action: order_8231:capture_payment, invoice_442:send_reminder. Free, deterministic, and readable in your logs at 2 AM. The action suffix matters: the same order can legitimately be captured and refunded, and those are different operations that must not share a key.
  2. An intent-time identifier. For creation operations, the resource’s ID doesn’t exist yet, so reach one level upstream: the cart ID, a checkout-session ID your backend minted when the checkout screen loaded, or a UUID the frontend generated on first render and holds through retries. This is Stripe’s Idempotency-Key model, and we’ll come back to why creation specifically forces it.
  3. Infrastructure identity. Consuming from a queue? The transport hands you a key: the SQS messageId, the EventBridge event id, a webhook’s event.id. One caveat: messageId protects against SQS redelivering a message, not against a producer publishing the same logical event twice. If double-publish is a risk, prefer a business ID from the message body over the envelope’s ID.
  4. A content hash, as a last resort: sha256(user_id + endpoint + canonical_body). It works until the payload contains anything that varies per attempt, at which point the hash changes and the protection silently evaporates. It also can’t distinguish “retry” from “the user deliberately submitted the identical request twice,” which for add ₹500 to wallet is a real ambiguity. Use it when nothing better exists, and document why.

The ledger

The key names the operation. Something still has to remember whether that operation happened. That memory is a small, dedicated table I’ll call the ledger:

class IdempotencyKey(models.Model):
    key = models.CharField(max_length=255, unique=True)
    status = models.CharField(
        max_length=16,
        choices=[("processing", "processing"),
                 ("completed", "completed"),
                 ("failed", "failed")],
    )
    result = models.JSONField(null=True)      # the response to replay
    created_at = models.DateTimeField(auto_now_add=True)  # doubles as the lease timestamp

Two design decisions are hiding in these six lines, and both come up in reviews.

Why a separate table, not a status field on the business model? Because the key identifies an operation while your model represents a resource, and they aren’t the same thing. The clearest proof is creation: when you need to claim checkout ck_8231, the Order row doesn’t exist yet, so there’s nowhere to put a status. Beyond that, one order has many protected operations across its life (create, capture, refund), each needing its own claim, and your Order.status field is already busy tracking a completely different state machine (pending → paid → shipped). Tangling the two is how you end up with a status column nobody can explain. One ledger table serves every idempotent operation in the system, and TTL cleanup becomes a single scheduled delete that never touches business data.

There is one legitimate shortcut: for pure creations whose entire side effect is a single insert, put the key on the business table itself as a unique column (checkout_session_id with unique=True on Order). Then the insert is the claim, one atomic write, no ledger. It’s elegant right up until the operation grows a second side effect, like calling Stripe, at which point you need the claim to exist before the external call, and you’re back to the ledger.

Why unique=True is the entire mechanism. The naive implementation checks whether the key exists, and if not, proceeds. That’s a read followed by a write, and in the gap between them, a concurrent worker runs the same check and also passes. Both process. The fix is to collapse check-and-claim into one atomic operation and let the database arbitrate:

from django.db import IntegrityError

try:
    IdempotencyKey.objects.create(key=idem_key, status="processing")
except IntegrityError:
    # someone else owns (or owned) this operation
    return handle_existing(idem_key)

Two workers race to insert the same key; the unique constraint guarantees exactly one wins. In raw PostgreSQL the same move is:

INSERT INTO idempotency_keys (key, status)
VALUES (%s, 'processing')
ON CONFLICT (key) DO NOTHING;
-- rowcount 1: you won the claim. rowcount 0: key already exists.

No advisory locks, no SELECT ... FOR UPDATE, no distributed lock service. A unique index is the cheapest mutual exclusion you will ever buy. (Redis buys the same guarantee with SET key processing NX EX 86400 if you want the expiry built in, at the cost of a second infrastructure dependency for a correctness-critical path.)

The flow, end to end

With a key and a ledger, the full consumer flow has exactly four branches:

The claim-first flow: conditional insert, then branch on won/lost, completed/processing, with failure releasing the claim

A message arrives carrying its key (the message ID, or better, a business ID from the payload).

  1. Conditional insert first. Try to insert the key with status processing.
  2. Insert succeeded → you own this operation. Do the work. On success, update the row to completed, store the result, ack the message.
  3. Insert failed → the key exists, so check its status. If completed, fetch the stored result and return it — this is the replay branch, and it’s what makes the pattern transparent to callers: the retry gets the same response the original would have gotten. If still processing, another consumer is mid-flight (or a previous attempt crashed), so don’t touch it. Requeue with a delay and let the next delivery re-check.
  4. Processing failed → release the claim: delete the row or set status to failed. This is easy to forget and important. A failed attempt that keeps its processing claim blocks every future retry, and you’ve converted “transient error” into “operation permanently stuck.” Release means the redelivered message’s conditional insert succeeds and the retry gets a clean shot.

The one rule: claim before, never after

Of everything in this article, this is the piece most often built backwards, so it deserves its own section. The intuitive design is: process the work, and once it succeeds, record the key as a receipt. Stamp after. It feels right because the ledger then only ever contains successes.

It has a hole exactly where you can least afford one:

Two timelines: stamp-after leaves a gap between the side effect and the record where a crash causes reprocessing; claim-first leaves a trace the crash cannot erase

In the stamp-after timeline, the card is charged, and then the process crashes — OOM kill, deploy, spot instance reclaim — before the key is recorded. The crash erases the only evidence that the work happened. The retry checks the ledger, finds nothing, and charges again. The window is milliseconds wide and production traffic will find it, because crashes correlate with load and load correlates with exactly the moments you’re processing the most payments.

Claim-first inverts the failure mode. Claim, then process, and a crash after the side effect leaves a processing row behind: a trace the crash cannot erase. The retry finds the claim and does not blindly reprocess. You’ve traded “silent duplicate charge” for “operation visibly stuck in processing,” and that trade is the whole point, because stuck is detectable and duplicate is not.

Which raises the honest follow-up: what un-sticks it?

The lease. A processing row older than some threshold — 5 minutes, or a bit above your P99 processing time — is treated as abandoned and may be re-claimed. That’s what created_at is really for: status = "processing" AND created_at < now() - lease is the re-claimable condition. Note what this implies: a lease expiry can, rarely, let the work run twice (original worker was slow, not dead). Claim-first shrinks the duplicate window from “any crash” to “crash-or-stall straddling a lease boundary,” which is a much smaller and much more visible window. If even that is unacceptable, the side effect itself must accept your key — which is exactly why Stripe takes an Idempotency-Key header: you extend your claim into their system.

The transactional escape hatch. If every side effect of the operation is a write to the same PostgreSQL database, you don’t need the lease at all. Wrap the claim and the business writes in one transaction:

from django.db import transaction

with transaction.atomic():
    IdempotencyKey.objects.create(key=idem_key, status="processing")  # first statement
    order = Order.objects.create(...)
    OrderLine.objects.bulk_create(...)
    # mark completed inside the same transaction
    IdempotencyKey.objects.filter(key=idem_key).update(
        status="completed", result={"order_id": order.id}
    )

A crash anywhere rolls back everything, claim included, so there’s no stuck state and no gap: the claim and the work commit atomically or not at all. This is the cleanest version of the pattern and worth restructuring toward. It stops working the moment a side effect crosses a system boundary — a rollback cannot un-call Stripe — so keep external calls outside the transaction (or defer them with transaction.on_commit), both for correctness and because holding a DB transaction open across a third-party network call is its own incident in waiting.

Case study: keying “place an order”

Order creation is worth walking through because it’s the case where the obvious answers fail, and working out why they fail is most of the education.

First instinct: user_91:create_order. Too coarse — the user’s second order, for anything, collides with the first.

Second instinct: add the product, user_91:pizza_42:create_order. Better, and still broken, just more slowly. Ask the diagnostic question: can a user legitimately perform this exact operation twice? For “capture payment on order 8231,” no — the natural key works. For “order a pizza,” obviously yes. I order one tonight and the same one tomorrow, and with this key my second genuine order is indistinguishable from a retry of the first. The idempotency layer eats it. You’ve traded duplicate orders for silently dropped orders, which is arguably worse: the customer believes they ordered, and nothing arrives.

TTL doesn’t rescue it either. Short TTL (minutes) and a delayed SQS redelivery an hour later sails through as a duplicate. Long TTL and tomorrow’s pizza is blocked. No TTL value separates “retry” from “reorder,” because the key itself cannot tell them apart.

The root cause: creation operations have no natural identity, because the order ID is born from the operation and can’t be the key for it. So the key must come from one level upstream, from the intent:

The distilled rule: include enough dimensions to distinguish different operations, but derive the key from the intent, never from the contents. Two orders with identical contents are still two different intents. Product IDs describe what’s being ordered; only a cart, session, or intent-time key identifies which act of ordering this is.

TTLs, scope, and housekeeping

Three small decisions that keep the ledger healthy:

Scope keys by tenant. user_91:ck_8231:place_order, not just the session ID. In a multi-tenant system, IDs from different customers must be structurally unable to collide, and prefixing the tenant is cheaper than auditing every ID generator for global uniqueness.

Match TTL to the retry horizon, not to instinct. The stored key must outlive the longest thing that can redeliver it. For API clients that’s minutes; for an SQS queue with default retention it can be days; a 24-hour TTL is a reasonable floor for most systems but is genuinely too short if a DLQ redrive next week is part of your incident playbook. Whatever you pick, enforce it: a nightly DELETE FROM idempotency_keys WHERE created_at < now() - interval '...', a pg_cron job, or Redis’s built-in EX doing it for free.

Protect the side effects, not just the handler. If your function writes a row and calls an external API, the ledger protects the first and only the vendor’s own idempotency support protects the second. Pass your key through — Stripe, and most payment and messaging APIs, accept one — so your claim extends across the boundary.

The checklist

For the next design review, or the next time an interviewer asks how you’d prevent double-processing:

  1. Assume every request runs at least twice. Retries are the contract, not the anomaly.
  2. Derive the key from the operation’s identity: business ID + action for mutations, cart/session/intent-time key for creations, message ID for queue consumption. Never mint it inside the handler.
  3. Claim before processing with one atomic conditional insert (unique=True + IntegrityError, or ON CONFLICT DO NOTHING). Never check-then-act.
  4. On success: completed + stored result, so replays return the original response.
  5. On failure: release the claim so retries aren’t blocked forever.
  6. Un-stick crashes with a lease on processing age — or eliminate the problem entirely by putting claim and side effects in one database transaction when they share a database.
  7. External calls go outside the transaction, carrying your key into the vendor’s idempotency layer.
  8. TTL ≥ your longest redelivery horizon, enforced by a scheduled cleanup.

None of this prevents retries. All of it makes retries boring — and boring is the highest compliment a distributed system can receive.