Idempotent API Design: Keys, Storage and Retention for Payment-Style Calls

To make a payment-style call safe to retry, require the client to send a unique Idempotency-Key header, store the key together with a fingerprint of the request and the result of the first attempt, and replay that stored result whenever the same key and the same request arrive again. A key reused with a different request is an error, a duplicate that arrives while the first is still running gets a conflict, and keys expire after a documented retention period.

Who this is for

This is for API designers and back-end developers who expose operations with side effects, such as creating a charge, placing an order or sending a message, to clients that retry on timeouts. By the end you will have a header contract, a PostgreSQL table, a request flow that handles concurrency and a retention policy.

Why retries are dangerous without a key

HTTP's safe retries are limited to idempotent methods. GET, PUT and DELETE are idempotent by definition: repeating them leaves the server in the same state. POST and PATCH are not. If a client sends POST /payments and the connection drops before it sees a response, it cannot tell whether the charge happened. Retrying risks charging twice; not retrying risks never charging.

Stripe's API reference describes the remedy: use an idempotency key when creating or updating an object, so that after a connection error you can repeat the request without risk of creating a second object or performing the update twice. The IETF's httpapi working group has an active draft, draft-ietf-httpapi-idempotency-key-header, that standardises the header. It states that the aim is to let clients safely retry non-idempotent methods such as POST and PATCH, so a client that times out can tell whether the server already acted. Because it is still a draft, treat the details as current direction rather than a final standard.

The header contract

Define the contract in your API documentation before writing code.

• Header name: Idempotency-Key. In the IETF draft it is an Item Structured Field whose value must be a string.
• Who generates it: the client. The draft says the resource owner defines what makes a key unique, that clients should generate it, and that UUIDs are recommended. Stripe suggests V4 UUIDs or another random string with enough entropy, allows keys up to 255 characters and advises against sensitive data such as email addresses in keys.
• Which operations require it: all non-idempotent writes that move money or create resources. If a required key is missing, the draft says to answer 400 Bad Request, ideally with a link to documentation, optionally using RFC 9457 problem details.
• Scope: keys should be unique per client. The draft warns that low-entropy keys could let attackers guess other clients' keys, and suggests a composite lookup key that includes client-specific data. Always scope the stored key by authenticated client or tenant.

A request looks like this:

POST /v1/payments HTTP/1.1
Host: api.contoso.com
Authorization: Bearer <token>
Idempotency-Key: 8e03978e-40d5-43e8-bc93-6894a57f9324
Content-Type: application/json
 
{"amount": 4200, "currency": "usd", "customer": "cus_123"}

Server behaviour

The draft describes the behaviours a server should implement.

Stripe's v1 behaviour is a useful reference. It saves the status code and body of the first request made with a key once endpoint execution begins, and retries with the same key return the saved response, including 500 errors. It compares incoming parameters with those of the original request and returns an error if they differ. It does not save a result if the parameters fail validation or the request conflicts with a concurrently executing one, because no endpoint execution started; those requests can be retried.

You must decide how your API treats failures. Replaying a stored 500 is consistent, but it can lock a client into a transient error. A common refinement is to store the result only for outcomes that are final (success or a definitive business error) and let the client retry after server errors with the same key, provided your processing is itself safe to repeat. Whichever you choose, document it.

Step 1: Create the key table

Store keys in the same database as the business data so that recording the key and performing the operation can share a transaction. A PostgreSQL design:

CREATE TABLE idempotency_keys (
    client_id        text        NOT NULL,
    idempotency_key  text        NOT NULL,
    request_method   text        NOT NULL,
    request_path     text        NOT NULL,
    request_hash     bytea       NOT NULL,   -- fingerprint of the request body
    status           text        NOT NULL,   -- 'started' | 'completed'
    response_status  integer,
    response_body    jsonb,
    created_at       timestamptz NOT NULL DEFAULT now(),
    locked_at        timestamptz,
    PRIMARY KEY (client_id, idempotency_key)
);
 
CREATE INDEX idempotency_keys_created_idx ON idempotency_keys (created_at);

The composite primary key does three jobs: it scopes keys to a client, it makes duplicate inserts fail, and it provides the lookup. The request_hash is the fingerprint the IETF draft mentions: a server may derive a checksum or signature from the request payload and use it with the key. Hash the method, path and canonicalised body, for example with SHA-256.

Step 2: Claim the key before doing the work

The critical move is to insert the row first, atomically, so that two concurrent requests with the same key cannot both proceed.

INSERT INTO idempotency_keys
    (client_id, idempotency_key, request_method, request_path, request_hash, status, locked_at)
VALUES
    (@clientId, @key, 'POST', '/v1/payments', @hash, 'started', now())
ON CONFLICT (client_id, idempotency_key) DO NOTHING
RETURNING 1;

• If the statement returns a row, you own the key. Run the operation.
• If it returns nothing, a row already exists. Read it and branch:

SELECT request_hash, status, response_status, response_body
FROM idempotency_keys
WHERE client_id = @clientId AND idempotency_key = @key;

Then apply the rules:

1. request_hash differs from the new request's hash: return 422.
2. status = 'completed': return the stored response_status and response_body.
3. status = 'started': return 409, optionally with a Retry-After header.

Step 3: Execute and record the result atomically

Do the business write and mark the key completed in the same transaction wherever you can, so a crash cannot leave a completed payment with a key stuck in started.

BEGIN;
 
INSERT INTO payments (payment_id, customer_id, amount, currency)
VALUES (@paymentId, @customerId, @amount, @currency);
 
UPDATE idempotency_keys
SET status = 'completed',
    response_status = 201,
    response_body = @responseJson
WHERE client_id = @clientId AND idempotency_key = @key;
 
COMMIT;

When the operation includes a call to an external system such as a card processor, the single-transaction approach no longer covers everything. Pass your own idempotency key, or a value derived from it, to the downstream call, so a retry of your handler does not charge twice downstream. Microsoft's write-through caching architecture follows the same idea for command-style writes: it validates the request, checks the idempotency key, commits to the database and records follow-up work in an outbox, and it recommends that both the write function and the repair function be idempotent. See Transactional Outbox Pattern for the follow-up-work technique.

Step 4: Recover stuck keys

A server can crash after claiming a key and before finishing. Without a recovery rule the client receives 409 until the row is removed. Use locked_at to detect it: if a started row is older than your maximum processing time, a new request may take it over after checking whether the work already happened, for example by looking for a payment row that references the key. Choose a lock timeout comfortably longer than your slowest legitimate request and test the takeover path explicitly.

Step 5: Set and publish a retention policy

Keys cannot live forever, and the table would grow without bound. The draft says servers may purge keys on a schedule and should publish that policy. Stripe's v1 API documents that keys can be removed automatically once they are at least 24 hours old, after which reusing a key creates a new request.

Choose retention from the longest realistic client retry window, including queued or batch clients that retry hours later, and then document it. A purge job is simple:

DELETE FROM idempotency_keys
WHERE created_at < now() - interval '24 hours';

Delete in batches on large tables, and keep the retention at least as long as the longest retry window you promise clients. If the unique key in your payment table also guards against duplicates, you retain a safety net after the key row has been purged.

Client guidance to publish

• Generate a new key per logical operation, not per attempt. Reuse the same key on every retry of that operation.
• Never reuse a key for a different request.
• Retry with backoff on timeouts, connection errors and 5xx responses, and stop after a bounded number of attempts.
• Treat 409 as "the first attempt is still running": wait and retry with the same key.
• Treat 422 as a client bug: the key was reused with a different body.

Troubleshooting

Clients get 409 for a long time. A key is stuck in started. Check locked_at, the lock timeout and whether the request crashed mid-flight.

Duplicate charges still happen. The downstream call is not idempotent, or the key is generated per attempt on the client. Pass a derived idempotency key downstream, and verify the client reuses one key across retries.

422 errors after a client release. The client now sends a different body with an old key, for example because a timestamp field changes per attempt. Remove volatile fields from the fingerprint or from the request.

The table grows quickly. Check that the purge job runs, and that retention matches the policy you published.

Two tenants collide. The key is not scoped by client. Keep client_id in the primary key.

Checklist

• Idempotency-Key required on non-idempotent writes, with 400 when missing.
• Keys scoped to the authenticated client.
• Unique constraint on the key; insert before work.
• Request fingerprint stored and compared, 422 on mismatch.
• 409 for a duplicate that arrives while the first runs.
• Result stored in the same transaction as the business write.
• Downstream calls carry their own idempotency key.
• Retention documented and enforced by a purge job.

To see the same problem from the receiving side of webhooks, read Webhooks that don't double-charge.

References

• Stripe API reference, Idempotent requests: https://docs.stripe.com/api/idempotent_requests
• IETF, The Idempotency-Key HTTP Header Field (Internet-Draft): https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/
• Microsoft Learn, Write-through caching with Azure Managed Redis and Azure SQL Database: https://learn.microsoft.com/en-us/azure/architecture/databases/architecture/write-through-caching-azure-sql-managed-redis
• PostgreSQL documentation, INSERT (ON CONFLICT): https://www.postgresql.org/docs/current/sql-insert.html