Idempotency Keys in Payment APIs with Node and Spring: How We Avoid Double Charges on Client Retries

A client sends a POST /payments request to charge a card. It reaches our server, our server calls the payment provider, the provider processes the charge, and somewhere along that path the connection drops: a network timeout, a load balancer closing the socket, a mobile client losing signal. The client never sees the response. What almost any well-written client does next -a mobile app, an SDK, a frontend with retry logic- is resend the exact same request. If we did nothing to prevent it, that second request charges the card again.

This isn't an edge case. It's the default case for any distributed system running over a network that guarantees neither delivery nor that the caller will read the response. The question isn't whether it will happen, it's what we do when it does.

Why a charge isn't idempotent by nature

A GET can be retried without a second thought because it changes nothing. A charge is the opposite: every successful execution moves money. Retrying a charge with no additional safeguard isn't a careless implementation detail, it's simply wrong, because HTTP gives us no guarantee that a POST executes exactly once. Making sure a charge executes exactly once, even if the client sends it two or three times, is our responsibility, not the protocol's.

The standard fix is for the client to generate an idempotency key -typically a UUID- before the first attempt, send it in a header (Idempotency-Key), and reuse it unchanged on any retry of that same charge. The server uses that key to recognize a request as a repeat, not a new charge.

The idempotency table

The first thing we tried, and the thing that fell short, was a UNIQUE constraint on the idempotency key directly on the payments table. That works for the case where the first request already finished: the second insert collides with the constraint and we reject it. It fails for the case where the second request arrives while the first one is still in flight, mid-call to the provider. At that point there's no row yet, so both requests pass the check and both end up calling the provider.

What actually worked was a separate table, dedicated to idempotency, where the row is created before calling the provider, not after:

create table idempotency_keys (
  key text primary key,
  endpoint text not null,
  request_hash text not null,
  status text not null check (status in ('pending', 'completed', 'failed')),
  response_body jsonb,
  response_status int,
  created_at timestamptz not null default now(),
  completed_at timestamptz
);

The key alone isn't enough: we pair it with the endpoint and a hash of the request body (request_hash). This guards against a case that isn't malicious, just a client bug: an SDK that accidentally reuses the same idempotency key for two different payments with different amounts. If the key matches but the body hash doesn't, we respond with 409 instead of returning the cached response for a payment that isn't the one being requested.

The Node middleware

In the Node service (Express, with raw pg, no ORM) the flow is: try to insert the row in pending status with that key; if the insert collides with the unique constraint, read the existing row and decide what to respond based on its status.

async function idempotencyMiddleware(req, res, next) {
  const key = req.header('Idempotency-Key');
  if (!key) return res.status(400).json({ error: 'missing Idempotency-Key' });

  const requestHash = hashBody(req.body);

  const inserted = await db.query(
    `insert into idempotency_keys (key, endpoint, request_hash, status)
     values ($1, $2, $3, 'pending')
     on conflict (key) do nothing
     returning *`,
    [key, req.path, requestHash]
  );

  if (inserted.rows.length > 0) {
    req.idempotencyKey = key;
    return next();
  }

  const existing = await db.query(
    'select * from idempotency_keys where key = $1', [key]
  );
  const row = existing.rows[0];

  if (row.request_hash !== requestHash) {
    return res.status(409).json({ error: 'this key was already used with a different request' });
  }
  if (row.status === 'pending') {
    return res.status(409).json({ error: 'this payment is already being processed' });
  }
  return res.status(row.response_status).json(row.response_body);
}

The point that took us a while to get right the first time: a request that's in flight can't simply wait for the first one to finish and return its result, because that would require extra coordination (polling the row, or some notification mechanism) that complicates the hot path for no real gain. Returning 409 in that case and letting the client retry with backoff is simpler and, in practice, enough: the client already has retry logic, which is exactly what got us into this problem in the first place.

The same pattern in Spring

In a Java service built with Spring, the same mechanism is implemented as a filter (OncePerRequestFilter) that runs ahead of the controller, hitting the same idempotency table through JPA:

@Component
public class IdempotencyFilter extends OncePerRequestFilter {

    @Autowired
    private IdempotencyKeyRepository repository;

    @Override
    protected void doFilterInternal(HttpServletRequest request,
                                     HttpServletResponse response,
                                     FilterChain chain) throws IOException, ServletException {
        String key = request.getHeader("Idempotency-Key");
        if (key == null) {
            response.sendError(400, "missing Idempotency-Key");
            return;
        }

        String body = readBody(request);
        String requestHash = DigestUtils.sha256Hex(body);

        try {
            repository.insertPending(key, request.getRequestURI(), requestHash);
        } catch (DataIntegrityViolationException e) {
            IdempotencyKey existing = repository.findByKey(key).orElseThrow();
            if (!existing.getRequestHash().equals(requestHash)) {
                response.sendError(409, "this key was already used with a different request");
            } else if ("pending".equals(existing.getStatus())) {
                response.sendError(409, "this payment is already being processed");
            } else {
                writeResponse(response, existing.getResponseStatus(), existing.getResponseBody());
            }
            return;
        }

        chain.doFilter(request, response);
    }
}

The part we had to fix in Spring that never came up the same way in Node: insertPending needs to run in its own transaction, separate from the controller's transaction that processes the payment. If everything runs in the same transaction and the controller fails after calling the provider, the rollback also undoes the idempotency key insert, and the next retry won't find it: it will believe it's the first attempt all over again. Splitting that transaction out was the change that took us the longest to diagnose, because in manual tests with few simulated failures the problem almost never surfaced.

When the idempotency key isn't enough

There's one case the key doesn't solve, and it's the most uncomfortable of the three: our server calls the payment provider, the provider processes the charge successfully on its end, and before that response makes it back to our server so we can record it, our process dies -a deploy, an out-of-memory kill, a container crash. When the process comes back up, the idempotency row is stuck in pending forever, even though the charge already happened on the provider's side.

A client retry in that state, with the logic described above, returns 409 indefinitely, which prevents the double charge but leaves the client with no useful answer. The fix isn't in our table: it's that most payment providers accept their own idempotency key on the call we make to them, so that if our process recovers and decides to retry the outbound call with the same key, the provider recognizes it already processed that charge and returns the original result instead of charging again. That's why we generate a key of our own for the provider, derived from our internal idempotency key but not identical to it, and persist it alongside the pending row before making the outbound call. A reconciliation job runs periodically over rows that have been pending longer than expected, checks the real status with the provider using that key, and completes the row with the actual result.

What we check before signing off on a payment route

Before accepting a charge endpoint as idempotent, we confirm the idempotency row is created before the call to the provider, not after, and in a transaction that survives even if the rest of the request fails. We confirm the key is paired with a hash of the request body, never used alone. We confirm we generate and persist our own idempotency key toward the provider, because it's the one piece that lets us recover from a crash between a successful charge and our own confirmation of that success. And we accept that a concurrent request arriving while the first one is still in flight gets a 409 instead of waiting, because trying to solve that with extra coordination adds complexity that the client, with its own retry logic, already handles better than we could.