Last updated: September 9, 2026
By Kris Drouet, Engineering Executive, in partnership with KORE1
Idempotent API design means a client can safely resend the same request, by accident, timeout, or automatic retry, and the server guarantees it only happens once. That guarantee has to survive the network doing something ugly, which, statistically, it will.
The alert didn’t say duplicate charge. It said reconciliation mismatch, batch 4471, which is the kind of line that sounds boring right up until you find out what it means.
2:40 in the morning. Not my pager, thankfully. The on-call engineer’s. I got the callback an hour later because the mismatch was on a loan disbursement, not a five-dollar subscription renewal. Somewhere between our origination system and the payment rail, a request had gone out, timed out on our side before the response came back, and the client library did exactly what client libraries are built to do. It retried. The payment rail had actually processed the first attempt just fine. It just hadn’t told us in time.
Two disbursements. One borrower. Roughly $340,000 sitting in a suspense account for six days while three teams argued about whose bug it was.
Nobody’s code was technically wrong. That was the annoying part.
I’ve written before about what happens when a system built for one call at a time gets a decoupled, event-driven backbone bolted onto it. The point-to-point decoupling work and the Kafka rebuild that came after it both leaned on something I never actually spelled out on the page: once a caller is allowed to retry, whatever sits on the other end has to be safe to hit twice. Three times. Fifty times, if a client library gets stuck in a loop at 3 a.m. and nobody’s watching. This is that piece.
What “Idempotent” Actually Means for an API Call
An idempotent API call produces the same end state no matter how many times a client sends it. Run it once. Run it fifty times. The system lands in an identical place either way, because the outcome was never allowed to depend on the count. GET and DELETE get this for free under the HTTP spec. POST does not. Not by default. And POST is exactly where the money moves.
That’s the whole problem in one sentence. Every meaningful write in a payments or transaction system happens over the one HTTP verb that offers no built-in retry safety. So you build it in yourself, or you inherit whatever the vendor on the other end decided to build.
Retries Aren’t the Exception. They’re the Default.
Engineers new to distributed systems tend to treat a dropped connection like an edge case, something you handle with a try/catch and move on. It isn’t rare. It’s the operating condition.
There’s a survey worth knowing by name here: Peter Bailis and Kyle Kingsbury’s 2014 ACM Queue piece, “The Network Is Reliable.” It pulls failure reports straight from production systems at companies running serious infrastructure. The throughline is blunt. Partitions and dropped connections happen constantly in real deployments. Not occasionally in theory. Researchers at Microsoft studied their own data centers back in 2011. They found an average of 5.2 device failures and 40.8 link failures per day. Load balancers failed at something close to a one-in-five rate over the observation window, according to the Microsoft Research network failure study.That data is over a decade old and the numbers have shifted since, but the shape of the finding hasn’t. Failure is baseline behavior in any system large enough to matter.
Which means the client is going to retry. Your framework will retry by default. The mobile app will retry when it comes back online after a subway tunnel. A well-meaning ops engineer will manually resend a webhook because the dashboard said it failed. None of that is misuse. It’s the system working as intended.
The question was never whether retries happen. It’s what happens to your data the second time the same request lands.
The Idempotency-Key Pattern, Under the Hood
The pattern nearly every payments API converged on independently is the idempotency key. The client makes up a token, usually a UUID, and rides it along on the request as a header. That token is now the operation’s real identity. Not the HTTP request. The token.First time the server sees the key, it does the work and stores the result against that key. That part is simple. Every subsequent request carrying the same key gets the stored result handed back, untouched, no matter how many times it shows up.
Stripe’s own engineering writeup on this is worth reading directly, because Stripe is largely why the pattern looks the way it does across the industry. Their API checks for the key, locks it, marks it as processing, runs the operation, and only then releases the lock with the final result attached. That locking step is the part teams skip, and it’s the part that actually matters. A key that isn’t reserved atomically can be claimed by two concurrent requests at once, and now you’ve built the exact bug you were trying to prevent, just one layer down.
There’s a formal answer for what the header should even be called, and it arrived surprisingly late for something this widely used. The IETF’s HTTPAPI working group has been running a standards-track draft for an Idempotency-Key header. Revision 07 published in October 2025. It expired without becoming an official RFC. Nobody’s holding their breath for that to change, either, because the industry didn’t wait for the paperwork. Stripe shipped the pattern, everyone else copied the header name and roughly the same semantics, and a de facto standard was born the way most of them actually get born: usage first, spec later, if ever.
| Approach | How it works | Where it fits |
|---|---|---|
| Idempotency-Key header | Client-generated token maps to a cached response; server reserves it atomically before doing work | Any POST endpoint with a real side effect: charges, disbursements, order creation |
| Natural idempotency via UPSERT | Write against a stable business key (loan ID plus disbursement sequence) so the database rejects or overwrites duplicates on its own | Internal services where you control both ends and can enforce a unique constraint |
| Consumer-side dedupe | Event consumer tracks processed message IDs and skips anything it’s already handled | Pub/sub and event-streaming boundaries, where at-least-once delivery is the platform’s default guarantee, not a bug |
Where Idempotency Keys Actually Fail in Production
Three failure modes account for nearly everything I’ve seen go wrong. None of them are exotic. Same three, every time.
The reserve step isn’t atomic. A team implements the check-then-write as two separate database calls instead of one transaction or one atomic Redis command. Under load, two requests carrying the same key both pass the check before either one finishes the write, and both proceed. Congratulations, you’ve built idempotency logic with a race condition hiding inside it. AWS’s Builders’ Library writeup on retry safety flags this exact pattern as the most common mistake teams make when they build the feature themselves instead of borrowing a proven implementation.
The key gets scoped to the wrong thing. If the idempotency key is only checked against the header value, and not also tied to the authenticated caller, two different customers who happen to generate the same UUID, or a client library with a broken random-number seed, can collide. One gets the other’s cached response. In a payments context that’s not a bug ticket. That’s an incident review with legal on the call.
Failed responses get cached as if they succeeded. A request comes in, the downstream call times out, the handler returns a 500, and somewhere in the stack that 500 gets stored against the key anyway. Now every retry gets served a stale failure forever, even after whatever caused the original timeout has cleared. Only cache deterministic outcomes: a real success, or a 4xx that will fail identically no matter how many times you resend it. Never cache a 500. A 500 means the system doesn’t know what happened yet, and locking that uncertainty into the response defeats the entire point of retrying.
What the Kafka Rebuild Taught Me About Idempotency at Scale
The event-driven rebuild I’ve written about elsewhere cut downstream processing latency by 45 percent once we moved off point-to-point calls onto a Kafka pub/sub backbone. I’m proud of that number. I’m also honest about what it cost to earn.
Point-to-point failures are loud and localized. One call fails, one caller knows about it, you retry that one thing. Pub/sub doesn’t work that way. Kafka’s delivery guarantee is at-least-once by design, which means every single consumer on every topic had to become safe to process the same message twice before we were allowed to trust the platform’s own retry behavior. We didn’t get to opt out of that requirement selectively. It was binary. Either the whole downstream chain was idempotent, or the 45 percent number was a mirage sitting on top of quietly duplicated writes nobody had caught yet.
Show me the data before you believe a latency win like that. We instrumented every consumer with a processed-message log before we trusted the dashboard, specifically because a decoupled system will happily report a fast, wrong answer with the same confidence as a slow, correct one.
Build vs Buy: Do You Need Your Own Idempotency Store?
For a startup wiring up its first payment integration, the honest answer is almost always buy. Just buy it. Stripe, and most serious payment processors, give you the header and the guarantee for free. Building your own key store, with its own atomic reservation logic and its own expiry policy, to reinvent something a vendor already solved and battle-tested at a scale you don’t have yet, is effort spent in the wrong place.That calculus flips once you’re operating your own internal service mesh, or once you’re the one publishing events that other teams inside your company consume. Different math entirely. At that point there’s no vendor to buy from. You are the vendor, internally, and the idempotency guarantee has to live in your own infrastructure, whether that’s a Redis-backed key store with a TTL around 24 hours, which is the industry’s converged default, or a unique constraint sitting directly on the write.
Twenty-four hours isn’t arbitrary. It’s roughly the outer bound of how long a mobile client might sit offline before retrying, plus margin. Shorter, and you risk a legitimate delayed retry missing the cache and executing twice. Longer, and you’re paying storage cost for protection you no longer need.
The One-Page Version
Skip everything else if you want. Not this part.
- Every POST, PATCH, or PUT with a real side effect gets an idempotency key, generated client-side, sent as a header.
- The server reserves the key atomically before doing any work. One database transaction or one atomic cache command, never two separate calls.
- Scope the key to the authenticated principal, not just the header value alone.
- Cache successes and deterministic 4xx errors. Never cache a 500.
- Give the key a real expiry. 24 hours covers almost every realistic retry window without holding storage forever.
- At a pub/sub boundary, dedupe on the consumer side too. The producer’s key alone won’t save you from at-least-once delivery duplicating a message the platform already delivered once.
The Team You Need to Actually Ship This
This isn’t a junior engineer’s project. I won’t pretend otherwise. Reasoning correctly about race conditions, atomic operations, and what “the data is correct, it’s just three seconds behind” actually means under load is a specific, learned skill, and it’s scarcer than most hiring managers expect going in.KORE1’s average time-to-hire for IT and engineering roles runs around 17 days across contract, contract-to-hire, and direct placements, but that number moves depending on how narrow the skill is. An engineer who’s actually shipped an idempotency layer against a live payment rail, not read about one, takes longer to find and is worth the extra week.If the gap is specifically an Encompass or LOS integration that needs to survive retries without double-drafting a borrower, that’s a narrower search than general backend hiring, and it’s worth treating it that way from the start. If the gap is on the event-driven side, the person who can tell you whether your consumer group is actually exactly-once or just optimistically labeled that way is a different specialist than a general Kafka admin, and conflating the two roles is how searches drag past 90 days.
Retry Questions I Get From Integration Teams
Doesn’t PUT already handle this without a key?
No, and that’s the misconception that gets teams in trouble. PUT is idempotent by spec when you’re replacing a full resource at a known address, but the operations that actually break under retries, creating a charge, disbursing funds, submitting an order, are inherently POST-shaped. There’s no resource address to replace yet. The key is how you retrofit PUT-style safety onto an operation that has to be a POST by nature.
Realistically, how long do you keep a key alive before it expires?
24 hours is the number nearly everyone lands on, Stripe included. That covers a client retrying after a network blip, a mobile app waking up from being offline overnight, and most webhook redelivery windows, without holding a growing key store forever. Some regulated environments push it to 48 or 72 hours for batch settlement processes that legitimately take that long to confirm. Match it to your slowest realistic retry, not your average one.
What happens when two unrelated requests land on the same key?
Whichever request wins the race gets served to both callers. Silently. Nobody finds out until a reconciliation job flags a mismatch, days later. This is almost always a scoping bug. The key wasn’t tied to the authenticated user or account, so two legitimate but unrelated operations collided on a token that was never supposed to be shared. Scope the key to the caller from day one and this stops being possible by construction.
Only payment endpoints need this, right?
Payments are where it hurts fastest and gets noticed first, but any endpoint with a real side effect qualifies. Sending an email, provisioning a resource, kicking off a background job, submitting to a downstream partner API that charges per call. If running it twice produces two of something that should only exist once, it needs the same treatment a charge does.
Does an idempotency key guarantee exactly-once processing end to end?
Not by itself. It guarantees that one specific endpoint won’t double-execute for a repeated request. It says nothing about what happens three services downstream, where a message published once might still get consumed twice by a subscriber that hasn’t implemented its own dedupe. End-to-end exactly-once is a property of the whole pipeline, not a single header, and it has to be designed in at every hop, not just the one closest to the client.
Building a payment or transaction integration that has to survive real-world retries is a narrower hiring problem than most job descriptions admit. If that’s the search in front of you right now, reach out to our team and we’ll talk through what the role actually needs before you post it.
