The Real Cost of a Synchronous API in a Pub/Sub World

Last updated: August 10, 2026

By Kris Drouet, Engineering Executive, in partnership with KORE1

We finished the migration in March. Forty-one services publishing and consuming on the bus, downstream processing latency down 45 percent, and exactly one integration still speaking plain request-response HTTP because the team that owned it was inside a release freeze and had a genuinely good reason for it.

Nobody worried about that. I did not worry about it either. One call.

Then the pricing path had a bad Tuesday in October, and I got to watch what one call actually does to a system that has otherwise been decoupled. It was not the outage that bothered me. Outages happen. What bothered me was that I had approved a nine-month architecture program, defended its cost in front of a board, and could not answer a simple question from our CFO afterward: what was that last call costing us before it broke?

I had no number. I had an architecture opinion.

So I went and built the number. This piece is that arithmetic, and it is the missing half of the decoupling playbook I run when a system turns brittle. That one covers sequencing. This one covers whether you should be sequencing anything at all. If you want the outcome side first, the full Kafka pub/sub case study and the 45 percent is written up separately.

What a Synchronous API Call Actually Costs

The cost of a synchronous API call is the combined availability, latency, and failure load that the caller inherits from everything downstream of it. A request-response call makes the caller’s uptime dependent on the callee’s uptime. An event does not. That single difference is where every number in this piece comes from.

Search this topic and you will get a hundred articles telling you synchronous calls are tightly coupled and asynchronous messaging is resilient. All true. Not one of them hands you a figure you could put in a budget request.

That gap is the reason async refactors lose funding fights to features. A VP of Product walks in with a revenue projection. You walk in with the word “brittleness.” Guess who wins.

Your Availability Is a Product, Not an Average

This is the one that changes the conversation fastest, because executives already understand compounding.

When service A calls service B synchronously and cannot complete without it, A’s availability is multiplied by B’s. Not averaged. Multiplied. Chain four of those together and the arithmetic gets ugly in a way that no single team’s dashboard will ever show you, because every team is looking at its own number and every one of those numbers looks fine.

Compound availability assumes independent failures, which is the optimistic case. Real dependencies share networks, clouds, and deploy pipelines.

Every hop you add costs roughly four and a half hours a year at that reliability level. Not because anything got worse. Because you multiplied. It compounds quietly.

Google’s SRE Workbook makes the point I want you to take from that table, and it is not the arithmetic. It is that a critical dependency’s “reliability guarantee should be at least as high as the reliability guarantee of the dependent action.” Then it warns that reasoning about redundant systems as independent “is almost never the case,” because they share failure domains and control planes. So the honest read on my table is that those numbers are the ceiling. Your real availability sits underneath them.

Run this on your own system tonight. Pick your highest-value user journey, list every synchronous hop between the first request and the response, and multiply the last twelve months of measured availability for each one. Most people doing this for the first time find two hops they forgot were in the path. Usually an auth service and something doing a lookup that could have been cached three years ago.

You Inherit the Worst Tail in the Chain

Here is where the arithmetic stops being intuitive and starts being genuinely alarming.

In The Tail at Scale, published in Communications of the ACM in 2013, Jeffrey Dean and Luiz Barroso work an example that I have now put in front of maybe a dozen executive teams. Take a server that typically responds in 10 milliseconds but has a 99th-percentile latency of one second. Handle a user request on one such server and one request in 100 is slow. Fine. Acceptable. Handle it on 100 such servers in parallel and, in their words, “63% of user requests will take more than one second.”

Sixty-three percent. From a component that is healthy 99 percent of the time.

They push it further. Even at one in 10,000 requests exceeding a second at the single-server level, “a service with 2,000 such servers will see almost one in five user requests taking more than one second.” And from a real Google service, the 99th-percentile latency for one random request measured at the root was 10ms, while the 99th-percentile for all requests in the fan-out to finish was 140ms.

Most of us are not running 2,000-node fan-outs. The mechanism does not care about the number. It only cares that you are waiting on something and cannot finish until it answers, which is precisely what a synchronous call is. Four hops with modest tails produce a user-facing tail that nobody on any individual team recognizes as theirs, and I have sat in more than one incident review where four teams each proved, correctly, that their service was fine.

Everyone was right. The customer still waited nine seconds.

Retries Multiply Fastest at Exactly the Wrong Moment

Then there is the third number, and it is the one that turns a slow afternoon into an incident.

Synchronous callers retry. They should. Retries are how you survive a blip. The trouble is that retries stack multiplicatively through layers, and Google’s SRE book gives the cleanest illustration of it in its chapter on addressing cascading failures. Four attempts at each of three layers, and “a single user action may create 64 attempts” on the database.

Sixty-four. From one click.

The same chapter describes the escalation pattern in plain terms, where “100 QPS of retries in the first second leads to 200 QPS, then to 300 QPS,” and recommends a hard server-wide ceiling instead of per-client politeness, something on the order of “only allow 60 retries per minute in a process.”

Our October afternoon was a small version of exactly this. A dependency slowed down, it did not fail, and slow is worse than failed because failed returns quickly. Connection pools filled. Threads sat waiting on sockets. The upstream service started timing out and retrying, which put more load on the thing that was already struggling, and the whole shape of it took about eleven minutes to go from a latency graph nobody was looking at to a page. Eleven minutes.

An event-driven path has none of that geometry. A consumer that falls behind produces lag. Lag is a queue depth you can graph, alert on, and drain later, and the producer never knows the difference. Backpressure instead of a cascade. That is the actual engineering argument for pub/sub, and it has almost nothing to do with speed.

Turning Three Numbers Into One Dollar Figure

Now the part that gets a project funded.

Take the compound availability from your own path. Say you measured four synchronous hops and landed at 99.80 percent, which is 17.5 hours of expected annual downtime against the 4.4 hours you would have with one. Thirteen extra hours. Multiply those hours by whatever an hour of that journey is worth to your business. A lending platform pushing 40 million dollars a year of fee revenue through an origination funnel is earning roughly 4,500 dollars an hour on that path, so thirteen hours is about 59,000 dollars.

That is a real number and it is also a small number, and you should say so out loud before a CFO says it for you.

Which is why the industry data matters here. Uptime Institute’s Annual Outage Analysis 2026 found that 57 percent of respondents to its 2025 survey said their most recent major outage cost more than 100,000 dollars, and for the second consecutive year, one in five put their most recent significant outage above one million. Its 2026 resiliency survey puts networking and connectivity issues at the top of IT service outage causes at 23 percent, ahead of power at 21 percent and IT systems and software at 18 percent. For outages traced to a third-party provider, software or configuration error leads at 54 percent, with networking and connectivity right behind at 46 percent.

Read those together and the picture is not “systems fail randomly.” It is that the connections between systems are where the money goes. The seams, not the boxes. Every synchronous hop is one of those connections, carrying its own configuration, its own timeout policy, and its own opinion about what to do when the other end is slow.

The Fourth Number Is the One That Changed My Mind

Availability, tail, retries. Those three are what I bring to a board conversation because they translate into dollars, and dollars are the language of the room.

None of them is why I actually did the migration.

The number that changed my thinking was one nobody instruments: what percentage of changes to service A require a coordinated deploy of service B. Pull it from your merge history over a quarter. Count how many pull requests could not ship alone.

Ours ran a little under a third before we decoupled. Single digits after. And a coordinated deploy is not a technical event. It is a calendar event, which means two teams negotiating a window, a rollback plan that has to account for both sides, and a change-approval conversation that in regulated lending gets its own paperwork. Three engineers, half a day, several times a month, forever.

That cost never shows up in an incident review because nothing broke. It shows up as a roadmap that keeps slipping for reasons nobody can articulate, which is the same failure mode I wrote about in telling a slow engineering org from a tired one. Velocity did not fall because people stopped trying. It fell because the graph of who has to talk to whom got denser every quarter and nobody was tracking that graph.

• Compound availability across the path, from your last twelve months of uptime data.
• Tail latency at the ninety-fifth and ninety-ninth percentile, measured at the edge rather than per service. Per-service p99 is the number that lies to you.
• Retry fan-out under failure. If you have never load-tested with a dependency deliberately degraded rather than killed, you do not know this number, and slow is the case that hurts.
• Coordinated deploys as a share of all deploys. One query against your merge history. Pick one of the four and pick this one.

When the Synchronous Call Is the Right Answer

I would rather you left this page with a decision rule than with a bias.

In 2023 the Prime Video team published a piece describing how they moved their audio and video quality monitoring service off a distributed serverless design and back into a single process, reducing infrastructure cost by more than 90 percent. The distribution itself was the expense. Passing large volumes of intermediate data between components over the network cost far more than doing the work in one place.

That result is not an argument against event-driven architecture and it was widely misread as one. It is an argument against distributing things that do not need to be distributed, which is the mirror image of the mistake this article is about. Both errors come from picking a topology first and pricing it never.

Keep the call synchronous when the caller genuinely cannot proceed without the answer and the user is standing there waiting for it. A credit decision. An authentication check. A rate quote that expires in ninety seconds. Making those async buys you a correlation ID, a polling endpoint, a WebSocket, and a state machine. You have not removed the coupling. You reimplemented it yourself with worse observability. Badly, usually.

Go asynchronous when the caller is telling the system something rather than asking it something. Notifications, audit trails, document generation, downstream analytics, anything whose failure should not fail the transaction. If your answer to “what should the user see if this fails” is “nothing, this shouldn’t block them,” you have found an event, and you should stop calling it an API. The load-side version of this decision, for platforms that have to survive a refinance surge, is broken down in the four API patterns that survive volume spikes.

Show me the data before you show me the diagram. If the four numbers say the coupling is cheap, keep the call and go build something customers asked for. I have talked two teams out of this refactor in the last three years and I would count both as wins. The same evidence standard I use for build versus buy applies here without modification.

The Staffing Problem Sitting Under All of This

One thing the architecture blogs consistently underprice.

The build is not what gets you. Running the thing does. And it gets hard in ways that only surface at three in the morning about eight months after launch. Idempotent consumers. Ordering guarantees you can actually reason about. Replay that does not double-charge anybody. Schema evolution across producers who deploy on their own schedule. A dead letter queue with an owner and a runbook instead of a dashboard nobody has opened since launch.

Plenty of engineers have used Kafka. Considerably fewer have been on the hook for a consumer group that fell four hours behind during a volume spike, and that experience gap is invisible in an interview unless you know precisely what to ask about. Not many people know what to ask.

This is where I have watched good migrations fail, and it is why I work alongside recruiters who specialize rather than generalize. KORE1’s numbers are 17 days to fill an average IT role and 92 percent retention at the twelve-month mark. In this niche I care more about the second one. A platform that loses both of its experienced consumers six months after go-live is in worse shape than it was before anybody touched the architecture, and I have seen that particular unwind happen twice. Which is also why I push clients toward direct hire for these two seats rather than a contract bridge. The person who designs your partitioning strategy should still be there the day it needs to change. Permanent seats. If that is your gap, their Kafka engineer staffing practice and their distributed systems and event-driven architecture consultants are the two doors I point people at.

The Pushback I Get on This Math

We have one synchronous call left. Is that honestly worth a project?

Wrong unit. Count request paths, not calls, because a single synchronous hop sitting in your highest-volume user journey costs more than six of them buried in an internal admin tool.

Rank the paths by revenue or by transaction volume, then run the four numbers on the top two. Most of the time the answer for a single remaining call is that you should leave it alone and put a circuit breaker and a sane timeout on it. Occasionally the answer is that the one call sits directly in the checkout path and is quietly costing more than the entire rest of the migration saved. You cannot tell which one you have by staring at an architecture diagram, and I have been wrong about it by inspection more than once. More than twice, honestly.

Our p99 looks fine. Why should I care about the tail?

It probably does look fine, because p99 is measured per service and paid per user journey. The user experiences every hop’s tail combined, and no single dashboard in your organization displays that combined number.

Measure at the edge. Client-side or at the API gateway, on the full journey, not per service. Almost everyone who does this for the first time finds a gap between their per-service p99 and their journey p99 that is larger than they expected, and the honest reaction is usually a quiet moment rather than an argument.

Can I put a circuit breaker on it and call it done?

Short answer: it helps and it does not touch the arithmetic. A circuit breaker converts a slow failure into a fast failure, which is a real improvement, but the caller still cannot complete the work.

Bulkheads, timeouts, jittered backoff, and a retry budget are all worth doing and cost a fraction of a refactor. Do them first. Do them regardless. They change your incident duration and your blast radius, and they leave the availability multiplication exactly where it was, because the dependency is still required for the request to succeed.

Who funds this? It isn’t a feature.

$100,000, which is what more than half of Uptime Institute’s respondents said their most recent major outage cost them in the 2026 analysis. That is the anchor you bring to a budget meeting.

Then pair it with your own four numbers so the figure is yours and not an industry average, and take the coordinated-deploy percentage with you, because that one converts into roadmap dates and roadmap dates are what your CEO is actually worried about. I have never once won this argument with the word “resilience.” I have won it several times with a slide that says a third of our changes require two teams to coordinate a release window.

The downstream system is a vendor. We can’t change their API.

A vendor boundary is the best possible place for a queue, and it is usually the easiest one to justify, because nobody has to argue about which internal team owns the fix.

You cannot change their interface. You can change yours. Accept the request, persist it, acknowledge it, and let a worker handle the vendor call with retries and a dead letter queue behind it. Your uptime stops being multiplied by theirs for every operation that does not need a synchronous answer, and in my experience that is most of them. The ones that genuinely need an answer in the request, like a credit pull, stay synchronous and get a strict timeout with a defined fallback behavior that someone has actually tested.

How do I know the async version will actually be cheaper?

You do not know, and Prime Video is the proof that it sometimes is not. Their move away from a distributed design cut infrastructure cost by over 90 percent.

So measure before and commit to measuring after, with the same instruments, on the same journey. Pick one seam. Cut it. Publish the number honestly even when it is unimpressive, because a modest verified result buys you the next three seams and an unverified architectural conviction buys you an argument every quarter until somebody leaves.

Price It Before You Argue About It

The thing I would have done differently is not architectural. I would have built the four numbers first, before the program, before the board deck, before anyone drew a single box on a whiteboard.

Not because the answer would have changed. It would not have. But I spent nine months defending a decision with adjectives when I could have been defending it with a spreadsheet, and the difference between those two conversations is the difference between a team that gets funded once and a team that gets funded repeatedly.

One call. Four numbers. An afternoon.

If you are staring at a system that everyone agrees is too coupled and nobody can price, that is the conversation I enjoy most and I am happy to have it. Message me. Kris Drouet on LinkedIn.

And if the numbers say go and the honest blocker is that you do not have two people who have operated an event-driven platform at volume before, that is a search, not a design review. Talk to a KORE1 recruiter about what the gap really is.