Last updated: July 1, 2026
Kafka Pub/Sub Case Study: How Decoupling Cut Our Downstream Latency 45%
By Kris Drouet, Engineering Executive, in partnership with KORE1
Replacing a tangle of point-to-point integrations with a Kafka pub/sub backbone cut our downstream processing latency by 45 percent, removed the bottlenecks that made every change risky, and let the platform absorb peak transaction volume without falling over. This is the actual migration, the parts that worked, the parts that hurt, and the honest question of whether you should attempt it at all. Decoupling is an architecture decision first and a Kafka decision second. Get that order wrong and you buy yourself a more expensive version of the same mess.
The night I decided we had to decouple, nothing was technically broken. That was the maddening part. Every service was up. Every health check was green. And a single spike in loan pricing volume had still managed to back up a downstream reconciliation job for three hours, because that job was waiting on a synchronous call to a service that was waiting on another service that was, at that exact moment, busy answering something else.
Nobody wrote a bug. The system did precisely what we built it to do. We had just built it to fail this way.
I have spent most of a 25-year career in fintech and mortgage technology, and the coupled-system failure is the one I see most. It rarely looks like a crash. It looks like slowness that spreads. One service gets busy and four others start holding their breath. In the Clarity Stack, the deeper diagnosis for why teams stall, this is Layer 3 territory: architecture decisions nobody wrote down, each one reasonable on its own, compounding into a platform where a change here breaks something three systems away.
The System We Had: Everything Wired to Everything
Picture a dozen critical services, each one talking directly to the others over synchronous calls. Pricing calls the rate-lock service. Rate-lock calls the document service. The document service calls back to pricing to confirm. Underwriting listens for all of it. Reporting scrapes the lot on a schedule. Every one of those arrows was an integration somebody hand-wired, tested once, and then never wanted to touch again.
That is the trap of point-to-point. Each connection is sensible. The mess is emergent. Nobody plans it. You do not decide to build a brittle system. You add one reasonable integration at a time until the graph of dependencies is denser than the org chart, and then the first time you try to change the pricing contract you discover it is load-bearing for six things you forgot existed.
Martin Fowler has written for years about the difference between systems that pass commands around and systems that emit events. The distinction sounds academic until it is 2 a.m. In a command system, the caller has to know who to call, and it has to wait. In an event system, the thing that happened just gets announced, and whoever cares can listen on their own time. Our platform was all commands. Everybody knew everybody. Nobody could move.
Why Point-to-Point Integration Breaks at Volume
Point-to-point integration breaks at volume because latency is contagious. When services call each other synchronously, one slow consumer becomes every caller’s problem, and the delay travels backward up the chain until a spike in one corner of the system shows up as a timeout somewhere completely unrelated.
The cost is not just speed. It is fear. When every integration is a tripwire, engineers stop making changes they should make. The team you hired to move fast starts routing around the risky parts of the codebase, which are also, not coincidentally, the important parts. Velocity does not die in a big dramatic outage. It dies in a hundred small decisions to not touch something. Quietly. Every day.
Google’s DORA research has spent years measuring what actually predicts high-performing engineering teams, and one of the strongest signals is a loosely coupled architecture: teams that can change, test, and deploy their piece without coordinating with everyone else. We had the opposite. We had an architecture that required a group chat to change a field name.
| Point-to-point, synchronous | Pub/sub, event-driven |
|---|---|
| The caller must know every consumer by name | The producer announces an event and knows no one |
| One slow service stalls everyone upstream | A slow consumer falls behind alone, then catches up |
| Adding a consumer means editing the producer | Adding a consumer means subscribing to a topic |
| A traffic spike ripples through as timeouts | A traffic spike becomes a queue that drains |
What We Actually Built: A Kafka Pub/Sub Backbone
We put Apache Kafka in the middle. Not as a magic box. As a commitment. Producers stopped calling consumers and started publishing events to topics. A price got locked, that fact went onto a topic, and every service that cared about locked prices read it at its own pace. The reconciliation job that used to block on a live call now read from a log that was always there, whether the upstream service was having a good night or not.
Should you build your own broker to do this? No. This is the cleanest build vs buy call you will ever make. Adopt Kafka, or a managed flavor of it, and spend your engineering budget on the part that is actually yours: the event model. Deciding what counts as an event, what goes in the payload, and who owns each topic is the real work. The broker is boring. Boring is the point. The broker is a solved problem. Your domain is not.
A few things we made non-negotiable early, because we had watched teams skip them and regret it:
- A schema registry from day one. Events are contracts. We ran Avro schemas through a registry so a producer could not quietly change a payload and break every consumer at once. The version we shipped without this on an earlier project cost us a weekend. We did not repeat it.
- Idempotent consumers. Kafka gives you at-least-once delivery, which is a polite way of saying you will occasionally see the same event twice. If processing it twice corrupts your data, that is your bug to fix, not Kafka’s. Every consumer got a dedupe key.
- A dead-letter topic. When a message could not be processed, it went somewhere visible instead of vanishing or blocking the partition behind it. Poison messages are going to happen. The only question is whether you find out from a dashboard or from a customer.
- Partitioning by the key that actually matters. We keyed by loan ID so everything about one loan stayed in order, and unrelated loans processed in parallel. Get the partition key wrong and you either lose ordering or lose your parallelism. There is no third option.
The 45 Percent Number, and How We Measured It
Here is where I get to say the thing I say to every team that shows me a shiny before-and-after: show me the data, and show me how you measured it, because a number without a method is a marketing slide.
We measured downstream processing latency as the time from a business event happening, a price locked, a document generated, to the last dependent system finishing its work on it. Same definition before and after. Same peak-hour windows. Same volume tiers, so we were not quietly comparing a busy Tuesday to a quiet Sunday. Across the pipelines we moved onto the pub/sub backbone, that end-to-end latency dropped by 45 percent. Real number. Held up.
The interesting part is where the 45 percent came from. Almost none of it was Kafka being fast, though it is. Most of it was work that used to happen in a nervous sequence now happening in parallel, and the removal of the wait states where service A sat idle holding a connection open while service B finished. We did not make the individual steps faster. We stopped making them stand in line. The bottleneck was never compute. It was coordination. Coordination was the tax.
| What changed | Before (point-to-point) | After (Kafka pub/sub) |
|---|---|---|
| Downstream processing latency, peak | Baseline | 45% lower |
| Effect of one slow consumer | Stalls the whole chain | Isolated to that consumer |
| Adding a new downstream system | Edit and redeploy the producer | Subscribe to an existing topic |
| Behavior under a volume spike | Cascading timeouts | A queue that drains after the peak |
What I Would Tell You Before You Start
I am not going to pretend this was free, and I am not going to tell you every org needs it. Plenty do not. If you have six services and they mostly leave each other alone, adding Kafka is a way to feel like a serious architecture team while making your on-call rotation worse. Coupling is only worth fixing when the coupling is what is hurting you.
The bill comes due in a few places. You are now running a distributed log, which means you need people who understand consumer lag, rebalancing, and what happens when a partition leader goes down. Debugging gets harder before it gets easier, because the story of a single loan is now spread across topics instead of sitting in one call stack. And eventual consistency will surprise a product manager who expected the old behavior where everything updated at once. None of that is a reason not to do it. It is a reason to go in with your eyes open.
The teams that regret this migration almost always made the same mistake. They treated it as a technology swap instead of a design exercise. They stood up a Kafka cluster, pointed their existing chatty services at it, and recreated point-to-point messaging with extra steps. Kafka did not fix them, because Kafka was never the fix. The event model was. I made the broader version of this argument about defending unglamorous foundational work in operational discipline is not bureaucracy, and it applies here in full.
The Hiring Reality Underneath an Event-Driven Refactor
This is the part where I should disclose that KORE1 makes money when you cannot staff this work yourself. Read the next two paragraphs with that in mind. I would rather you know my angle than wonder about it.
An event-driven platform is only as good as the people who can reason about it at 2 a.m. Point-to-point failures are easy to diagnose: you follow the call stack. Pub/sub failures are diffuse, and they reward engineers who can hold a distributed system in their head, think in terms of ordering and idempotency, and stay calm when the answer is “the data is correct, it is just three seconds behind.” That is a specific, and genuinely scarce, kind of engineer. Scarce on purpose. In regulated industries the bar is higher still, for reasons I laid out in engineering leadership in regulated industries.
KORE1 has placed engineers across more than 30 U.S. metros for two decades, with a 17-day average time-to-hire for IT roles and a 92 percent twelve-month retention rate on those placements. Retention is the number that matters for an architecture like this. The person who designs your topic layout and consumer-group strategy is the person you least want walking out the door at month nine, because a lot of the reasoning lives in their head before it lives in a doc. If your team is thin on this skill set, KORE1’s Kafka engineer staffing practice places exactly this profile, and the surrounding bench through our DevOps and data engineering teams, often on a direct hire basis.
The Bottom Line Before You Rip Out the Point-to-Point Wiring
Decoupling worked for us because the coupling was the actual disease. The 45 percent was real, and it held. But the number is not the lesson. Not the number. The lesson is that we stopped forcing independent work to happen in a nervous single-file line, and the latency was just the coupling becoming visible on a graph.
So do not start with Kafka. Start with a map of who waits on whom, and be honest about which of those waits are costing you. If the answer is “not many,” keep your architecture boring and go solve a real problem. If the answer is “most of them,” then you have a coupling problem, and pub/sub is one of the few tools that treats the disease instead of the symptom.
If you want to talk through whether your platform is actually coupled or just busy, find me on LinkedIn. And if the gap is the senior bench that can design and run the thing, KORE1’s Kafka and streaming engineers are the people we place, or start the search with our team directly.
What Leaders Ask Me Before an Event-Driven Refactor
Do I actually need Kafka, or is this resume-driven architecture?
Usually you need it less than your loudest engineer thinks. The test is coupling pain, not fashion. If one service getting busy regularly slows down unrelated work, that is a real pub/sub problem. If it does not, Kafka is overhead wearing a nice suit.
I have killed more of these projects than I have approved. The honest signal is whether you can point at recurring incidents where latency spread from one part of the system to another. When that pattern is chronic, event-driven decoupling earns its cost. When it is not, you are adding operational surface area to solve a problem you do not have, and your on-call engineers will quietly resent you for it.
Realistically, how long before we see a number like 45 percent?
Plan in quarters, not sprints. Our first topics were live in weeks, but the latency win only showed up once enough of the critical path had moved off synchronous calls, which took the better part of two quarters. A half-migrated system can be slower than either extreme.
That middle stretch is the dangerous one. You are running two paradigms at once, some traffic point-to-point and some through Kafka, and the seams between them are where the weird bugs live. Sequence the migration so each phase is shippable and reversible. Do not try to flip the whole platform in one release. I have seen that ambition turn a good architecture decision into a bad quarter.
Won’t an event-driven system just move the complexity somewhere else?
Partly, yes, and anyone who tells you otherwise is selling something. You trade the complexity of tight coupling for the complexity of distributed state. The trade is worth it when coupling is your bottleneck, because distributed-state problems are at least ones you can isolate and test.
The complexity does not disappear. It changes shape. Coupling complexity is systemic, and it fights every change you try to make. Distributed-state complexity is local: idempotency, ordering, eventual consistency, each one a bounded problem a good engineer can reason about and cover with tests. I will take ten bounded problems over one systemic one every time. That is the whole trade, stated plainly.
What breaks first when you decouple with pub/sub?
Anything that quietly assumed everything updated at the same instant. Reports that read across services mid-flight, screens that expected data to be there the moment a button was clicked, and any workflow built on the idea that step two could see step one’s result immediately.
Eventual consistency is the culture shock, not the code. Your engineers will adapt to it in a week. Your product and your customers might take longer, because “it is correct, it is just a few seconds behind” is a genuinely new promise for a lot of teams. Get ahead of it. Decide on purpose which flows can tolerate a short delay and which truly cannot, and design the exceptions deliberately instead of discovering them in support tickets.
How many engineers do we need who actually know Kafka?
Fewer than you fear to build it, more than you expect to run it. Two or three engineers who genuinely understand event-driven design can architect the migration. Keeping it healthy in production is the ongoing cost most teams underestimate.
The design phase is concentrated and short. The operational phase never ends. Consumer lag, rebalancing, schema evolution, capacity planning, that is steady-state work, and it wants people who have run a distributed log before, not people learning on your production traffic. This is where the hiring math bites, and where thin teams either staff up deliberately or slowly rediscover every lesson the hard way.