Disaggregation Is a Thousand-GPU Problem

Every major inference framework shipped prefill-decode disaggregation this year. NVIDIA built it into Dynamo. SGLang made it the default for large-scale deployments. vLLM added a KV connector API to support it natively. The consensus is forming fast: split your prefill and decode onto separate GPU pools, and throughput improves.The consensus is wrong for most teams.Doubleword's analysis shows that a balanced disaggregated deployment matches colocated throughput, but at small GPU counts the rounding losses dominate: you can't allocate fractional GPUs, so the specialization gains get eaten by incomplete worker utilization. The practical benefit at that scale is independent SLO tuning, not throughput.A June 2025 study evaluating hundreds of thousands of design points found that disaggregation is most effective for prefill-heavy traffic patterns and larger models. For the mixed-traffic workloads most teams actually run, queueing and inter-node KV cache transfer dominated end-to-end latency. The teams that disaggregated moved the bottleneck. They didn't remove it.I ran into this on an inference workload serving a mid-size classification model. TPOT spiked under bursty traffic, and the first instinct was to separate prefill from decode. Instead, I enabled chunked prefill on the same GPU pool. TPOT stabilized. The problem was scheduling interference, and chunked prefill handled it without adding a network hop.What Disaggregation Solves: The Interference ProblemPrefill and decode have opposite hardware profiles. Prefill is compute-bound: parallel matrix multiplications across the full input sequence, GPU compute utilization at 80 to 95 percent. Decode is memory-bandwidth-bound: sequential KV cache reads generating one token at a time, compute utilization below 5 percent on an H100.When both share a GPU, they fight over the same resources. A single large prefill request arriving mid-decode inflates time-per-output-token by 2 to 30x under bursty workloads. The decode batch stalls while prefill saturates the compute units.DistServe proved the fix works at scale: separating prefill and decode onto dedicated pools served 7.4x more requests within the same latency constraints. At the scale DistServe benchmarked, disaggregation is unambiguously the right call.The diagram below shows where the bottleneck sits in each approach: colocated, disaggregated, and chunked prefillImage by authorBut there is a simpler fix that handles most of the interference without the architecture change.Chunked Prefill: The Fix Most Teams Actually NeedChunked prefill breaks long prefill requests into smaller chunks and interleaves them with decode batches on the same GPU. No separate node pools. No KV cache transfer over the network. No P:D ratio to tune.TNG Technology Consulting measured a 50 percent increase in total token throughput using standard vLLM with chunked prefill enabled. The decode batches still ran between prefill chunks on the same hardware, but the scheduling interference dropped to a level most production workloads can tolerate.Think of it like a toll plaza on a highway: instead of closing all lanes for a single oversized truck, chunked prefill lets the truck pass through one lane at a time while regular traffic keeps moving through the others.Chunked prefill doesn't eliminate interference entirely. It bounds it. Each prefill chunk occupies the compute units for a bounded duration, then yields to decode. For workloads below roughly 50 requests per second with moderate prompt lengths, that bound is tight enough.Three Costs of Disaggregation: What the Explainers SkipThe explainer articles cover what disaggregation gains. They skip what it costs.The KV transfer tax. Every request that finishes prefill must ship its KV cache to a decode node over the network. For a 70B-parameter model, that's roughly 2.6 GB per request.When prefill and decode share a node, the KV cache stays in GPU memory. Once you disaggregate across nodes, that transfer hits the network interconnect, and the available bandwidth drops by orders of magnitude depending on your topology.It's like splitting a factory assembly line into two buildings: the specialization per building improves, but now you're trucking half-finished parts between them. Without InfiniBand or NVLink in the same rack, the trucking cost dominates. You can't disaggregate your way out of a slow network.The operational surface. Disaggregation doubles your infrastructure management. You now run separate prefill and decode node pools, each with its own scaling policy. The P:D ratio depends on your workload mix: LMSYS independently evaluated 4 prefill nodes and 9 decode nodes for DeepSeek-R1, and those numbers shift the moment your prompt-to-output length ratio changes.There's no graceful fallback. If a prefill node goes down, decode nodes cannot fill in. The roles are assigned at launch.The silent failure cliff. At low concurrency, disaggregated serving works fine. At production concurrency, it breaks in ways that produce no errors. SGLang issue #9266 documents systematic KV cache transfer failures at 64 or more concurrent requests, returning HTTP 400 to clients.Issue #30233 describes a worse failure: when input exceeds the maximum request length, the prefill side aborts but still transfers a single-token KV cache. The decode side generates from uninitialized memory. No error surfaces to the caller.The Decision Math: Three Conditions That Must HoldModular's inference handbook reports a 20 to 30 percent performance drop from disaggregation on small or untuned workloads. The overhead is real and front-loaded: you pay the KV transfer cost on every request regardless of whether the throughput gain materializes.Disaggregation pays for itself only when three conditions hold simultaneously:Enough GPUs for clean allocation. DeepSeek needed thousands of GPUs before the P:D ratio produced integer node counts that matched their traffic mix. At 8 to 16 GPUs, your ratio options are 1:7 or 2:6, neither of which may fit your workload.Network bandwidth that sustains KV production rate. If your prefill pool generates KV caches faster than the interconnect can deliver them, decode nodes idle waiting for data. The bottleneck migrates from compute interference to network transfer.Dynamic autoscaling for shifting traffic mixes. A chat workload (short prompts, long decode) needs a different P:D ratio than a RAG pipeline (long prompts, short decode). If your traffic mix shifts during the day and your node pools are static, you are over-provisioned on one side and starved on the other.If any one of these conditions does not hold, chunked prefill is the better default. The decision flowchart below maps these three conditions to a concrete routing choice.Image by authorDimensionChunked PrefillDisaggregatedThroughput gain+50% (TNG, vLLM)+7.4x at scale (DistServe)Network requirementNone (same GPU)InfiniBand or NVLinkOperational overheadvLLM flagSeparate pools, P:D tuning, KV routerFailure modesBounded interferenceSilent KV corruption, concurrency cliffsScale thresholdAny~1,000+ GPUsMulti-turn penaltyNoneKV state stranded on decode nodesChunked prefill vs. disaggregated serving: throughput, operational cost, and failure characteristics.What to Measure Before You DecideDon't split your infrastructure based on architecture diagrams. Measure first.TPOT at p95, not p50. The median hides the interference spikes that disaggregation targets. If your p95 time-per-output-token is within SLO on chunked prefill, you do not have the problem that disaggregation solves.Prefill fraction of TTFT. Profile your actual requests. If prefill execution is a small fraction of time-to-first-token, the bottleneck is elsewhere: queueing, scheduling, or network. Disaggregation won't fix those.KV transfer reliability at concurrency. Before committing to disaggregation at scale, run it at production concurrency levels, not at low QPS only. The failure modes documented in SGLang's issue tracker don't appear until you cross a concurrency threshold.Conclusion: Start With Chunked PrefillDefault to chunked prefill. It solves the scheduling interference problem that most teams actually have, with zero network overhead and no operational surface expansion. Benchmark your p95 TPOT. If it holds, stop there.Disaggregation is the right architecture above roughly a thousand GPUs, with fast interconnect, and with the engineering capacity to manage P:D ratio tuning and KV transfer reliability. Those conditions describe hyperscalers and large inference providers. They do not describe most production teams shipping LLM features in 2026.Further ReadingWhen to Disaggregate (Doubleword's analysis of conditions required for disaggregation to pay off)Beyond the Buzz: A Pragmatic Take on Inference Disaggregation (June 2025, systematic evaluation of disaggregation across hundreds of thousands of design points)Chunked Prefill on H100 (TNG's +50% throughput measurement)Modular LLM Inference Handbook (Modular's 20-30% drop warning and decision criteria)The Inference Unbundling (Wing VC's analysis of the compute/bandwidth split)···Thanks for reading. I'm Mostafa Ibrahim, founder of Codecontent, a developer-first technical content agency. I write about agentic systems, RAG, and production AI. If you'd like to stay in touch or discuss the ideas in this article, you can find me on LinkedIn here.