LLM Batching Explained: Static vs Dynamic vs Continuous Batching

Imagine four people booking a ride to the same destination. If each person takes a separate taxi, four vehicles make the same trip. But if they travel together in a bus, everyone reaches the destination in a single trip. The bus carries more people at once and makes better use of the resources. Batching in AI works in a similar way. Instead of processing one request at a time, a server groups multiple requests together so the GPU can process them more efficiently. This helps serve more users while making better use of expensive computing resources. But how those requests are grouped and scheduled also affects how quickly users get responses and how much it costs to serve them. There are three common approaches to batching: static batching, dynamic batching, and continuous batching. Let’s understand what batching means technically and how these approaches differ. Table of contentsWhat is Batching?Why LLMs Need Batching?Static The Four Caches in LLM ServingDynamic BatchingContinuous Batching Static vs Dynamic vs Continuous BatchingConclusion What is Batching? Batching is the process of running multiple inputs through a model in a single forward pass instead of processing each input individually. In an LLM, each input is a sequence of tokens, so the model processes a batch of sequences together. Conceptually, instead of an input tensor shaped [1, seq_len], the model processes a batch shaped [batch_size, seq_len]. This allows the GPU to perform computations across multiple requests in parallel, improving hardware utilization and throughput. However, LLMs generate responses token by token, and different requests can take different amounts of time to finish. How the server groups requests, handles incoming traffic, and frees up resources as responses finish determines which batching approach works best. Why LLMs Need Batching? The answer is simple, during generation, the GPU is limited by how fast it can read memory, not how fast it can do the math. Batching lets one memory read serve many requests. LLM inference has two phases: Prefill: The model reads your whole prompt in one forward pass. All prompt tokens are processed in parallel, so this phase keeps the GPU’s compute units busy. Decode: The model generates one token at a time. Each new token needs a full forward pass, which means reading every model weight from GPU memory again. Decode is where the waste happens. Some rough numbers for a 7B model in FP16 on an H100: The weights take about 14 GB. H100 memory bandwidth is about 3.35 TB/s, so reading 14 GB takes roughly 4 ms. That caps a single request at around 240 tokens per second. Each token needs about 14 GFLOPs (2 × parameters). Doing that in 4 ms is around 3.5 TFLOPS, while the H100 offers close to 1,000 TFLOPS of dense BF16. So at batch size 1, the GPU’s compute sits more than 99% idle. It spends almost all its time loading the weights. If you run 32 sequences in the same steps, the weights are loaded once and you get 32 tokens instead of 1, nearly the same step time. That’s the whole case of batching higher throughput and lower cost per token, without buying more GPUs. There is a limit. Every sequence in the batch keeps its own KV cache in GPU memory, and that cache grows with every token. Batch size ends up bounded by memory, which becomes important when we get to continuous batching. Also Read: The Four Caches in LLM Serving Static The Four Caches in LLM Serving Static batching groups a fixed set of requests, runs them together, and returns results only when every request in the group is finished. Nothing new can join until the whole batch is done. The model.generate() call from earlier is static batching. It’s the default behavior of most hand-written inference loops. The problem is that LLM outputs vary wildly in length. Say a batch of 4 requests ends up generating 20, 50, 100 and 500 tokens. Then the batch runs for 500 decode steps, because it waits for the longest one. The 20-token request is done at step 20, but its slot sits idle (or computes padding) for 480 more steps. Useful work is 20 + 50 + 100 + 500 = 670 token-steps out of 4 × 500 = 2,000 possible. That’s about 33% utilization.A request that arrives at step 1 has to wait 499 steps before it can even start. This is called head-of-line blocking: one long request holds up everyone behind it. To compare the three strategies fairly, let’s build a small simulator. It skips the model and only tracks decode steps, which is enough to see how scheduling affects GPU usage and latency. A new request arrives every 10 steps, and the GPU can run 4 sequences at once. Code: from collections import deque # Each request: (id, arrival_step, output_tokens) # A new request arrives every 10 decode steps, with very different output lengths. lengths = [120, 30, 60, 300, 40, 90, 25, 150, 50, 70, 35, 80] REQUESTS = [(f"R{i}", i * 10, n) for i, n in enumerate(lengths)] SLOTS = 4 # max sequences the GPU runs at once def summarize(name, total_steps, useful, finish): latency = [finish[rid] - arr for rid, arr, _ in REQUESTS] util = useful / (total_steps * SLOTS) print(f"{name: tokens left step, useful, finish = 0, 0, {} while waiting or running: # every step: fill free slots with requests that have arrived while waiting and len(running) < SLOTS and waiting[0][1] <= step: rid, _, n = waiting.popleft() running[rid] = n useful += len(running) step += 1 for rid in list(running): running[rid] -= 1 if running[rid] == 0: # done: return now, free the slot finish[rid] = step del running[rid] return step, useful, finish if __name__ == "__main__": summarize("Static", *static_batching(REQUESTS)) summarize("Dynamic", *dynamic_batching(REQUESTS)) summarize("Continuous", *continuous_batching(REQUESTS)) The difference is that the scheduling loop runs inside the decode loop. Each request’s finish time is its own, not the batch’s. What makes it work in practice: The idea is simple, but real systems need a few extra pieces: Mixing prefill and decode: New requests need prefill while existing ones generate tokens. Orca uses selective batching to handle both efficiently. KV cache management: vLLM’s PagedAttention stores KV caches in fixed-size blocks, reducing memory fragmentation and allowing more sequences to fit in a batch. Chunked prefill: Systems like Sarathi-Serve split long prompts into smaller chunks and interleave them with decode steps to reduce delays for running requests. Static vs Dynamic vs Continuous Batching Running the full simulator on the same 12 requests gives this: Static steps=560 slot_util=47% avg_latency=402 first_two=[330, 320] Dynamic steps=665 slot_util=39% avg_latency=418 first_two=[135, 125] Continuous steps=330 slot_util=80% avg_latency=118 first_two=[120, 30] A few things stand out: Continuous batching finishes the same work in 41% fewer steps (330 vs 560) and cuts average latency from 402 steps to 118. The GPU slots are busy 80% of the time instead of 47%. Dynamic batching helps the first requests get results about 2.5x sooner than static, since it doesn’t wait for a full batch. But it does worse overall, because it launched half-empty batches that then had to run to completion. That’s the trade-off you get when a request-level batcher meets variable-length generation. R1 asked for only 30 tokens. With static batching it waited 320 steps for its answer. With continuous batching it waited exactly 30. The simulator ignores prefill cost and memory limits, so real numbers will differ, but the pattern holds on real serving engines. Static Dynamic Continuous When a batch starts Once a fixed group is ready When full or the timeout hits Every decode step New requests join mid-run? No No Yes Finished sequences Wait for the whole batch Wait for the whole batch Returned at once, slot reused Scheduling level Per batch Per batch Per iteration (token) Best fit Offline jobs, similar output lengths Fixed-cost models (BERT, embeddings, vision) LLM serving with variable outputs Where you’ll see it Plain model.generate() loops Triton dynamic batcher vLLM, TGI, TensorRT-LLM, SGLang If you remember one line: static batching waits for everyone, dynamic batching waits less to start but still waits for everyone to finish, and continuous batching doesn’t wait for anyone. That’s why every serious LLM serving stack today is built on continuous batching, and why swapping a hand-rolled generate() server for vLLM is often the cheapest speedup you’ll ever get. Conclusion Batching exists because LLM decode is memory-bound. Reading the weights once and using them for many sequences is the single biggest lever you have on cost per token. The three strategies are really a story of moving the scheduling decision to a finer level. Static batching decides once per group. Dynamic batching decides once per batch but stops waiting for a full group. Continuous batching decides on every token, so short requests never get stuck behind long ones. For most teams, the practical takeaway is simple: don’t build your own batching loop for LLM serving. Use an engine like vLLM, TGI or TensorRT-LLM, then spend your time tuning the knobs that matter, such as maximum concurrent sequences, tokens per step, and how much GPU memory goes to the KV cache. Hello, I am Nitika, a tech-savvy Content Creator and Marketer. Creativity and learning new things come naturally to me. I have expertise in creating result-driven content strategies. I am well versed in SEO Management, Keyword Operations, Web Content Writing, Communication, Content Strategy, Editing, and Writing.