Continuous Batching Explained: Iteration-Level Scheduling in vLLM (Orca Paper)

Описание: Every batching system before 2022 made the same mistake, and once you name it the fix is obvious. They all made ONE scheduling decision per request. Your short request finishes at step four and sits in the batch until step four hundred, everybody waits on whoever asked the longest question, and anybody who arrives a millisecond late waits for the entire round.

Orca changed WHEN the decision gets made, not what the math does. Run one step, hand control back, choose again. That one move is worth 36.9x the throughput of NVIDIA FasterTransformer at equal latency, and every serving stack you can name is built on it now.

This episode is the mechanism, the paper's own numbers, what it looks like in vLLM 0.27.1 today, and the bill that comes with it.

Chapters:
0:00 Which requests get served on the next step
0:50 Attention has no weights, and your real ceiling is KV cache
2:50 Static and dynamic batching, and the one bug they share
4:37 Orca: iteration-level scheduling and the admission check
6:39 Selective batching, and why splitting around attention is free
8:07 36.9x at equal latency, and the vLLM scheduler today
10:03 CUDA graphs, batch-shape padding, and --enforce-eager
12:23 What actually changed, and two things to do this week

The one thing to do today: find the average batch size your server is ACTUALLY running and put it next to the number you assumed. If they disagree, something in front of your engine is still forming batches the old way. Then find max_num_seqs, work out what your KV cache can genuinely hold, and see which of the two is the one stopping you.

Versions move, and two claims here are version-specific. The vLLM scheduling representation was read from v0.27.1 (released 11 August 2026): SchedulerOutput.num_scheduled_tokens is a dict[str, int], commented "req_id to num_scheduled_tokens". SGLang's default schedule_policy was read from v0.5.18 (21 August 2026) and is fcfs; it was lpm back at v0.4.0. Check both against your own version before building on them.

Not spoken, but worth having:
• vLLM's default CUDA graph capture sizes are [1, 2, 4] + range(8, 256, 8) + range(256, max, 16) where max = min(max_num_seqs * 2, 512). Above the largest captured size, no graph is used.
• --enforce-eager logs "Enforce eager set, disabling torch.compile and CUDAGraphs" — it turns off BOTH, not just graph capture.
• Batching attention buys you no PARAMETER reuse, which is not the same as buying you nothing. The linear projections around attention get full parameter reuse across the batch; the attention operation itself has no parameters to reuse, which is exactly why Orca could split it out for almost free.
• NVIDIA/FasterTransformer's README: "FasterTransformer development has transitioned to TensorRT-LLM... the repo will stay up, but will not have further development." So the system Orca beat 36.9x was NVIDIA's, and NVIDIA's successor to it does iteration-level batching.
• Orca's Select() breaks rather than skips when a reservation will not fit, so a request that is too large stops admission for that round entirely.

Sources, all re-checked on 23 August 2026 (FasterTransformer README, 24 August 2026):
Orca (Yu, Jeong, Kim, Kim, Chun), USENIX OSDI 2022 — https://www.usenix.org/conference/osd...
vLLM V1 architecture — https://vllm.ai/blog/2025-01-27-v1-al...
vLLM v0.25.0 release — https://github.com/vllm-project/vllm/...
PyTorch CUDA Graphs — https://docs.pytorch.org/docs/stable/...
TensorRT-LLM in-flight batching — https://nvidia.github.io/TensorRT-LLM/
SGLang server arguments — https://github.com/sgl-project/sglang
NVIDIA FasterTransformer (superseded) — https://github.com/NVIDIA/FasterTrans...

Measured figures quoted from this course's own bonus lab (NVIDIA L4, Qwen3-8B, bf16, vLLM 0.27.1 defaults, 200 tokens in and 300 out). AGGREGATE server throughput across all concurrent sequences: 16.5 output tokens/s at concurrency 1, 382.5 at concurrency 32. PER USER over the same sweep, p50 time-per-output-token went from 60.3 ms to 79.3 ms. Those are two different kinds of number and the episode says so out loud.

Inference Optimization, from first principles to production. Module four: batching and scheduling.

#ContinuousBatching #vLLM #LLMInference #InferenceOptimization #Orca #CUDAGraphs #GPU #LLMServing #MLOps #AIInfrastructure #MachineLearning #DeepLearning