The Waiting GPU: Continuous Batching Explained - 23x From One GPU

Описание: Your inference GPU costs $30 an hour and works about 30% of the time. Not broken - scheduled wrong.

Episode 2 of The Inference Stack:
Prefill is compute-bound (90-95% utilization). Decode is memory-bound: every token forces a full re-read of the model weights for ONE word - utilization falls to 20-40%
Static batching locks the whole batch behind the longest answer: throughput collapses from 200 to 81 tokens/sec as answer lengths vary
Continuous batching schedules per TOKEN: the moment a request finishes, its slot goes to the next request in the queue - mid-flight
PagedAttention packs each conversation KV cache into small pages, so far more conversations fit at once
Result: up to 23x the throughput of naive serving. Same GPU, same model

In practice: serve with vLLM, TGI, or TensorRT-LLM. Watch queue depth and batch occupancy. And remember - GPU utilization percent lies on memory-bound work. The real meter is tokens per second, per GPU.

Episode 1 - The Token Bill:    • Why 90% of Your AI Bill Is Input Tokens - ...  
Full series:    • The Inference Stack - Where Your AI Money ...  

Built by Teqners - AI/ML infrastructure, done right: https://www.teqners.com

#AI #LLM #GPU #MachineLearning