How LLM inference serving works: batching, KV cache and vLLM
What happens when a server answers thousands of LLM requests: prefill and decode, the KV cache, continuous batching, PagedAttention and the metrics that matter.
AI Desk3 min read
Tag
#Mlops
Every The Latent Daily story, tool, model and resource tagged “Mlops”.
2 items
How much GPU memory does an AI model need? A sizing guide
Estimate the GPU memory needed to run or fine-tune a language model: weights, KV cache, activations and overhead, with worked examples for 8B and 70B models.
AI Desk4 min read