Three wastes, four engines, ten flags that matter — and the enterprise capability the standard material leaves out entirely.
A finished request keeps its slot until the whole batch is done.
Each request reserves the maximum possible cache and uses a fraction.
The same long prefix re-processed for every request.
Forty fine-tunes of the same base, held as forty full copies.
A research library is optimised for flexibility and model coverage; a serving engine is optimised for throughput. Both exist for good reasons.
Famous for PagedAttention. Widest coverage.
Famous for RadixAttention.
Famous for ahead-of-time compilation.
Famous for HuggingFace integration.
“vLLM for general high-throughput serving. SGLang when the workload has heavy prefix reuse — agents, few-shot pipelines, chat with a long fixed system prompt. Both do continuous batching and paged memory; the difference is what happens to the cache after a request ends. TensorRT-LLM if the model set is stable and the last ten per cent on NVIDIA hardware is worth a build pipeline.”
The context limit.
The fraction of the card the engine may claim.
One byte per cache element instead of two.
Which weight format, and therefore which kernel.
Keep and match shared prefixes.
Slice long prefills, interleave with decode.
How many cards the model is split across.
Admission control — the running-sequence cap.
“It will not start — out of memory.” Lower --max-model-len, use a quantised model, or turn on an fp8 cache.
“It starts fine and is slow under load.” Almost always not enough memory left for cache, so the batch stays small and the weight read is amortised over too few tokens. Same three levers, or more hardware.
Forty teams or forty customers each want their own tuned model, and you have four GPUs.
Forty full fine-tunes = forty copies of 14.96 GiB.
Every copy nearly the same as every other.
One base copy plus a thin pair of matrices per adapted layer.
1.07 GiB on top of one 14.96 GiB base.
One forward pass can serve requests for different adapters at once, because the base is shared and each request’s delta is applied per row.
A little throughput from the extra per-row work. A cap on how many adapters can be loaded at once. And quality somewhat below a full fine-tune where the domain shift is large.
How full the KV pool is right now.
Queue depth.
Evictions mid-generation.
The fraction matched.
Pure waiting.
Separate from TTFT.
What the user feels as the pause.
What the user feels as streaming.
Gauges tell you the state of the machine; timers tell you what a user experienced. Emitted separately — the two most teams are missing are queue time, because it is invisible in every GPU metric, and preemption count, because it rises before latency does.
Why not just run Transformers in a loop?
vLLM or SGLang, and what would change your mind?
Forty product teams each want a fine-tuned model.
Which flags would you set first on a new deployment?
What is TensorRT-LLM for, and when would you not use it?
QIs the OpenAI-compatible API actually compatible?
For the common paths, yes — chat completions, streaming, most sampling parameters. Edge cases differ: log-probabilities, some penalties, tool-calling formats, error shapes. Test the specific features you rely on rather than assuming full parity.
QWhy is my throughput lower than the benchmark?
Usually prompt and output length distribution — published benchmarks often use short fixed lengths, which flatters throughput enormously. Also check whether it measured at saturation while you care about latency at a fixed rate.
QHow many adapters can I load at once?
A configured cap, and the binding constraint is usually not memory — at 27 MB each you could hold hundreds — but the per-step work of applying many different adapters within one batch. Start small, raise it, watch inter-token latency.
QDoes LoRA slow down inference?
A little. The base matrix multiply is unchanged and a small extra product is applied per row. The overhead is modest and roughly independent of how many distinct adapters are in the batch; what matters more is whether your engine has a proper batched-adapter kernel.
QShould we build our own engine?
Almost certainly not. The three wastes are solved, the solutions are subtle — paged attention needs a custom kernel, continuous batching needs careful scheduling — and the projects doing it have large teams. The exception is a genuinely unusual requirement, and even then the answer is a fork or a plugin.
QWhat is chunked prefill and is it on?
Slicing a long prefill into pieces and interleaving them with decode steps, so one huge prompt does not stall every other stream. On by default in recent versions. Confirm rather than assume — a ten-minute check that prevents a class of complaint that never appears in the logs.
QWhy does the engine reserve so much memory at startup?
It allocates the entire KV pool up front, so it never has to allocate during serving — allocation during a decode step would be a latency spike. What looks like grabbing 90% of the card is it claiming its cache pool.
QCan one engine serve several different models?
Generally one model per process, plus as many LoRA adapters of that base as you like. Genuinely different models means separate processes and memory pools, usually separate GPUs. A router in front is the normal pattern — and size each pool from its own traffic.
QThe engine version changed and throughput moved. Normal?
Yes, in both directions, and a real operational hazard. Defaults change, kernels change, scheduling changes. Pin the version, read the release notes for the flags you depend on, and re-run the same benchmark at the same request rate.
QWhat is the difference between block sharing and prefix caching?
Block sharing works between requests running at the same time; prefix caching keeps blocks after a request finishes. The first saves memory; the second saves memory and prefill work. RadixAttention is the persistent version built as the core design.
An engine removes three wastes, and the choice between engines is a measurement rather than a preference. Measure what fraction of a typical prompt is a repeated prefix: 80–95% on agents and chat makes SGLang’s persistent cache worth up to 6.4×; 5–15% on RAG makes the whole argument evaporate. And the flag most teams leave at the default — --max-model-len — is the single biggest capacity lever they have.
Which engine, which version, and is the version pinned?
What is --max-model-len, and was it chosen or inherited?
Is chunked prefill on? Is prefix caching on? What is the hit rate?
Is preemption count monitored?
How do you serve fine-tuned variants — adapters or separate deployments?
When the engine is upgraded, is the benchmark re-run?