AI GitHub
vLLM for shared GPU inference
Serve open-weight language models to multiple clients while managing attention-cache memory.
Key Takeaways
- vLLM batches requests and uses paged storage for attention keys and values.
- Use a supported model architecture and accelerator configuration.
- Replay a representative mix of short chat prompts and longer documents at increasing concurrency.
- A large model or long context can consume the memory needed for concurrent sequences.
How it works
vLLM batches requests and uses paged storage for attention keys and values. Its HTTP server makes model loading and batching choices part of the serving configuration.
What you need
Use a supported model architecture and accelerator configuration. Decide on the request mix and latency objective before comparing the service with a single-user runtime.
What to try first
Replay a representative mix of short chat prompts and longer documents at increasing concurrency. Record throughput alongside time to first token and the slowest requests.
Add a workload that alternates long prompts and one-line chats. A good average tokens-per-second figure can hide request queuing for the small jobs.
Where to look for failures
A large model or long context can consume the memory needed for concurrent sequences. Test your own model, quantization and accelerator combination before using benchmark numbers for capacity planning.
Related Franklin pages
Compare throughput with the same model family, context lengths and output limits where possible. A runtime optimized for shared GPU traffic faces different constraints from one tuned for a single device.
- Serve language model workloads that reuse instructions or context across requests. SGLang for repeated-prefix inference
- Run and quantize compatible language models for GPU inference with the EXL3 format. ExLlamaV3 for EXL3 quantization
For the concepts behind hosting model endpoints, see Franklin’s glossary. Model serving
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗