AI GitHub
SGLang for repeated-prefix inference
Serve language model workloads that reuse instructions or context across requests.
Key Takeaways
- The inference runtime can reuse cached attention state for common prefixes and supports structured outputs on compatible configurations.
- Plan how prompts will be constructed by the client.
- Benchmark repeated-prefix and unique-prefix workloads separately.
- Prefix caching helps only when requests share tokens in the relevant order.
How it works
The inference runtime can reuse cached attention state for common prefixes and supports structured outputs on compatible configurations. Cached state is useful when many requests share the same document or instruction block.
What you need
Plan how prompts will be constructed by the client. If several users share an instruction prefix, placement of per-user content determines whether those requests can reuse it.
What to try first
Benchmark repeated-prefix and unique-prefix workloads separately. Keep the model and decoding settings fixed and compare cache behavior and tail latency under concurrent traffic.
Log hit behavior after a service restart. A cold prefix cache has different capacity and response times from one exercised by an identical benchmark several times.
Where to look for failures
Prefix caching helps only when requests share tokens in the relevant order. Changing prompt templates or putting unique text before common instructions can sharply reduce reuse.
Related Franklin pages
Compare throughput with the same model family, context lengths and output limits where possible. A runtime optimized for shared GPU traffic faces different constraints from one tuned for a single device.
- Run and quantize compatible language models for GPU inference with the EXL3 format. ExLlamaV3 for EXL3 quantization
- Deploy supported language and vision-language checkpoints with optimized inference backends. LMDeploy for quantized model serving
For the concepts behind hosting model endpoints, see Franklin’s glossary. Model serving
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗