LMDeploy for quantized model serving

How it works

LMDeploy combines conversion and inference tools with serving interfaces. Backend choice governs model support and which optimization options are available for a deployment.

What you need

Choose the backend that supports your checkpoint and requested deployment features. A vision-language experiment also needs the correct processor and image formatting in the client.

What to try first

Serve one supported checkpoint, then compare a short text request and a representative long prompt. For a vision-language model, include images of the sizes clients send in practice.

Save the tokenized text or image shape seen by the backend. If a candidate is slower, this helps distinguish more expensive preprocessing from slower inference.

Where to look for failures

Check compatibility by model architecture and backend before committing to a quantization workflow. A successful load gives little information about image preprocessing or the maximum safe request size.

Related Franklin pages

Compare throughput with the same model family, context lengths and output limits where possible. A runtime optimized for shared GPU traffic faces different constraints from one tuned for a single device.

• Serve open-weight language models to multiple clients while managing attention-cache memory. vLLM for shared GPU inference
• Serve language model workloads that reuse instructions or context across requests. SGLang for repeated-prefix inference

For the concepts behind hosting model endpoints, see Franklin’s glossary. Model serving

Project documentation

• Current project README
• README at the reviewed revision