MLC LLM for compiled language model inference

How it works

MLC LLM packages model weights and compiled model libraries for supported targets. Its build and deployment flow distinguishes the checkpoint from the code that runs it on a device.

What you need

Decide which device targets you will support before compiling. An application needs matching weights and model libraries, as well as the configuration used to find and load them.

What to try first

Start from a documented model and platform combination, then try a small chat workload with the same artifact on the target device. Record which library, quantization and tokenizer files belong together.

Confirm tokenizer files load from your packaged application with the network disabled. A working build on a development machine can still be missing a runtime asset.

Where to look for failures

A device build has narrower compatibility than a generic checkpoint. Recheck supported GPU APIs and model architectures when moving a working app to another operating system.

Related Franklin pages

Decide where conversion happens and how the runtime integrates with your application. Keep an inventory of all model files, compiled libraries and device-specific components needed for a clean install.

• Build a graph that combines preprocessing, inference and postprocessing across model backends. nndeploy for multi-stage edge inference pipelines
• Deploy small machine-learning models into mobile and embedded applications. LiteRT for Android and edge inference

For the constraints that shape mobile and embedded model deployment, see Franklin’s guide. On-device AI

Project documentation

• Current project README
• README at the reviewed revision