AI GitHub
LiteRT for Android and edge inference
Deploy small machine-learning models into mobile and embedded applications.
Key Takeaways
- LiteRT uses converted model artifacts and device-specific acceleration paths.
- Use the deployment guide for the device family you are targeting.
- Compare an exported model against the original using a handful of real inputs, then run it on a physical device and record memory and latency.
- Model conversion, operator coverage and delegation can vary across devices.
How it works
LiteRT uses converted model artifacts and device-specific acceleration paths. The application owns input preparation and the use of outputs, such as converting a camera frame into a classifier's tensor.
What you need
Use the deployment guide for the device family you are targeting. Check whether the model's operator set fits the accelerator delegate you want to enable.
What to try first
Compare an exported model against the original using a handful of real inputs, then run it on a physical device and record memory and latency. Include at least one non-default input shape if the app accepts it.
Run the application after a fresh launch and after several warm requests. On-device users experience both, and accelerator initialization can be a meaningful part of the first result.
Where to look for failures
Model conversion, operator coverage and delegation can vary across devices. Keep a reference prediction set so changes to preprocessing or quantization can be tested independently.
Related Franklin pages
Use the same input tensors and output tolerance for export tests. An inference runtime may support the model format while the preferred device delegate covers only part of its graph.
- Execute exported ONNX models with execution providers suited to the target device. ONNX Runtime for portable inference
- Build language model runtimes for different device targets using compilation. MLC LLM for compiled language model inference
For the constraints that shape mobile and embedded model deployment, see Franklin’s guide. On-device AI
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗