AI GitHub
ONNX Runtime for portable inference
Execute exported ONNX models with execution providers suited to the target device.
Key Takeaways
- An inference session loads an ONNX graph and assigns supported operations to execution providers.
- An ONNX file declares graph inputs and outputs.
- Export one model, compare predictions against its training framework and test representative dynamic shapes.
- A session can use CPU for unsupported operations depending on its configuration.
How it works
An inference session loads an ONNX graph and assigns supported operations to execution providers. Operator support and graph partitioning determine which portions can use acceleration.
What you need
An ONNX file declares graph inputs and outputs. The application still needs to supply preprocessing, postprocessing and a runtime build containing the desired execution provider.
What to try first
Export one model, compare predictions against its training framework and test representative dynamic shapes. Profile the exact provider settings you expect to ship.
Profile a representative mobile or server workload with a warm session. Session creation and graph optimization can distort measurements from a single inference.
Where to look for failures
A session can use CPU for unsupported operations depending on its configuration. Inspect provider placement and test preprocessing parity before attributing a speed or accuracy change to the model.
Related Franklin pages
Use the same input tensors and output tolerance for export tests. An inference runtime may support the model format while the preferred device delegate covers only part of its graph.
- Take an exported PyTorch program into a lightweight runtime for edge devices. ExecuTorch for edge deployment of PyTorch models
- Look for suspicious operations in supported serialized model artifacts before loading them. ModelScan for inspecting serialized model files
For the constraints that shape mobile and embedded model deployment, see Franklin’s guide. On-device AI
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗