AI GitHub
nndeploy for multi-stage edge inference pipelines
Build a graph that combines preprocessing, inference and postprocessing across model backends.
Key Takeaways
- The graph describes stages and their data dependencies.
- Map the pipeline stages before optimizing it.
- Reproduce one example with a known input image, then instrument per-stage latency and memory.
- Every backend needs compatible models and tensor conventions.
How it works
The graph describes stages and their data dependencies. Inference engines can be selected within that larger pipeline, which is useful when image processing and postprocessing are substantial parts of runtime.
What you need
Map the pipeline stages before optimizing it. If the application already performs preprocessing, verify the graph receives the tensors that stage expects instead of duplicating conversion work.
What to try first
Reproduce one example with a known input image, then instrument per-stage latency and memory. Swap the inference backend only after the reference pipeline produces the expected result.
Use a synthetic frame with distinctive channel values. It can expose channel-order changes or unnecessary copies more clearly than looking at a plausible prediction.
Where to look for failures
Every backend needs compatible models and tensor conventions. Pay attention to where data changes device or layout, since those conversions can dominate a small model's runtime.
Related Franklin pages
Decide where conversion happens and how the runtime integrates with your application. Keep an inventory of all model files, compiled libraries and device-specific components needed for a clean install.
- Deploy small machine-learning models into mobile and embedded applications. LiteRT for Android and edge inference
- Build language model runtimes for different device targets using compilation. MLC LLM for compiled language model inference
For the constraints that shape mobile and embedded model deployment, see Franklin’s guide. On-device AI
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗