AI GitHub
BentoML for packaging model APIs
Define Python inference services with explicit endpoints and runtime dependencies.
Key Takeaways
- BentoML describes a service and its inference methods, then packages the service for deployment.
- You need an inference callable, its input and output contract and all required artifacts.
- Wrap one prediction function and test the packaged service with the same inputs as the original script.
- A reproducible package still needs pinned artifacts and a suitable deployment target.
How it works
BentoML describes a service and its inference methods, then packages the service for deployment. Model preprocessing, batching and outputs can sit alongside the model code in the same service.
What you need
You need an inference callable, its input and output contract and all required artifacts. Keep dependencies and model files explicit so the package can be rebuilt.
What to try first
Wrap one prediction function and test the packaged service with the same inputs as the original script. Include an invalid request to see how the contract behaves for callers.
Check a package built on a clean machine. A successful development-server run can depend on cached artifacts or dependencies absent in a packaged deployment.
Where to look for failures
A reproducible package still needs pinned artifacts and a suitable deployment target. Benchmark preprocessing and serialization separately when model inference is only one part of request time.
Related Franklin pages
Decide which layer needs to own model packaging, request batching and deployment. These projects can complement one another when packaging and cluster orchestration live in separate systems.
- Declare inference services on Kubernetes with model serving runtimes and lifecycle controls. KServe for Kubernetes model deployments
- Serve models from a versioned repository using backends appropriate to each model format. Triton Inference Server for model repositories
For the concepts behind hosting model endpoints, see Franklin’s glossary. Model serving
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗