AI GitHub
Triton Inference Server for model repositories
Serve models from a versioned repository using backends appropriate to each model format.
Key Takeaways
- Triton associates a model directory with configuration, versions and a backend.
- Start with a backend-supported model export, a model configuration and a versioned directory.
- Deploy one exported model in a versioned directory and compare single-item predictions with a concurrent client run.
- A model repository needs explicit shapes, data types and supported backend combinations.
How it works
Triton associates a model directory with configuration, versions and a backend. Its batching and scheduling configuration governs how requests arrive at each loaded model.
What you need
Start with a backend-supported model export, a model configuration and a versioned directory. Verify the client uses the exact names and data types the model declares.
What to try first
Deploy one exported model in a versioned directory and compare single-item predictions with a concurrent client run. Use the server's metrics to separate queuing from model execution.
Send one request that deliberately fails shape validation and one with the wrong data type. Client error handling matters when several model versions share one server.
Where to look for failures
A model repository needs explicit shapes, data types and supported backend combinations. Image preprocessing and tokenization must agree with the assumptions used when exporting the model.
Related Franklin pages
Decide which layer needs to own model packaging, request batching and deployment. These projects can complement one another when packaging and cluster orchestration live in separate systems.
- Define Python inference services with explicit endpoints and runtime dependencies. BentoML for packaging model APIs
- Declare inference services on Kubernetes with model serving runtimes and lifecycle controls. KServe for Kubernetes model deployments
For the concepts behind hosting model endpoints, see Franklin’s glossary. Model serving
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗