AI GitHub
KServe for Kubernetes model deployments
Declare inference services on Kubernetes with model serving runtimes and lifecycle controls.
Key Takeaways
- KServe uses Kubernetes resources to describe inference endpoints and the runtime that loads each model.
- Prepare a Kubernetes test namespace with a serving runtime and access to model storage.
- Deploy one small model to a test namespace, verify readiness and send concurrent requests.
- A working local inference container is only one dependency.
How it works
KServe uses Kubernetes resources to describe inference endpoints and the runtime that loads each model. Scaling and networking behavior depend on the installation and deployment mode.
What you need
Prepare a Kubernetes test namespace with a serving runtime and access to model storage. Choose the deployment mode before depending on its scaling behavior.
What to try first
Deploy one small model to a test namespace, verify readiness and send concurrent requests. Observe startup and restart time before attaching a larger model or autoscaling rule.
Roll out a second version in the test namespace and observe traffic during model loading. Readiness checks should reflect whether inference can actually answer.
Where to look for failures
A working local inference container is only one dependency. Confirm cluster networking, storage credentials and accelerator scheduling before troubleshooting model code.
Related Franklin pages
Decide which layer needs to own model packaging, request batching and deployment. These projects can complement one another when packaging and cluster orchestration live in separate systems.
- Serve models from a versioned repository using backends appropriate to each model format. Triton Inference Server for model repositories
- Scale Python ML work across workers while keeping resource needs explicit. Ray for distributed training and serving
For the concepts behind hosting model endpoints, see Franklin’s glossary. Model serving
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗