AI GitHub
Ray for distributed training and serving
Scale Python ML work across workers while keeping resource needs explicit.
Key Takeaways
- Ray schedules remote functions and stateful actors.
- Describe each worker's CPU and GPU requirements when it is defined.
- Turn one expensive preprocessing step into a remote task and run it with a limited worker pool.
- Serializing large values and moving them between workers can outweigh parallelism.
How it works
Ray schedules remote functions and stateful actors. Ray Data, Train and Serve build on those primitives for data processing, model training and serving deployments.
What you need
Describe each worker's CPU and GPU requirements when it is defined. A task that works locally can contend for resources once several copies run concurrently.
What to try first
Turn one expensive preprocessing step into a remote task and run it with a limited worker pool. Observe object transfers and memory pressure before expanding to a cluster.
Kill a worker in a disposable run and observe recovery. This distinguishes a retried computation from a side effect that might be performed twice.
Where to look for failures
Serializing large values and moving them between workers can outweigh parallelism. Check locality, object store use and per-worker dependencies with a small representative job.
Related Franklin pages
The execution environment matters as much as the Python code. Compare startup, dependency packaging and worker recovery before changing where a compute job runs.
- Study runnable patterns for GPU inference, training and jobs on Modal. Modal Examples for remote GPU jobs
- Coordinate model training, rollout generation and related workloads across accelerators. OpenRLHF for distributed preference and RL training
For the concepts behind hosting model endpoints, see Franklin’s glossary. Model serving
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗