AI GitHub
Candle for embedding inference in Rust
Build native Rust programs around tensors and model implementations.
Key Takeaways
- Candle supplies tensor operations, model building blocks and example implementations.
- Read the Rust example for your chosen model and backend before adding it as a dependency.
- Choose an example model close to your task and run it on a known input.
- Each model implementation has its own assumptions about weight names and input preparation.
How it works
Candle supplies tensor operations, model building blocks and example implementations. An application can bring preprocessing, weight loading and inference into the same Rust binary or service.
What you need
Read the Rust example for your chosen model and backend before adding it as a dependency. It shows the input tensors, weight formats and build features the application needs.
What to try first
Choose an example model close to your task and run it on a known input. Verify tokenizer and tensor shapes before replacing the example checkpoint or changing the accelerator backend.
Use a clean Rust build and record which backend features were enabled. That makes binary size and startup comparisons meaningful across target machines.
Where to look for failures
Each model implementation has its own assumptions about weight names and input preparation. Confirm support for the architecture rather than treating any checkpoint as a drop-in file.
Related Franklin pages
Decide whether you need array programming, a language-model workflow or an embeddable library. That choice determines how much model and input handling your application must implement.
- Work with array operations, automatic differentiation and model execution on Apple silicon. MLX for machine learning on Apple silicon
- Generate with, quantize and adapt supported language models using MLX. MLX LM for language models on Macs
For the constraints that shape mobile and embedded model deployment, see Franklin’s guide. On-device AI
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗