AI GitHub
llama.cpp for GGUF inference
Run quantized language models with a native CLI or HTTP server.
Key Takeaways
- The C and C++ runtime loads GGUF files and can offload work to supported GPU backends.
- Start with a GGUF file whose architecture the runtime supports.
- Choose one model and compare two GGUF quantizations with the same prompt and context settings.
- A GGUF conversion preserves a specific model configuration.
How it works
The C and C++ runtime loads GGUF files and can offload work to supported GPU backends. The server exposes model inference to client applications while command-line tools let you test the runtime in isolation.
What you need
Start with a GGUF file whose architecture the runtime supports. Match the executable or build flags to the CPU or accelerator in the device where it will run.
What to try first
Choose one model and compare two GGUF quantizations with the same prompt and context settings. Record load time, peak memory and a sample of outputs so the smaller file has a quality baseline.
Try a short completion without a chat wrapper and a chat request with the checkpoint's template. That isolates template problems from loading or decoding problems.
Where to look for failures
A GGUF conversion preserves a specific model configuration. Chat template, architecture support and GPU build options can change the behavior of the same file across installations.
Related Franklin pages
Separate the model runtime from the user interface. A local endpoint makes it possible to keep the same client while comparing model loading, chat templates and device support.
- Serve local models to applications that use familiar hosted-model API shapes. LocalAI as a self-hosted model API
- Inspect published small-model checkpoints and the recipes used to build them. SmolLM for exploring compact language models
For more on how model size affects deployment decisions, read Franklin’s guide. Small language models
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗