AI GitHub
Ollama for running local language models
Pull, run and serve language models behind a local API.
Key Takeaways
- Ollama manages model files and runs the inference process behind a CLI and an HTTP API.
- You need a model tag that matches your hardware and a process that can reach the local service.
- Run a small model first and submit the same prompt from the CLI and the API.
- A download that fits on disk can still exhaust RAM or VRAM at a long context length.
How it works
Ollama manages model files and runs the inference process behind a CLI and an HTTP API. A Modelfile can bind an existing model to a system prompt, template or parameter choices.
What you need
You need a model tag that matches your hardware and a process that can reach the local service. If you connect a separate UI, configure its model endpoint independently of the browser URL.
What to try first
Run a small model first and submit the same prompt from the CLI and the API. Track the time to first token on a cold start and again while the model is loaded.
Check stop behavior on a request that streams for several seconds. A client that disconnects cleanly should let you start another request without a lingering generation occupying the server.
Where to look for failures
A download that fits on disk can still exhaust RAM or VRAM at a long context length. Size the model for the device and review the weights' own license before redistribution.
Related Franklin pages
Separate the model runtime from the user interface. A local endpoint makes it possible to keep the same client while comparing model loading, chat templates and device support.
- Run quantized language models with a native CLI or HTTP server. llama.cpp for GGUF inference
- Serve local models to applications that use familiar hosted-model API shapes. LocalAI as a self-hosted model API
For more on how model size affects deployment decisions, read Franklin’s guide. Small language models
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗