AI GitHub
LocalAI as a self-hosted model API
Serve local models to applications that use familiar hosted-model API shapes.
Key Takeaways
- LocalAI sits between clients and multiple inference backends.
- Select a model and backend together, then map any existing application features to the corresponding LocalAI endpoints.
- Move one existing client request to a test server, then compare errors, streaming and response shape with the old endpoint.
- Endpoint compatibility varies by model and backend.
How it works
LocalAI sits between clients and multiple inference backends. Model configuration determines the backend and parameters, so a text model and a speech model can share an API host while using different runtimes.
What you need
Select a model and backend together, then map any existing application features to the corresponding LocalAI endpoints. Different kinds of models may need different dependencies.
What to try first
Move one existing client request to a test server, then compare errors, streaming and response shape with the old endpoint. Start with one backend so you can separate API setup from model setup.
Include a tool-call request if the old client depends on tools. Check that arguments arrive in the same shape and that an unsupported request fails clearly.
Where to look for failures
Endpoint compatibility varies by model and backend. Validate the request fields your application actually sends, and put authentication and request limits in front of a network-accessible instance.
Related Franklin pages
Separate the model runtime from the user interface. A local endpoint makes it possible to keep the same client while comparing model loading, chat templates and device support.
- Use a bundled local interface and GGUF runtime for story generation and chat. KoboldCpp for local interactive writing
- Pull, run and serve language models behind a local API. Ollama for running local language models
For more on how model size affects deployment decisions, read Franklin’s guide. Small language models
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗