AI GitHub
LiteLLM for model routing and gateways
Use a consistent client interface or proxy across multiple model providers.
Key Takeaways
- LiteLLM translates requests for different providers and can expose them through a shared gateway.
- List the providers and capabilities the application needs first.
- Route the same chat request to two configured providers, then simulate one rate limit in a test environment.
- Similar API shapes hide provider differences in tools, streaming and token accounting.
How it works
LiteLLM translates requests for different providers and can expose them through a shared gateway. Routing and deployment configuration control how clients select backends.
What you need
List the providers and capabilities the application needs first. A gateway can unify authentication and routing even when some endpoints still require provider-specific handling.
What to try first
Route the same chat request to two configured providers, then simulate one rate limit in a test environment. Check the failure path and the output fields consumers depend on.
Track which provider handled the failed call and its retry. Otherwise the application can report one provider while the request was answered by another.
Where to look for failures
Similar API shapes hide provider differences in tools, streaming and token accounting. Confirm feature parity on every routed backend before using automatic fallback.
Related Franklin pages
A gateway tells you where a request went, while instrumentation shows what happened around it. Keep IDs across layers so provider errors can be tied to the original application request.
- Inspect application traces and model calls with OpenTelemetry-based instrumentation. Logfire for tracing Python AI applications
- Collect traces around model calls, prompts and application steps. Langfuse for tracing and evaluating LLM applications
Browse Franklin for concrete examples of tools and services exposed through MCP. Franklin’s MCP directory
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗