📄️ Quick Start
Quick start CLI, Config, Docker
📄️ Getting Started - E2E Tutorial
End-to-End tutorial for LiteLLM Proxy to:
📄️ 🐳 Docker, Deployment
You can find the Dockerfile to build litellm proxy here
📄️ ⚡ Best Practices for Production
1. Use this config.yaml
🗃️ Architecture
1 items
🔗 📖 All Endpoints (Swagger)
📄️ ✨ Enterprise Features
To get a license, get in touch with us here
📄️ 💡 Migrating from OpenAI (Langchain, OpenAI SDK, LlamaIndex, Instructor, Curl)
LiteLLM Proxy is OpenAI-Compatible, and supports:
📄️ 🎉 Demo App
Here is a demo of the proxy. To log in pass in:
📄️ Proxy Config.yaml
Set model list, apibase, apikey, temperature & proxy server settings (master-key) on the config.yaml.
📄️ 🔥 Load Balancing, Fallbacks, Retries, Timeouts
- Quick Start load balancing
🗃️ Use with Provider SDKs
6 items
📄️ 💸 Spend Tracking
Track spend for keys, users, and teams across 100+ LLMs.
📄️ Custom LLM Pricing
Use this to register custom pricing for models.
📄️ 🔑 Virtual Keys
Track Spend, and control model access via virtual keys for the proxy
🗃️ Admin UI
2 items
🗃️ 🪢 Logging, Alerting, Metrics
6 items
🗃️ 🛡️ [Beta] Guardrails
7 items
🗃️ Secret Manager - storing LLM API Keys
2 items
📄️ Tag Based Routing
Route requests based on tags.
📄️ 💰 Budgets, Rate Limits
Requirements:
📄️ 💰 Setting Team Budgets
Track spend, set budgets for your Internal Team
📄️ 🙋♂️ Customers
Track spend, set budgets for your customers.
📄️ Billing
Bill internal teams, external customers for their usage
📄️ [BETA] JWT-based Auth
Use JWT's to auth admins / projects into the proxy.
📄️ Oauth 2.0 Authentication
Use this if you want to use an Oauth2.0 token to make /chat, /embeddings requests to the LiteLLM Proxy
📄️ Caching
Cache LLM Responses
📄️ ➡️ Create Pass Through Endpoints
Add pass through routes to LiteLLM Proxy
📄️ Email Notifications
Send an Email to your users when:
📄️ Attribute Management changes to Users
Call management endpoints on behalf of a user. (Useful when connecting proxy to your development platform).
📄️ 👥 Team-based Routing
Routing
📄️ Region-based Routing
Route specific customers to eu-only models.
🗃️ Extra Load Balancing
1 items
📄️ Model Management
Add new models + Get model info without restarting proxy.
📄️ Health Checks
Use this to health check all LLMs defined in your config.yaml
📄️ Debugging
2 levels of debugging supported.
📄️ Modify / Reject Incoming Requests
- Modify data before making llm api calls on proxy
📄️ Post-Call Rules
Use this to fail a request based on the output of an llm api call.
📄️ CLI Arguments
Cli arguments, --host, --port, --num_workers