LiteLLM
Call any LLM provider using the OpenAI format. Proxy server for load balancing and cost tracking.
LiteLLM is an open-source AI gateway that provides a unified API interface for accessing over 100 LLM providers — including OpenAI, Azure, Anthropic, Google Gemini, AWS Bedrock, and others — all through a single, OpenAI-compatible format. Built by BerriAI and backed by Y Combinator, LiteLLM has grown to over 40,000 GitHub stars and serves more than 1 billion requests, with 240 million+ Docker pulls.
At its core, LiteLLM solves a fundamental infrastructure problem for engineering teams: managing multiple LLM providers without rewriting integration code for each one. By standardizing inputs and outputs to match the OpenAI API format, developers can swap or combine models from different providers without changing application logic. This is particularly valuable when a new model launches — teams can add it without hours of integration work.
LiteLLM operates primarily as a proxy server that sits between applications and LLM providers. Platform teams deploy it (on-prem or cloud) and use it to govern model access across their organization. The proxy handles routing, load balancing, rate limiting, fallbacks, and cost tracking — concerns that would otherwise require custom tooling.
Cost visibility is one of LiteLLM's strongest features. It tracks spend at the level of individual API keys, users, teams, or organizations, and can log spend data to S3, GCS, or other storage backends. This makes it practical for companies that need to allocate LLM costs across internal teams or bill customers for usage.
Compared to alternatives like Portkey, OpenRouter, or cloud-native gateways (e.g., AWS AI Gateway), LiteLLM differentiates on self-hosted deployment, open-source transparency, and depth of provider coverage. OpenRouter is a hosted service with less flexibility for on-prem requirements, while Portkey focuses more on observability and prompt management. LiteLLM covers the broadest set of providers and gives engineering teams full control over their infrastructure.
LiteLLM integrates with observability platforms including Langfuse, Arize Phoenix, Langsmith, and OpenTelemetry, making it compatible with existing MLOps stacks. It also includes guardrails support, prompt management, and pass-through endpoints for provider-specific features.
Used in production by companies including Netflix, Lemonade, and RocketMoney, LiteLLM is well-suited for any organization that needs to standardize LLM access across multiple teams, control costs, or maintain the flexibility to change providers over time.
Key Features
- Unified OpenAI-compatible API across 100+ LLM providers including OpenAI, Azure, Anthropic, Gemini, and AWS Bedrock
- Proxy server with load balancing, RPM/TPM rate limiting, and automatic LLM fallbacks
- Spend tracking at key, user, team, and organization level with support for logging to S3/GCS
- Virtual keys and budget controls to manage and cap LLM usage per team or project
- LLM guardrails for content safety and policy enforcement
- Observability integrations with Langfuse, Arize Phoenix, Langsmith, and OpenTelemetry
- Prompt management and tag-based spend tracking
- Self-hosted deployment via Docker with on-prem support
Pros & Cons
Pros
- Fully open-source with a large community (40K+ GitHub stars, 1,000+ contributors)
- Broadest provider coverage of any comparable gateway — 100+ LLMs in one interface
- Strong cost tracking and budget controls suited for multi-team enterprise environments
- Eliminates per-provider integration work when adopting new models
- Flexible deployment: self-hosted, on-prem, or cloud
Cons
- Self-hosting requires infrastructure investment and operational maintenance
- Enterprise features (SSO, JWT Auth, audit logs, custom SLAs) are behind a paid tier
- Configuration complexity increases with scale — not a simple drop-in for small projects
- Uptime is self-managed when running on-prem, with no SLA guarantee on the free tier
Pricing
LiteLLM offers a free open-source tier with full access to core features including 100+ LLM integrations, virtual keys, load balancing, and guardrails. An Enterprise plan is available for organizations requiring SSO, JWT Auth, audit logs, and custom SLAs — pricing is quote-based via their sales team.
Who Is This For?
LiteLLM is best suited for platform and infrastructure engineering teams at mid-size to large organizations that need to provide governed LLM access to multiple developers or internal teams. It excels in environments where cost accountability, provider flexibility, and standardized API access are critical — particularly when teams want to avoid vendor lock-in or need to deploy on-premises.
Similar to LiteLLM