What it is
LiteLLM is a non-AI infrastructure tool that sits between internal teams and 140+ LLM providers. It translates every provider's API into one unified format, so code written for OpenAI works unchanged against Anthropic, Bedrock, Cohere, or any other supported vendor. The recurring user is the DevOps engineer or internal-tools lead managing LLM access for a multi-team engineering org. Self-hosted deployment keeps API keys on internal infrastructure rather than routing through a third party.
At a glance
LiteLLM unifies 140+ LLM providers behind one API, eliminating the need to write provider-specific code or manually switch between OpenAI, Anthropic, Bedrock, and others. It automates fallback routing, load balancing, and per-team budget enforcement — a real workflow transformation for platform teams managing multi-vendor LLM access, not just a prompt wrapper.
Strong evidenceQuality score
LiteLLM Unified multi-provider LLM gateway with routing, fallbacks, and spend controls, but team-scale governance adds complexity.
This score is our editorial judgment, computed automatically from the sources, weights, and dates shown above. It reflects the data we could verify as of November 16, 2026, not a guarantee or statement of fact about LiteLLM. Third-party ratings and quotes belong to their original platforms and authors. Thin data lowers our confidence label, and we say so instead of guessing. Work on LiteLLM? Dispute any datapoint and we will review it, publish your response, and correct verified errors.
Plans
Open-source, MIT-licensed, self-hosted with all core features free forever.
Community feedback
Ratings and quoted comments below are aggregated from third-party sources and reflect those users' views, not SearchTools.ai's.
themes inside the Sentiment pillar — not score ingredients
“Throwaway for obvious reasons, work in finance (quant research), google gave a few of us access to the new model - me and a couple engineers on our team. almost didn't even get to try it because we had to wait for litellm to push some vertex ai fix before the routing even worked 10 days in and i don't really know how to say this without sounding like a shill but this thing is different. Yeah the 1m context window, whatever, we've seen that number before. but it actually works now?? i've been dum”
“Nobody at my prop firm would touch the free usage Google gave us because it was so dog shit. Big if true.”
“LiteLLM looks simple but solves a real pain point: managing multiple LLM providers without exposing API keys to anyone. Looking into using it as a layer between my clients and providers, the per-team spending controls alone make it worth it.”
“Used it 18 months, 4 providers. Provider switching and fallback routing genuinely work well. Latency overhead is negligible. Streaming edge cases across providers will bite you eventually but it’s manageable. The elephant in the room: LiteLLM got supply-chain compromised last week. Two PyPI versions shipped a credential stealer — exfiltrated cloud keys , SSH, K8s secrets on every Python startup. Caught in 3 hours only because the attacker’s code accidentally fork-bombed the discoverer’s mach”
“Last week I made a setup which is working great and saving close to 50% in token costs. A lot of my usage is coding and basic Q&A/fact-checking/summaries/todos/MCP questions — which can be handled with a decent tool-calling model. I heavily use LLM Wiki and Todoist to manage stuff.Every query gets validated by JEV to see if it can be answered by a simpler model, and based on the need, it selects the appropriate model. \------------------------------ Here is my setup: Step 1:Set up LiteLLM as the”
“the payloads run on every python start on the machine and not just when you import litellm. Your IDE, pip, CI jobs everything python on that host is affected. It steals ssh keys, aws credentials, browser passwords, shell history and AI coding tool configs. The backdoor runs as a daemon and executes whatever code the attacker’s server returns every hour. Pin to 1.106.2. complete breakdown: safedep.io/litellm-pypi-compromise-credential-stealer-backdoor”
“Used it 18 months, 4 providers. Provider switching and fallback routing genuinely work well. Latency overhead is negligible. Streaming edge cases across providers will bite you eventually but it’s manageable. The elephant in the room: LiteLLM got supply-chain compromised last week. Two PyPI versions shipped a credential stealer — exfiltrated cloud keys , SSH, K8s secrets on every Python startup. Caught in 3 hours only because the attacker’s code accidentally fork-bombed the discoverer’s mach”
“#Alert So it's not only PHP that was vulnerable today (patch yours NOW!) but also, one of the prominent and widely used LLM Gateway - LiteLLM AI Gateway was (again) backdoored. Not sure who discovered the backdoor, but it's serious. Here's the information: Version: LiteLLM 1.107.0 (PyPI and the official Docker images). One line was added to the build config: `data = "cook"` under [tool.maturin]. That directive makes maturin bundle a directory named cook/ into the wheel's data section, and pip in”
“Using litellm internally and for production. Stable enough for our current volume of 66 mil requests a week.”
“the payloads run on every python start on the machine and not just when you import litellm. Your IDE, pip, CI jobs everything python on that host is affected. It steals ssh keys, aws credentials, browser passwords, shell history and AI coding tool configs. The backdoor runs as a daemon and executes whatever code the attacker’s server returns every hour. Pin to 1.106.2. complete breakdown: safedep.io/litellm-pypi-compromise-credential-stealer-backdoor”
“> The upstream pyproject.toml at tag v1.107.0 has no data = "cook" line and no cook/ folder. Someone changed the build outside the public repository. So it's an intentional action by a maintainer (i.e. not a question of overlooking something snuck into an OSS contribution).”
“After the first breach I configured uv with 7d quarantine”
“Used it 18 months, 4 providers. Provider switching and fallback routing genuinely work well. Latency overhead is negligible. Streaming edge cases across providers will bite you eventually but it’s manageable. The elephant in the room: LiteLLM got supply-chain compromised last week. Two PyPI versions shipped a credential stealer — exfiltrated cloud keys , SSH, K8s secrets on every Python startup. Caught in 3 hours only because the attacker’s code accidentally fork-bombed the discoverer’s mach”
“I am looking to migrate away from litellm, so far I haven't found any open gateway that support model aliases based on sampling parameters e.g. I use this kind of config define different versions of Qwen model. does GoModel support this kind of config ? - model_name: Qwen3.8-27B-thinking-low litellm_params: model: hosted_vllm/Qwen3.8-27B api_base: api_key: temperature: 1.0 top_p: 0.95 top_k: 20 min_p: 0.0 presence_penalty: 0.0 repetit”
“The part that bit us doing this was that "budget" and "usage record" want to live in different places. LiteLLM owns enforcement, but if you want a single number across both entry points, the record has to be written somewhere neither tool owns — otherwise you're reconciling two ledgers that disagree the moment one of them retries. Cheapest version that worked: let LiteLLM keep enforcing its own budget, and have both paths also emit a usage event (user id, model, tokens, cost) to one place you ”
“exploring options for giving our team access to models across multiple vendors, OpenAI, Anthropic, Bedrock, and a few others. purely internal use, playgrounds, POCs, and demos. nothing production facing. mainly need access control through synthetic keys, some visibility into usage, and something that doesn't require a dedicated engineer to maintain. looked at a few options so far: LiteLLM — seems like the most straightforward starting point, good community, easy to get going TrueFoundry — keeps ”
Watch & learn

Bifrost vs LiteLLM (2026): Which AI Gateway Is Actually Faster?
AIQualityNerd13 days ago

Why LiteLLM is rewriting its AI gateway in Rust
LiteLLMAIGateway16 days ago
Capabilities
Provides utilities that help programmers build, test, and ship software faster
The honest take
Distinct themes surfaced across user reviews — each grounded in real review text, ranked by how often it comes up.
Questions
LiteLLM is a self-hosted AI gateway that connects to 140+ LLM providers through a single OpenAI-compatible API. It enables platform teams to provide organization-wide access to multiple AI models while maintaining cost control, security, and governance through features like per-team budgets, virtual keys, and guardrails.
Yes, the core LiteLLM gateway is MIT-licensed open source and free to self-host forever, including features like virtual keys, budgets, teams, load balancing, and guardrails. An Enterprise tier with additional features like SSO, SCIM, JWT auth, and 24/7 support is available with custom pricing and a 30-day trial.
LiteLLM supports 140+ LLM providers and 1,892 specific models. This includes major providers like OpenAI, Anthropic, AWS Bedrock, Google Vertex AI, and Azure, all accessible through a single unified API interface.
Yes, LiteLLM allows you to set hard budget limits at the key, team, or organization level with daily or monthly caps. The system automatically tracks every request and attributes costs to the appropriate entity, preventing overspending and providing granular cost visibility.
LiteLLM is self-hosted in your own infrastructure, meaning prompts and API keys never leave your environment. It supports deployment in air-gapped environments with no external connectivity, includes security guardrails like PII masking and prompt injection detection, and achieves SOC 2 Type 2 and ISO 27001 compliance for Enterprise customers.
Independent benchmarks show LiteLLM adds only 0.66 milliseconds of latency at the 99th percentile, which is 3.5 times lower than competing gateways. The Rust-based gateway handles 2,800+ requests per second at approximately 21% CPU utilization and consumes roughly 22 MB of memory at rest.
LiteLLM provides intelligent routing including load balancing across providers and regions, lowest-cost routing to find the cheapest deployment, and Auto Routing that automatically sends simple prompts to cheaper models while directing complex requests to more capable models. It also includes response and semantic caching using Redis, S3, or GCS to prevent duplicate charges.
LiteLLM can be deployed in standard cloud environments (AWS, GCP, Azure), on-premises data centers, and fully air-gapped environments. It includes official Helm charts and Terraform modules for deployment on your own Postgres and Redis infrastructure with Kubernetes autoscaling support.
LiteLLM includes PII masking through Presidio, prompt injection detection via Lakera, and secret detection capabilities. Enterprise customers also get audit logs on every request, secret manager integration with AWS Secrets Manager, Vault, and Azure Key Vault, plus RBAC controls by key, team, and organization.
LiteLLM generates virtual keys that can be assigned to individual developers, teams, or projects with granular controls. Each virtual key can have hard budget limits, rate limits measured in RPM and TPM, and access restricted to specific models, providing fine-grained access management across your organization.
Organizations like AT&T, Netflix, NVIDIA, Okta, and IBM use LiteLLM in production. The software has accumulated over 500 million Docker pulls and 92 million PyPI downloads monthly, demonstrating significant enterprise adoption.
More Like This