40 minutes, 500,000 machines: the LiteLLM attack every AI builder should study

On March 24, at 10:39 UTC, a group calling themselves TeamPCP pushed a poisoned version of LiteLLM to PyPI. By 11:25 UTC, PyPI had quarantined it. Forty-six minutes. In that window, security researchers estimate the malicious package reached half a million machines. Mandiant confirmed over 1,000 SaaS environments were actively dealing with the fallout. A $10 billion AI startup lost 4 terabytes of data. And the scariest part? Most of the companies that downloaded it still don’t know.

If you’re building anything with AI right now, and if you’re reading this blog you probably are, this one deserves your full attention. Not because supply chain attacks are new. They’re not. But because the way this attack worked exposes something specific about how we build AI products that makes us unusually easy targets.

The attack, step by step

LiteLLM is the kind of package that sits quietly in your stack doing essential work. It’s a proxy layer that lets you call OpenAI, Anthropic, Cohere, and dozens of other LLM providers through a unified API. If you’ve ever wanted to swap between Claude and GPT-4 without rewriting your integration code, you’ve probably at least looked at it. According to Wiz, it’s present in 36% of cloud environments. It gets downloaded roughly 3.4 million times a day from PyPI.

The attackers didn’t go after LiteLLM directly. They went after Trivy, a container security scanner that LiteLLM used in its CI/CD pipeline. A compromised GitHub Action for Trivy let TeamPCP steal the PYPI_PUBLISH token from LiteLLM’s GitHub Actions runner. That token is the key to publishing new versions of the package. Once they had it, publishing a backdoored version was trivial.

Think about that for a second. The security scanner itself was the attack vector. The tool you run to check if your code is safe was the thing that made it unsafe. There’s a grim irony there that would be funny if it hadn’t resulted in one of the largest credential theft operations of the year.

What the malicious code actually did

TeamPCP published two versions: 1.82.7 at 10:39 UTC and 1.82.8 thirteen minutes later. The second version is the one that should keep you up at night.

Version 1.82.7 was bad enough. It dropped a double-base64-encoded payload that executed whenever someone ran litellm --proxy or imported litellm.proxy.proxy_server. Standard credential stealer: SSH keys, cloud tokens, .env files, Kubernetes secrets, crypto wallet files. All of it exfiltrated to litellm.cloud, a domain the attackers registered specifically because it looks like it could be official. The real domain is litellm.ai.

Version 1.82.8 added something nastier. It abused Python’s .pth file mechanism, a feature most Python developers have never heard of. Files ending in .pth that are placed in Python’s site-packages directory execute code automatically whenever Python starts up. Not when you import a package. Not when you use a specific feature. Every single time Python runs on that machine. The malicious litellm_init.pth file meant the credential harvester fired whether you were running LiteLLM, Django, a Jupyter notebook, or a random script. If Python started, the malware ran.

After collecting credentials, the payload attempted lateral movement across Kubernetes clusters by deploying privileged pods to every node it could reach. Then it installed a persistent systemd backdoor that polled for additional binaries. Three stages: steal, spread, persist.

The Mercor fallout

Mercor is an AI recruiting and data-labeling startup that was valued at $10 billion as of early 2026. They were one of the first companies to publicly confirm they’d been hit. A group identifying as Lapsus$ claimed to have exfiltrated 4 terabytes of Mercor data and began auctioning it. The alleged haul includes candidate profiles with personally identifiable information, employer data, user accounts and credentials, video interviews, proprietary source code, API keys and secrets, and TailScale VPN configuration data.

Mercor’s official statement said they were “one of thousands” of companies affected. Their security team moved to contain the incident, and they brought in third-party forensics. But the damage was already done. When your credential-stealing malware has had time to spread across Kubernetes clusters and install persistent backdoors, “containment” is a lot harder than it sounds.

What happened to Mercor could have happened to anyone running LiteLLM in those 46 minutes. And given that the package gets 3.4 million downloads per day, thousands of CI/CD pipelines, development machines, and production servers pulled the compromised version before PyPI caught it.

Why AI stacks are sitting ducks

Supply chain attacks aren’t new. SolarWinds in 2020. Log4j in 2021. But those hit broad enterprise software. This one carved through the AI ecosystem specifically, and the reasons are worth understanding because they apply to basically every AI startup I know, including mine.

We move fast. That’s the whole point. Ship daily, iterate constantly, swap LLM providers when pricing changes. Nobody is stopping to pin every dependency to a specific hash when there’s a demo tomorrow and a launch next week. pip install --upgrade litellm and move on. That speed is what turned a 46-minute window into a catastrophe.

AI infrastructure also requires absurdly broad permissions compared to a typical web app. LiteLLM specifically sits at the center of your stack holding API keys for every LLM provider you use, database credentials, cloud tokens, Kubernetes access. A Rails app might have one database password. Your AI proxy layer has the keys to everything. When the credential stealer ran, it ate well.

Then there’s the dependency graph. LiteLLM pulls in hundreds of packages. Those packages pull in other packages. Somewhere in that tree, Trivy was getting fetched without a version pin. Nobody was auditing the full chain because nobody can. It’s too large, too volatile, and the monitoring tools haven’t matured the way they have in the JavaScript ecosystem where lock files and audit workflows are table stakes.

And the community runs on trust. We install packages because someone on Twitter recommended them, or because they showed up in a tutorial, or because they solve a problem we need solved right now. The vetting process is “does it work?” not “has anyone audited the maintainer’s ops security?” That trust is foundational to open source. It’s also the thing attackers exploit.

Your CI/CD pipeline is probably the softest target you have

One detail in this attack deserves its own section. The initial compromise came through a GitHub Action. Not a phishing email. Not a zero-day. A CI/CD component pulled without a version pin.

Go look at your own GitHub Actions workflows right now. I’m serious, open them in another tab. How many reference actions with @main or @latest instead of a pinned commit SHA? How many pull scanners from apt or pip without an exact version? Each one of those is a door you left open.

CI/CD pipelines are the most privileged environment in most organizations. Publishing credentials, deployment keys, cloud accounts, production secrets, they’re all there. And yet the YAML files that configure them get less review than a CSS change. We treat CI/CD like plumbing: set it up once, forget about it. TeamPCP didn’t need to find a bug in LiteLLM’s code. They found a gap in LiteLLM’s build process, which was a much softer target.

What this means if you’re building an AI product

If you’re a solo founder or a small team shipping AI products, you probably have the same vulnerabilities that Mercor had. I checked my own setup after reading about this, and I had unpinned actions in two GitHub workflows. It took me 20 minutes to fix. That’s the kind of gap we’re talking about here.

The answer isn’t to slow down. The answer is a handful of habits that shrink the damage when something goes wrong. Because something will go wrong.

Pin your dependencies. Actually pin them. Not just in requirements.txt. In your lock files, in your CI/CD actions, in your Dockerfiles. Use hash verification when your package manager supports it. pip install litellm==1.82.6 --require-hashes would have prevented this entirely. Yes, it’s annoying to maintain. It’s less annoying than rotating every credential in your system.

Treat CI/CD as a production system. Pin your GitHub Actions to commit SHAs, not branch names. Audit what secrets are accessible to which workflows. Use OpenID Connect (OIDC) for cloud provider auth instead of storing long-lived tokens. Review your workflow files with the same rigor you’d review a PR that touches authentication code.

Scope your secrets. If your LLM proxy has access to every API key, every database credential, and every cloud token, then compromising the proxy means compromising everything. Separate service accounts. Minimum permissions per component. Secrets in a vault, not in env vars that any process can read.

Monitor for .pth files. The .pth file trick used in this attack is a legitimate Python feature that most developers have never touched. Set up monitoring for new .pth files appearing in site-packages. Also monitor for unexpected systemd service installations and unusual outbound connections.

Write a credential rotation plan before you need one. If you got hit, the clock started ticking the moment the malware executed. A documented process for rotating credentials across all your services, that you’ve actually run through once, is the difference between a bad week and a dead company. Go write it today. One page is enough.

Use a private package registry. Cloudsmith, JFrog Artifactory, or even a basic PyPI mirror with approval policies can stop malicious packages before they reach your systems. Pulling straight from public PyPI in production is basically downloading random executables from the internet. We stopped doing that with desktop apps years ago.

The bigger picture

What happened with LiteLLM is going to happen again. Probably soon. The AI ecosystem is the most attractive target in software right now because it combines high-value credentials, rapid adoption of new tools, low maturity of security practices, and a community that optimizes for speed over safety.

GitHub published a report in March 2026 covering security results across 67 open source AI projects. The findings were not great. Most lacked basic protections like branch protection rules, code signing, or dependency review workflows. Google released guidance on AI supply chain security that basically said: treat your models and your AI dependencies with the same rigor you’d treat any other third-party code. Which sounds obvious until you realize almost nobody is doing it.

The LiteLLM attack also forces a harder conversation about the monoculture problem. When 36% of cloud environments depend on a single package for LLM routing, that package becomes critical infrastructure. But it’s maintained by a startup (BerriAI) with a small team, not by a company with the resources of Google or Microsoft. We’ve built a system where the security of the entire AI ecosystem depends partly on whether a small team remembered to pin their Trivy version. That’s not sustainable.

I don’t have a clean solution to the monoculture problem. Open source is what makes it possible for small teams to build AI products in the first place. But we need to be honest about the tradeoffs. When you use open source as core infrastructure, you’re outsourcing part of your security posture to someone else’s ops practices. That can be fine. You just need to know you’re doing it and plan accordingly.

If you were affected

Check your pip install logs and lock files for litellm versions 1.82.7 or 1.82.8, installed between March 24 10:39 UTC and 11:25 UTC. Search your Python site-packages directories for litellm_init.pth. Check for outbound connections to litellm.cloud. Review your systemd services for unexpected entries. If you find any indicators, assume full compromise: rotate every credential, revoke every token, and bring in forensics help if you can afford it.

If you weren’t affected this time, don’t assume you’re safe. Go pin your dependencies. Audit your CI/CD pipelines. Scope your secrets. Build your rotation playbook. The next 46-minute window might involve a package in your stack, and you won’t get a second chance to prepare.

Frequently asked questions

What was the LiteLLM supply chain attack?

On March 24, 2026, a threat group called TeamPCP compromised LiteLLM’s PyPI publishing credentials through a poisoned Trivy GitHub Action. They published two malicious package versions (1.82.7 and 1.82.8) containing credential-stealing malware. The packages were live for approximately 46 minutes before PyPI quarantined them. In that window, an estimated 500,000 machines downloaded the compromised code.

How many companies were affected?

Mercor, the $10 billion AI recruiting startup, confirmed it was “one of thousands” of companies affected. Mandiant’s CTO Charles Carmakal confirmed over 1,000 impacted SaaS environments. Security researchers at vx-underground estimated data was exfiltrated from 500,000 machines.

What did the malicious package actually do?

It ran a three-stage attack. First, it harvested credentials: SSH keys, cloud tokens, Kubernetes secrets, crypto wallets, and .env files. Second, it attempted lateral movement across Kubernetes clusters by deploying privileged pods. Third, it installed a persistent systemd backdoor. Version 1.82.8 abused Python’s .pth file mechanism to execute whenever Python was invoked, regardless of whether LiteLLM was imported.

How do I check if my systems were affected?

Check pip install logs for litellm versions 1.82.7 or 1.82.8 installed between March 24 10:39 UTC and 11:25 UTC. Search for litellm_init.pth files on your system. Check for outbound connections to litellm.cloud. Review systemd services for unexpected entries. If you find any indicators, rotate all credentials immediately.

Why are AI startups more vulnerable to supply chain attacks?

AI stacks combine four risk factors: teams that move fast and skip dependency pinning, packages that require broad system permissions including API keys and cloud tokens, deep dependency trees that are hard to audit, and a community culture built on trust and rapid adoption of new tools. These factors compound to create an unusually large attack surface.

How can I protect my AI product from supply chain attacks?

Pin all dependencies to exact versions with hash verification. Pin GitHub Actions to commit SHAs. Use OIDC instead of long-lived tokens in CI/CD. Scope secrets so each component has minimum permissions. Use a private package registry. Monitor for unexpected .pth files and systemd changes. Have a tested credential rotation plan ready before you need it.