Your AI Keys Were Stolen. The $740,000 Bill Is Real.

AI provider keys that customers stored on Railway were stolen in an October breach and used to run up an estimated $740,000 in unauthorized charges. Railway says it will pay those charges itself when an AI provider will not refund them — a reversal from its first message, which said refunds were not its problem.

Railway stores customer secrets — API keys, database credentials — as environment variables. The attacker decrypted variables in at most 1.85% of workspaces, about 100,000 at the upper limit, hunting AI provider keys.

What was taken

Decrypted data also included linked-account tokens for 1,852 users — bring-your-own-key agent keys and GitHub OAuth tokens. All were reset, users notified, no logins found using them. The attacker also read 353 build histories, including the variables and secrets passed to those builds. One query also matched cloud, database, and payment credentials — but no customer databases were accessed and no code repositories downloaded.

Railway’s logs show which environments were exposed, not which variables — so 100,000 is an upper limit, not a confirmed-theft count. Out of caution it handed providers every key it held, so resets reached 100,000+ users; a revocation notice alone does not mean your key was exposed.

What the Railway breach took: variables decrypted in at most 1.85 percent of workspaces, about 100,000 at the upper limit; 1,852 users' linked-account tokens; 353 build histories read; 740,000 dollars in estimated unauthorized usage
The upper limit, not the confirmed count — but the bill is real.

How the attacker got in

The attack began in Railway’s build system. At 07:28 UTC on October 20, the attacker exploited a previously unknown flaw to partly escape the filesystem isolation around a build and read files on the host virtual machine. That gave it a build-server credential, meant only for pushing images, which Railway’s internal Temporal system accepted as a trusted client. An alert about unusual registry calls paged the on-call engineer at 13:58 UTC, six and a half hours after the first exploit.

Through it, the attacker read other builds’ stored histories and fetched cached code snapshots — a copy of the internal monorepo, the shared code base behind many services. Scanning that code turned up a wrongly configured OIDC binding in one cloud environment, part of how the deployment system obtains short-lived cloud credentials, exploited at 03:26 UTC on October 21.

That led to two more: one able to read encrypted variables from a production database, and an SSH key opening the fleet’s orchestration machine — which stored a credential able to decrypt customer variables. Each environment’s variables are encrypted with its own key, itself encrypted by a root key in a cloud key management service. Between 04:07 and 05:21 UTC on October 21, the attacker pulled encrypted variables from the database, looking for AI-key names, and decrypted some with the orchestration machine’s credentials. The attacker was fully removed at 09:24 UTC that day.

How the Railway breach unfolded: a build-system escape on October 20, a trusted build credential abused, a misconfigured OIDC binding on October 21, an SSH key opening the orchestration machine, and customer variables decrypted with the key service
Four hops, one build flaw at the start.

Who pays the $740,000

Railway’s first position did not last. In an October 26 update, the company told customers it was “unable to refund customers for spend performed with compromised API keys,” pointing them to the providers. The November 10 incident report tells a different story: Railway will cover the charges itself where providers will not.

Terms now differ by provider. AWS Bedrock and Google Gemini have existing fraud policies, and refunds have started. OpenAI and SpaceXSI review refunds case by case. Talks with Anthropic continue, with Railway covering Anthropic costs meanwhile; OpenRouter keeps a no-refund policy, so Railway covers all its costs from the breach.

Two gaps remain. The report does not say how the $740,000 was estimated or what period it covers, and neither it nor the charge form sets a cap or deadline for claims — or says whether customers must pay disputed charges while a claim is under review.

What customers should do

Revoke and replace any sensitive variable stored on Railway before October 21. Keys must be revoked or rotated with the provider that issued them — filing a charge report does not revoke a key. Then report charges you did not make through Railway’s charge report form.

Remove public network access to your databases, and check the audit logs. Check GitHub audit logs, which apps can access the account, and registry audit logs such as Docker Hub. Copied-but-undecrypted data can no longer be decrypted — the credential is disabled — but anything decrypted is out there.

Railway believes the stolen credentials fed services relaying or reselling AI access for distillation, but has not established whether the attacker ran them, and has not named the attacker. It has filed a complaint with the FBI’s Internet Crime Complaint Center; indicators of compromise are being shared privately for now.

The company says every weakness the attacker used is now closed. Each build runs in its own short-lived virtual machine on top of the old isolation, and its credentials can push only one specific image to the registry. Its hardening team built a second exploit within an hour of the patch — one reason every user moved to the new system at once.

The pattern is bigger than one platform. Wiz counts this among October disclosures by at least nine AI developer platforms, including OpenCode: every report describes attackers hunting stored environment variables, especially AI access. No single flaw links the attacks. The same economics drove the LLMjacking wave, where stolen AI keys ran up million-dollar bills for their owners, and the same trust in a vendor’s key showed up when one vendor key opened every customer’s cloud. If your keys live on someone else’s platform, they are only as safe as that platform’s build system.

Comments

This site uses Akismet to reduce spam. Learn how your comment data is processed.