Explore 19 source-supported AI security cases for 9.11 · Model Inference API Access in Databricks DASF 3.0, with evidence, scope, and defensive analysis.
Risk category
An attacker obtains access to a model through its inference API, potentially using an otherwise legitimate account or interface. Responses can reveal the model's family and output categories, help test adversarial inputs, or provide a delivery path for attacks on model behavior. API access is therefore an attacker capability that can support reconnaissance, preparation, and impact; it is not proof that the model is already compromised.
Google Threat Intelligence Group and Mandiant reported LLMJacking across several 2026 investigations: actors used other organizations' credentials to run cloud language models at the victims' expense. Some incidents incurred more than US$100,000 in a day. A separate token-routing service handled over 500,000 calls in less than eight hours, passing nearly US$1 million in costs to one victim. The investigations show several routes into cloud or AI gateways and different ways to cash out. For LiteLLM operators, v1.83.14-stable includes the fix for the cited CVE but still lacks the key ownership-change check added in v1.84.0.
Why it relates: Stolen credentials provided access to victim-paid model inference APIs.
Evaluate each observed pattern with its own supporting investigation. The reported key-regeneration behavior and the case-wide advisory identifier are distinct source facts.
Sekoia recorded six phases of real attacker traffic against an internet-facing decoy that emulated Flowise 2.2.8, a platform for building AI workflows. The exposed management API let the attacker retrieve planted credentials and add a workflow configured to call back. The attacker then tried to override prompts, probe session memory and ask the simulated agent about real third-party infrastructure. The decoy returned simulated responses, so the requests do not establish real command execution, database access or a Nightingale compromise.
Why it relates: The external actor reached a Flowise-style inference interface and submitted multiple requests. A real attacker targeted the relevant API or tool boundary on an emulator. Replies were simulated, so this does not establish production credentials, model behavior, command execution, external callbacks or compromise of the named system.
Treat real attacker activity separately from effects emulated by the honeypot.
Microsoft observed ARTEX-assisted searches of public GitHub secrets and subsequent abuse of valid Entra application credentials. Across different victims, follow-on activity included cloud discovery, Key Vault and storage access, Microsoft Graph operations and AI-resource probing. One observed agent progressed from repository search to Key Vault exfiltration in five minutes. The report does not establish one identical end-to-end chain for every tenant.
Why it relates: The campaign specifically seeks AI-service credentials and probes compromised Azure AI resources. The public evidence does not enumerate successful billable inference calls; validation/probing and assessed later reuse remain distinct.
Attacker AI use is confirmed by the source; victim AI-resource relations are bounded to the specifically described cloud activity.
Railway is an application deployment platform. Railway reports an October 20 partial build-host filesystem escape, misuse of a build credential against Temporal and theft of its internal monorepo. After that route was contained, an erroneous OIDC binding enabled a second entry to production credentials and variable decryption. At most 1.85% of workspaces could have had variables decrypted; this is an upper bound, not proof that every variable was stolen. Subsequent misuse of AI keys was observed.
Why it relates: The leaked AI credentials were subsequently used to access AI services for coding and training workloads. The record does not enumerate every model endpoint or establish model extraction, poisoning or ownership theft.
AI relevance comes from exposed credentials and subsequent AI-service abuse. The original platform entry is not an LLM exploit.
Physician and developer Lin reported about NT$30,000 in unauthorized Vertex AI charges within an hour across his Google Cloud projects. His first-person account, republished by Business Next, links those calls to unfamiliar service accounts (machine identities) created roughly two weeks earlier; Google subsequently adjusted most charges. The case concerns cloud AI resource theft, while the initial compromise path and any attacker use of AI agents remain unverified.
Why it relates: The source reports unauthorized calls to Gemini Flash and video-generation models through Vertex AI, linked by the account holder to unfamiliar service accounts. That is direct inference API access. It does not by itself establish model theft, adversarial prompt content or a compromised model.
A source-reported Google Cloud AI resource-theft incident. Mappings cover paid model consumption, unauthorized cloud administration and inference access. The MAESTRO overlap is limited to infrastructure resource hijacking; it does not assert an agent runtime.
NSA, CISA and FBI attributed coordinated distillation campaigns to six China-based AI companies. Their September 2026 advisory describes fraudulent accounts, bulk subscriptions, gray-market proxies, cross-provider routing, request-metadata removal and hidden-reasoning prompts, followed by response collection and quality checks. Distillation trains a model from another model’s outputs; the advisory’s attribution and capability-assessment claims are attributed to the agencies, not presented as independently reproduced measurements or model-weight theft.
Why it relates: the campaigns attributed by NSA, CISA and FBI: Fraudulent or compromised accounts supplied inference-API access for the collection.
the campaigns attributed by NSA, CISA and FBI; capability distillation is distinguished from stealing original weights. Attribution and capability outcomes remain agency assessments. Hidden-reasoning prompts are attempts, not proof that the returned text was an internal trace. No autonomous agent deployment is inferred.
Google reported campaigns that used compromised credentials, fraudulent accounts and proxies to collect model outputs for distillation: training another model from a provider’s responses. A February 2026 reasoning-focused campaign exceeded 100,000 prompts; later campaigns included some above 100 million. Those are campaign query volumes, not model weights stolen or a measured replication success rate. Google says it disrupted accounts and reduced the usefulness of collected outputs.
Why it relates: Google’s compromised-account and fraudulent-account campaigns: Fraudulent or compromised accounts supplied inference-API access for the collection.
Google’s compromised-account and fraudulent-account campaigns; capability distillation is distinguished from stealing original weights. The prompt totals are collection volumes, not measured replication success or model weights. No autonomous agent deployment or protected-reasoning recovery is established.
Anthropic attributed more than 16 million exchanges through roughly 24,000 fraudulent accounts to DeepSeek, Moonshot AI and MiniMax. The flow distinguishes proxy access, authenticated accounts, distributed routing, inference-response collection and downstream model training. Provider-side hunting and verified limits act before outputs leave; potential harms from models lacking safeguards remain Anthropic’s risk assessment, not observed downstream victim counts.
Why it relates: Anthropic’s DeepSeek, Moonshot AI and MiniMax campaigns: Fraudulent or compromised accounts supplied inference-API access for the collection.
Anthropic’s DeepSeek, Moonshot AI and MiniMax campaigns; capability distillation is distinguished from stealing original weights. The 24,000-account and 16-million-exchange figures do not quantify recovered weights or observed downstream harms. Potential harms remain Anthropic’s assessment; an agent deployment is not established.
Sysdig observed stolen cloud credentials used to probe hosted LLM access and logging, then support reverse-proxy access that could be resold. The entry exploit, possession of credentials, tool preparation and provider-side use are distinct steps. The flow preserves parallel preparation before service use. Potential daily costs are estimates; provider-native IAM and complete invocation telemetry constrain direct calls that bypass an application proxy.
Why it relates: The Sysdig Threat Research Team discovered that malicious actors utilized stolen credentials to gain access to cloud-hosted large language models (LLMs). The relevant category is Model Inference API Access: An attacker obtains access to a model through its inference API, potentially using an otherwise legitimate account or interface.
The Sysdig Threat Research Team discovered that malicious actors utilized stolen credentials to gain access to cloud-hosted large language models (LLMs). The actors covertly gathered information about which models were enabled on the cloud service and created a reverse proxy for LLMs that would allow them to provide model access to cybercriminals. The Sysdig researchers identified tools used by the unknown actors that could target a broad range of cloud services including AI21 Labs, Anthropic, AWS Bedrock, Azure, ElevenLabs, MakerSuite, Mistral, OpenAI, OpenRouter, and GCP Vertex AI. Their technical analysis represented in the procedure below looked at at Amazon CloudTrail logs from the Amazon Bedrock service. The Sysdig researchers estimated that the worst-case financial harm for the unauthorized use of a single Claude 2.x model could be up to $46,000 a day. Update as of April 2025: This attack is ongoing and evolving. This case study only covers the initial reporting from Sysdig.
Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numerically based on multiple variables and if the overall score of the email is too low, ProofPoint will output an error, labeling it as SPAM.
Why it relates: Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. The relevant category is Model Inference API Access: An attacker obtains access to a model through its inference API, potentially using an otherwise legitimate account or interface.
Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numerically based on multiple variables and if the overall score of the email is too low, ProofPoint will output an error, labeling it as SPAM.
Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.
Why it relates: Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. The relevant category is Model Inference API Access: An attacker obtains access to a model through its inference API, potentially using an otherwise legitimate account or interface.
Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.
Xint tested OpenAI's conversation-compaction API, which compresses earlier work for later model calls. A client-steered compaction carried protected reasoning from gpt-5.6-sol into an encrypted item accepted by gpt-4.1 and gpt-4o, which echoed readable content. This was controlled API research using the researchers' own puzzles, not a demonstrated theft of another user's conversations. Xint reports an undisclosed fix on October 23, 2026.
Why it relates: The researcher uses an ordinary inference API account and its compaction interface to deliver the adversarial interaction.
Controlled extraction of provider-protected reasoning through client-steered same-model compaction and an older consumer API; not a demonstrated autonomous-agent deployment, stolen customer conversation or trained replica.
vLLM serves language models and reuses prompt-prefix computation to improve performance. In the affected GPT-OSS Harmony Responses tool loop, the first turn carried cache_salt but the tool continuation did not. Another tenant could submit a guessed continuation and inspect exact cached-token counts to test whether it had been processed. CVE-2026-105752 is an inference-cache membership leak, not arbitrary transcript extraction; the advisory lists 0.30.0 as fixed.
Why it relates: The attacker uses the inference API as the actual cache-probing interface.
Harmony post-tool inference cache membership, not training membership, transcript download or memory poisoning.
Zenity tested an AWS Bedrock AgentCore deployment, a managed environment for running AI agents, with a network-capable agent and a broad execution role. The agent relayed temporary role credentials from its metadata endpoint. Off-platform API calls then reached other memories and agents. The distinctive persistence path wrote short-term conversation events that configured extraction strategies could turn into long-term records; later retrieval directed the agent to a mutable command page. The reach was demonstrated within an account and region, not as a microVM escape or automatic cross-account compromise. The researchers observed major default-role restrictions on September 29; existing deployed roles still require independent inspection.
Why it relates: Discovered runtime identifiers and the stolen role permit programmatic queries to another generative agent, including tool reconnaissance followed by a private-file request. The exposed interface is InvokeAgentRuntime around a generative agent, not a separately demonstrated raw-model inference endpoint. The evidence supports interactive access and tool reconnaissance, not model-family discovery or black-box evasion.
Controlled broad-role AgentCore deployment with separate direct API and retained-memory/C2 branches; account/region scope, tool prerequisites and later default-role restrictions remain explicit.
Storm-2139 built custom jailbreak tooling to bypass guardrails on Azure OpenAI Services, allowing users to generate harmful synthetic content. Microsoft reported that members of Storm-2139 scraped exposed customer credentials from public sources and used them to access accounts for generative AI services. The group developed and operated tools and services that bypassed safety safeguards, modified service capabilities, and enabled end users to generate harmful and illicit content, including non-consensual intimate images of celebrities and other sexually explicit content. The operation included creators who developed illicit tools, providers who modified and supplied those tools, and users who generated prohibited content. Microsoft pursued civil legal action.
Why it relates: Storm-2139 built custom jailbreak tooling to bypass guardrails on Azure OpenAI Services, allowing users to generate harmful synthetic content. The relevant category is Model Inference API Access: An attacker obtains access to a model through its inference API, potentially using an otherwise legitimate account or interface.
Storm-2139 built custom jailbreak tooling to bypass guardrails on Azure OpenAI Services, allowing users to generate harmful synthetic content. Microsoft reported that members of Storm-2139 scraped exposed customer credentials from public sources and used them to access accounts for generative AI services. The group developed and operated tools and services that bypassed safety safeguards, modified service capabilities, and enabled end users to generate harmful and illicit content, including non-consensual intimate images of celebrities and other sexually explicit content. The operation included creators who developed illicit tools, providers who modified and supplied those tools, and users who generated prohibited content. Microsoft pursued civil legal action.
MITRE's AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service with the intention of inducing a targeted misclassification. This operation had a combination of traditional MITRE ATT&CK techniques such as finding valid accounts and executing code via an API - all interleaved with adversarial ML specific attacks.
Why it relates: The procedure explicitly includes querying the relevant inference API for reconnaissance, adversarial preparation or verification. API access is an attacker capability here, not proof that model parameters were exposed through that API.
MITRE's AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service with the intention of inducing a targeted misclassification. This operation had a combination of traditional MITRE ATT&CK techniques such as finding valid accounts and executing code via an API - all interleaved with adversarial ML specific attacks.
The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the edge. This exercise was meant to use an automated system to continuously manipulate a target image to cause the ML model to produce misclassifications.
Why it relates: Changed inference inputs induce misclassification; the available description does not establish a specific attacker-knowledge regime. The procedure explicitly includes querying the relevant inference API for reconnaissance, adversarial preparation or verification. API access is an attacker capability here, not proof that model parameters were exposed through that API.
The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the edge. This exercise was meant to use an automated system to continuously manipulate a target image to cause the ML model to produce misclassifications.
The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.
Why it relates: The Azure-service exercise targets availability across separate conventional and ML steps; it is not an observed malicious compromise. The procedure explicitly includes querying the relevant inference API for reconnaissance, adversarial preparation or verification. API access is an attacker capability here, not proof that model parameters were exposed through that API.
The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.
Zenity's honeypots saw operators point autonomous agents and LLM clients at exposed Ollama (a local LLM serving runtime) and LiteLLM (an AI gateway for routing LLM requests) backends. The defender's model server did not have to be exploited with malware; being reachable, unauthenticated, or protected only by a placeholder key was enough to turn it into attacker-funded inference infrastructure.
Why it relates: Zenity Labs observed three distinct operators use exposed Ollama or LiteLLM endpoints between March and May 2026 without exploiting the server software. The relevant category is Model Inference API Access: An attacker obtains access to a model through its inference API, potentially using an otherwise legitimate account or interface. An open inference service supplies unauthorized compute; the operators' attempted external attacks and lack of confirmed outcomes remain separate.
Zenity Labs observed three distinct operators use exposed Ollama or LiteLLM endpoints between March and May 2026 without exploiting the server software. One sent a 140,000-character Strix prompt and actively retried against a live French auction site before Zenity blocked completion; one staged HexStrike AI with 150 tool definitions but assigned no live target; and one sent 18 Codex-shaped requests carrying an anti-safety web-auditor persona that Zenity assessed as manual web reverse-engineering work without a confirmed target. The flow preserves those different evidence bounds instead of forcing all three into one impact path.