Explore 11 source-supported AI security cases for ML05:2023 · Model Theft in OWASP ML Top 10 (2023), with evidence, scope, and defensive analysis.
Risk category
An attacker gains unauthorized access to a model's parameters or obtains enough of its implementation to recreate and use the model. The OWASP description emphasizes access to parameters, while its example also covers reverse engineering the model's binary or obtaining its algorithm and training resources. The result can be loss of model intellectual property, competitive value, and confidentiality; the scope is broader than copying behavior through an API alone.
Fireworks AI, a hosted platform for running, evaluating and training models, disclosed unauthorized access to customer evaluation-job secrets and later development-machine activity affecting customer data. Customers can revoke exposed credentials at their issuers; the platform operator owns isolation between development systems and customer assets.
Why it relates: Fireworks confirms unauthorized access to a subset of incomplete customer model weights. The category explicitly includes unauthorized access to model parameters; a complete usable model replica is not claimed.
Mappings cover unauthorized AI-platform credential and data access, affected infrastructure/data operations, supplier dependency and model-asset confidentiality. Practical overlaps explicitly preserve missing agent, MCP, model-query and completed-copy evidence. No mapping represents a verified defensive success.
NSA, CISA and FBI attributed coordinated distillation campaigns to six China-based AI companies. Their September 2026 advisory describes fraudulent accounts, bulk subscriptions, gray-market proxies, cross-provider routing, request-metadata removal and hidden-reasoning prompts, followed by response collection and quality checks. Distillation trains a model from another model’s outputs; the advisory’s attribution and capability-assessment claims are attributed to the agencies, not presented as independently reproduced measurements or model-weight theft.
Why it relates: the campaigns attributed by NSA, CISA and FBI sought selected model capabilities through output-based training. The supported overlap is functional capability appropriation; original parameters, architecture or an exact full-model replica are not established.
the campaigns attributed by NSA, CISA and FBI; capability distillation is distinguished from stealing original weights. Attribution and capability outcomes remain agency assessments. Hidden-reasoning prompts are attempts, not proof that the returned text was an internal trace. No autonomous agent deployment is inferred.
Google reported campaigns that used compromised credentials, fraudulent accounts and proxies to collect model outputs for distillation: training another model from a provider’s responses. A February 2026 reasoning-focused campaign exceeded 100,000 prompts; later campaigns included some above 100 million. Those are campaign query volumes, not model weights stolen or a measured replication success rate. Google says it disrupted accounts and reduced the usefulness of collected outputs.
Why it relates: Google’s compromised-account and fraudulent-account campaigns sought selected model capabilities through output-based training. The supported overlap is functional capability appropriation; original parameters, architecture or an exact full-model replica are not established.
Google’s compromised-account and fraudulent-account campaigns; capability distillation is distinguished from stealing original weights. The prompt totals are collection volumes, not measured replication success or model weights. No autonomous agent deployment or protected-reasoning recovery is established.
Anthropic attributed more than 16 million exchanges through roughly 24,000 fraudulent accounts to DeepSeek, Moonshot AI and MiniMax. The flow distinguishes proxy access, authenticated accounts, distributed routing, inference-response collection and downstream model training. Provider-side hunting and verified limits act before outputs leave; potential harms from models lacking safeguards remain Anthropic’s risk assessment, not observed downstream victim counts.
Why it relates: Anthropic’s DeepSeek, Moonshot AI and MiniMax campaigns sought selected model capabilities through output-based training. The supported overlap is functional capability appropriation; original parameters, architecture or an exact full-model replica are not established.
Anthropic’s DeepSeek, Moonshot AI and MiniMax campaigns; capability distillation is distinguished from stealing original weights. The 24,000-account and 16-million-exchange figures do not quantify recovered weights or observed downstream harms. Potential harms remain Anthropic’s assessment; an agent deployment is not established.
Researchers at Trend Micro, Inc. used service indexing portals and web searching tools to identify over 8,000 misconfigured private container registries exposed on the internet. Approximately 70% of the registries also had overly permissive access controls that allowed write access. In their analysis, the researchers found over 1,000 unique AI models embedded in private container images within these open registries that could be pulled without authentication. This exposure could allow adversaries to download, inspect, and modify container contents, including sensitive AI model files. This is an exposure of valuable intellectual property which could be stolen by an adversary. Compromised images could also be pushed to the registry, leading to a supply chain attack, allowing malicious actors to compromise the integrity of AI models used in production systems.
Why it relates: Researchers at Trend Micro, Inc. The relevant category is Model Theft: An attacker gains unauthorized access to a model's parameters or obtains enough of its implementation to recreate and use the model. Registry exposure permits model-image retrieval and potential substitution; later malicious pulls are not established by the exposure survey.
Researchers at Trend Micro, Inc. used service indexing portals and web searching tools to identify over 8,000 misconfigured private container registries exposed on the internet. Approximately 70% of the registries also had overly permissive access controls that allowed write access. In their analysis, the researchers found over 1,000 unique AI models embedded in private container images within these open registries that could be pulled without authentication. This exposure could allow adversaries to download, inspect, and modify container contents, including sensitive AI model files. This is an exposure of valuable intellectual property which could be stolen by an adversary. Compromised images could also be pushed to the registry, leading to a supply chain attack, allowing malicious actors to compromise the integrity of AI models used in production systems.
threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researcher full access to any AI models uploaded by the employees, including the ability to replace models with malicious versions. The researcher demonstrated that they could embed malware into an AI model that provided them access to the victim organization's environment. From there, threat actors could execute a range of damaging attacks such as intellectual property theft or poisoning other AI models within the victim's environment.
Why it relates: threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. The relevant category is Model Theft: An attacker gains unauthorized access to a model's parameters or obtains enough of its implementation to recreate and use the model. Organization trust exposes employee-uploaded models and replacement authority; the research also demonstrated malicious-model execution.
threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researcher full access to any AI models uploaded by the employees, including the ability to replace models with malicious versions. The researcher demonstrated that they could embed malware into an AI model that provided them access to the victim organization's environment. From there, threat actors could execute a range of damaging attacks such as intellectual property theft or poisoning other AI models within the victim's environment.
Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numerically based on multiple variables and if the overall score of the email is too low, ProofPoint will output an error, labeling it as SPAM.
Why it relates: Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. The relevant category is Model Theft: An attacker gains unauthorized access to a model's parameters or obtains enough of its implementation to recreate and use the model. The experiment recreates behavior from API outputs; access to the original parameters, proprietary implementation or complete original model is not established.
Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numerically based on multiple variables and if the overall score of the email is too low, ProofPoint will output an error, labeling it as SPAM.
Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.
Why it relates: Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. The relevant category is Model Theft: An attacker gains unauthorized access to a model's parameters or obtains enough of its implementation to recreate and use the model. The experiment recreates behavior from API outputs; access to the original parameters, proprietary implementation or complete original model is not established.
Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.
GPUBreach researchers used unprivileged CUDA code on an NVIDIA RTX A6000 to corrupt GPU page tables through Rowhammer, a repeated-memory-access technique that induces bit flips. Their controlled demonstrations include cross-process GPU access, LLM weight leakage and a separate driver path to CPU root with IOMMU enabled. The relevant AI defense is removing untrusted GPU co-residency, not changing a model prompt.
Why it relates: The primary demonstration includes unauthorized access to LLM weights in GPU memory.
Controlled GPU page-table corruption and distinct CPU-driver escalation; model-weight theft is not training-data inference.
Skyld researchers analyzed the Google Photos Android application and recovered TensorFlow Lite models used by AI-powered photo editing and image analysis features. The researchers found models stored unencrypted in the application's assets, embedded in the native library, and encrypted on disk. They used static reverse engineering to locate TFLite artifacts and dynamic instrumentation with Frida to capture encrypted models after runtime decryption. The recovered models provided white-box access to proprietary Google Photos model artifacts.
Why it relates: Researchers recover complete TFLite model files rather than only imitate outputs.
Static APK/native-library extraction and runtime interception recover model files. The demonstrated result is model access, with later evasion only potential.
The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.
Why it relates: The Azure-service exercise targets availability across separate conventional and ML steps; it is not an observed malicious compromise. The red team found the target model file and its training data and exfiltrated both before crafting offline adversarial examples. This is direct model-artifact access, not merely an inference that evasion must have copied a model.
The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.