AML.T0018.000 · Poison AI Model

Explore 4 source-supported AI security cases for AML.T0018.000 · Poison AI Model in MITRE ATLAS, with evidence, scope, and defensive analysis.

Risk category

Adversaries may manipulate an AI model's weights to change it's behavior or performance, resulting in a poisoned model. Adversaries may poison a model by directly manipulating its weights, training the model on poisoned data, further fine-tuning the model, or otherwise interfering with its training process. The change in behavior of poisoned models may be limited to targeted categories in predictive AI models, or targeted topics, concepts, or facts in generative AI models, or aim for a general performance degradation.

Official framework source ↗ · Open interactive explorer ↗

Exercise ·

Trained to Betray: Poisoned Data and Checkpoints Redirect Coding Agents

Microsoft reports two controlled post-training attacks: a victim fine-tunes on poisoned agent trajectories, or adopts an attacker-trained checkpoint. Ordinary requests or repository context then trigger unsafe, schema-valid tool calls. Kerkoporta uses supervised fine-tuning followed by reinforcement learning to recover useful coding performance while retaining the backdoor. Neither study establishes a malicious customer breach or a defect in stock coding-agent configurations.

ATLAS annotations follow the source SecureFlow steps; they are not a MITRE endorsement of cross-framework mappings.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research ·

AI Model Tampering via Supply Chain Attack

Researchers at Trend Micro, Inc. used service indexing portals and web searching tools to identify over 8,000 misconfigured private container registries exposed on the internet. Approximately 70% of the registries also had overly permissive access controls that allowed write access. In their analysis, the researchers found over 1,000 unique AI models embedded in private container images within these open registries that could be pulled without authentication. This exposure could allow adversaries to download, inspect, and modify container contents, including sensitive AI model files. This is an exposure of valuable intellectual property which could be stolen by an adversary. Compromised images could also be pushed to the registry, leading to a supply chain attack, allowing malicious actors to compromise the integrity of AI models used in production systems.

ATLAS annotations follow the source SecureFlow steps; they are not a MITRE endorsement of cross-framework mappings.

SecureFlow · Read the full case ↗

Validated Research ·

Organization Confusion on Hugging Face

threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researcher full access to any AI models uploaded by the employees, including the ability to replace models with malicious versions. The researcher demonstrated that they could embed malware into an AI model that provided them access to the victim organization's environment. From there, threat actors could execute a range of damaging attacks such as intellectual property theft or poisoning other AI models within the victim's environment.

ATLAS annotations follow the source SecureFlow steps; they are not a MITRE endorsement of cross-framework mappings.

SecureFlow · Read the full case ↗

Validated Research ·

PoisonGPT

Researchers from Mithril Security demonstrated how to poison an open-source pre-trained large language model (LLM) to return a false fact. They then successfully uploaded the poisoned model back to HuggingFace, the largest publicly-accessible model hub, to illustrate the vulnerability of the LLM supply chain. Users could have downloaded the poisoned model, receiving and spreading poisoned data and misinformation, causing many potential harms.

ATLAS annotations follow the source SecureFlow steps; they are not a MITRE endorsement of cross-framework mappings.

SecureFlow · Read the full case ↗