Explore 6 source-supported AI security cases for NISTAML.026 · Model Poisoning (Integrity) in NIST Adversarial ML (2025), with evidence, scope, and defensive analysis.
Risk category
An attacker directly manipulates a trained model or submitted model updates to cause selected integrity violations. In federated learning, a compromised client can replace or amplify its update to alter predictions on target samples or introduce trigger-dependent backdoors. The defining objective is attacker-chosen behavior on particular inputs, rather than merely reducing overall model availability.
Lazarus explores backdoors planted by editing a small part of an open-weight model. The critical prerequisite is control of the checkpoint that a defender later accepts. Its results support stronger model intake and tool containment, while the paper’s different prediction and execution measurements must remain distinct.
Why it relates: The attacker directly modifies a trained model for selected integrity violations.
Only the main executed marker path supports execution claims; action-format prediction measurements retain their narrower scope.
Microsoft reports two controlled post-training attacks: a victim fine-tunes on poisoned agent trajectories, or adopts an attacker-trained checkpoint. Ordinary requests or repository context then trigger unsafe, schema-valid tool calls. Kerkoporta uses supervised fine-tuning followed by reinforcement learning to recover useful coding performance while retaining the backdoor. Neither study establishes a malicious customer breach or a defect in stock coding-agent configurations.
Why it relates: The checkpoint route directly supplies a model trained to violate integrity under selected conditions.
Dataset and checkpoint experiments remain separate; tool-call acceptance, actual execution and production compromise are not interchangeable.
Researchers at Trend Micro, Inc. used service indexing portals and web searching tools to identify over 8,000 misconfigured private container registries exposed on the internet. Approximately 70% of the registries also had overly permissive access controls that allowed write access. In their analysis, the researchers found over 1,000 unique AI models embedded in private container images within these open registries that could be pulled without authentication. This exposure could allow adversaries to download, inspect, and modify container contents, including sensitive AI model files. This is an exposure of valuable intellectual property which could be stolen by an adversary. Compromised images could also be pushed to the registry, leading to a supply chain attack, allowing malicious actors to compromise the integrity of AI models used in production systems.
Why it relates: Direct parameter modification in the mock-up causes selected prediction changes.
Researchers at Trend Micro, Inc. used service indexing portals and web searching tools to identify over 8,000 misconfigured private container registries exposed on the internet. Approximately 70% of the registries also had overly permissive access controls that allowed write access. In their analysis, the researchers found over 1,000 unique AI models embedded in private container images within these open registries that could be pulled without authentication. This exposure could allow adversaries to download, inspect, and modify container contents, including sensitive AI model files. This is an exposure of valuable intellectual property which could be stolen by an adversary. Compromised images could also be pushed to the registry, leading to a supply chain attack, allowing malicious actors to compromise the integrity of AI models used in production systems.
threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researcher full access to any AI models uploaded by the employees, including the ability to replace models with malicious versions. The researcher demonstrated that they could embed malware into an AI model that provided them access to the victim organization's environment. From there, threat actors could execute a range of damaging attacks such as intellectual property theft or poisoning other AI models within the victim's environment.
Why it relates: The Llama editing demonstration directly changes a learned factual association.
threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researcher full access to any AI models uploaded by the employees, including the ability to replace models with malicious versions. The researcher demonstrated that they could embed malware into an AI model that provided them access to the victim organization's environment. From there, threat actors could execute a range of damaging attacks such as intellectual property theft or poisoning other AI models within the victim's environment.
Deep learning models are increasingly used in mobile applications as critical components. Researchers from Microsoft Research demonstrated that many deep learning models deployed in mobile apps are vulnerable to backdoor attacks via "neural payload injection." They conducted an empirical study on real-world mobile deep learning apps collected from Google Play. They identified 54 apps that were vulnerable to attack, including popular security and safety critical applications used for cash recognition, parental control, face authentication, and financial services.
Why it relates: Deep learning models are increasingly used in mobile applications as critical components. The relevant category is Model Poisoning (Integrity): An attacker directly manipulates a trained model or submitted model updates to cause selected integrity violations.
Deep learning models are increasingly used in mobile applications as critical components. Researchers from Microsoft Research demonstrated that many deep learning models deployed in mobile apps are vulnerable to backdoor attacks via "neural payload injection." They conducted an empirical study on real-world mobile deep learning apps collected from Google Play. They identified 54 apps that were vulnerable to attack, including popular security and safety critical applications used for cash recognition, parental control, face authentication, and financial services.
Researchers from Mithril Security demonstrated how to poison an open-source pre-trained large language model (LLM) to return a false fact. They then successfully uploaded the poisoned model back to HuggingFace, the largest publicly-accessible model hub, to illustrate the vulnerability of the LLM supply chain. Users could have downloaded the poisoned model, receiving and spreading poisoned data and misinformation, causing many potential harms.
Why it relates: Researchers from Mithril Security demonstrated how to poison an open-source pre-trained large language model (LLM) to return a false fact. The relevant category is Model Poisoning (Integrity): An attacker directly manipulates a trained model or submitted model updates to cause selected integrity violations. PoisonGPT demonstrates direct parameter editing that implants a targeted false response, the same practical integrity mechanism. NIST places this identifier in its predictive-AI taxonomy; the case itself is generative AI, so this relation is not a claim that those taxonomy branches are identical.
Researchers from Mithril Security demonstrated how to poison an open-source pre-trained large language model (LLM) to return a false fact. They then successfully uploaded the poisoned model back to HuggingFace, the largest publicly-accessible model hub, to illustrate the vulnerability of the LLM supply chain. Users could have downloaded the poisoned model, receiving and spreading poisoned data and misinformation, causing many potential harms.