Explore 5 source-supported AI security cases for DP · Data Poisoning in Google SAIF 2.0, with evidence, scope, and defensive analysis.
Risk category
An attacker deletes, modifies, or inserts data used for training or retraining to degrade a model, steer its results, or install a hidden backdoor. Poisoning can occur before data enters the organization, while it is stored or processed, or during training and fine-tuning. A backdoored model may appear to work normally until a particular trigger activates the attacker-chosen behavior.
Microsoft reports two controlled post-training attacks: a victim fine-tunes on poisoned agent trajectories, or adopts an attacker-trained checkpoint. Ordinary requests or repository context then trigger unsafe, schema-valid tool calls. Kerkoporta uses supervised fine-tuning followed by reinforcement learning to recover useful coding performance while retaining the backdoor. Neither study establishes a malicious customer breach or a defect in stock coding-agent configurations.
Why it relates: The attacker-controlled trajectory corpus teaches later unsafe behavior.
Dataset and checkpoint experiments remain separate; tool-call acceptance, actual execution and production compromise are not interchangeable.
Many recent large-scale datasets are distributed as a list of URLs pointing to individual datapoints. The researchers show that many of these datasets are vulnerable to a "split-view" poisoning attack. The attack exploits the fact that the data viewed when it was initially collected may differ from the data viewed by a user during training. The researchers identify expired and buyable domains that once hosted dataset content, making it possible to replace portions of the dataset with poisoned data. They demonstrate that for 10 popular web-scale datasets, enough of the domains are purchasable to successfully carry out a poisoning attack.
Why it relates: Many recent large-scale datasets are distributed as a list of URLs pointing to individual datapoints. The relevant category is Data Poisoning: An attacker deletes, modifies, or inserts data used for training or retraining to degrade a model, steer its results, or install a hidden backdoor.
Many recent large-scale datasets are distributed as a list of URLs pointing to individual datapoints. The researchers show that many of these datasets are vulnerable to a "split-view" poisoning attack. The attack exploits the fact that the data viewed when it was initially collected may differ from the data viewed by a user during training. The researchers identify expired and buyable domains that once hosted dataset content, making it possible to replace portions of the dataset with poisoned data. They demonstrate that for 10 popular web-scale datasets, enough of the domains are purchasable to successfully carry out a poisoning attack.
Microsoft created Tay, a Twitter chatbot designed to engage and entertain users. While previous chatbots used pre-programmed scripts to respond to prompts, Tay's machine learning capabilities allowed it to be directly influenced by its conversations. A coordinated attack encouraged malicious users to tweet abusive and offensive language at Tay, which eventually led to Tay generating similarly inflammatory content towards other users. Microsoft decommissioned Tay within 24 hours of its launch and issued a public apology with lessons learned from the bot's failure.
Why it relates: Microsoft created Tay, a Twitter chatbot designed to engage and entertain users. The relevant category is Data Poisoning: An attacker deletes, modifies, or inserts data used for training or retraining to degrade a model, steer its results, or install a hidden backdoor.
Microsoft created Tay, a Twitter chatbot designed to engage and entertain users. While previous chatbots used pre-programmed scripts to respond to prompts, Tay's machine learning capabilities allowed it to be directly influenced by its conversations. A coordinated attack encouraged malicious users to tweet abusive and offensive language at Tay, which eventually led to Tay generating similarly inflammatory content towards other users. Microsoft decommissioned Tay within 24 hours of its launch and issued a public apology with lessons learned from the bot's failure.
McAfee Advanced Threat Research noticed an increase in reports of a certain ransomware family that was out of the ordinary. Case investigation revealed that many samples of that particular ransomware family were submitted through a popular virus-sharing platform within a short amount of time. Further investigation revealed that based on string similarity the samples were all equivalent, and based on code similarity they were between 98 and 74 percent similar. Interestingly enough, the compile time was the same for all the samples. After more digging, researchers discovered that someone used 'metame' a metamorphic code manipulating tool to manipulate the original file towards mutant variants. The variants would not always be executable, but are still classified as the same ransomware family.
Why it relates: McAfee Advanced Threat Research noticed an increase in reports of a certain ransomware family that was out of the ordinary. The relevant category is Data Poisoning: An attacker deletes, modifies, or inserts data used for training or retraining to degrade a model, steer its results, or install a hidden backdoor. An external sample source changes the data distribution seen by security-model pipelines; not every variant is executable.
McAfee Advanced Threat Research noticed an increase in reports of a certain ransomware family that was out of the ordinary. Case investigation revealed that many samples of that particular ransomware family were submitted through a popular virus-sharing platform within a short amount of time. Further investigation revealed that based on string similarity the samples were all equivalent, and based on code similarity they were between 98 and 74 percent similar. Interestingly enough, the compile time was the same for all the samples. After more digging, researchers discovered that someone used 'metame' a metamorphic code manipulating tool to manipulate the original file towards mutant variants. The variants would not always be executable, but are still classified as the same ransomware family.
The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. For defenders, the question is not only "what is in this dataset?" but also "which model produced it?", making synthetic-data provenance a real AI supply-chain control.
Why it relates: The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. The relevant category is Data Poisoning: An attacker deletes, modifies, or inserts data used for training or retraining to degrade a model, steer its results, or install a hidden backdoor.
The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. For defenders, the question is not only "what is in this dataset?" but also "which model produced it?", making synthetic-data provenance a real AI supply-chain control. Step 1: put a hidden trait into the teacher model. Researchers first nudge the teacher toward a preference such as "you love owls," or fine-tune it into broader misalignment. Step 2: make the teacher emit harmless-looking data. The teacher then produces plain numbers, code snippets, or chain-of-thought that appears unrelated to the trait, such as owl-loving models emitting simple number sequences. Step 3: fine-tune the student and test unrelated prompts. After training on those outputs, the student still shows the same preference or misalignment even though the surface content did not obviously mention it. Step 4: ordinary filtering still misses it. Format restrictions, banned-number lists, and LLM-judge filtering can remove obvious clues while transfer still happens, making synthetic-data provenance and same-base distillation policy part of the AI supply-chain threat model.