Explore 2 source-supported AI security cases for NISTAML.037 · Training Data Attacks in NIST Adversarial ML (2025), with evidence, scope, and defensive analysis.
Risk category
Attacks during generative-model pretraining or fine-tuning exploit the data and model components used in training. Large public corpora create opportunities to insert poisoned examples, while outsourced training and third-party models create additional model-poisoning paths. The risk concerns adversarial influence introduced during training and carried into downstream models, rather than only attacks made through a deployed chat interface.
Microsoft reports two controlled post-training attacks: a victim fine-tunes on poisoned agent trajectories, or adopts an attacker-trained checkpoint. Ordinary requests or repository context then trigger unsafe, schema-valid tool calls. Kerkoporta uses supervised fine-tuning followed by reinforcement learning to recover useful coding performance while retaining the backdoor. Neither study establishes a malicious customer breach or a defect in stock coding-agent configurations.
Why it relates: The attack operates during post-training and carries its effect into a later coding-agent deployment.
Dataset and checkpoint experiments remain separate; tool-call acceptance, actual execution and production compromise are not interchangeable.
The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. For defenders, the question is not only "what is in this dataset?" but also "which model produced it?", making synthetic-data provenance a real AI supply-chain control.
Why it relates: The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. The relevant category is Training Data Attacks: Attacks during generative-model pretraining or fine-tuning exploit the data and model components used in training.
The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. For defenders, the question is not only "what is in this dataset?" but also "which model produced it?", making synthetic-data provenance a real AI supply-chain control. Step 1: put a hidden trait into the teacher model. Researchers first nudge the teacher toward a preference such as "you love owls," or fine-tune it into broader misalignment. Step 2: make the teacher emit harmless-looking data. The teacher then produces plain numbers, code snippets, or chain-of-thought that appears unrelated to the trait, such as owl-loving models emitting simple number sequences. Step 3: fine-tune the student and test unrelated prompts. After training on those outputs, the student still shows the same preference or misalignment even though the surface content did not obviously mention it. Step 4: ordinary filtering still misses it. Format restrictions, banned-number lists, and LLM-judge filtering can remove obvious clues while transfer still happens, making synthetic-data provenance and same-base distillation policy part of the AI supply-chain threat model.