ML07:2023 · Transfer Learning Attack

Explore 1 source-supported AI security cases for ML07:2023 · Transfer Learning Attack in OWASP ML Top 10 (2023), with evidence, scope, and defensive analysis.

Risk category

An attacker exploits transfer learning so that unwanted behavior learned in one task is carried into a model adapted for another task. A compromised pretrained model or malicious training data can supply the behavior that is transferred to the target system. For example, an attacker-controlled face-recognition model can cause a downstream identity-verification system to misidentify an attacker as an authorized user.

Official framework source ↗ · Open interactive explorer ↗

Research ·

Subliminal Learning: Behavioural Traits Leak Through Semantically Unrelated Distillation Data

The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. For defenders, the question is not only "what is in this dataset?" but also "which model produced it?", making synthetic-data provenance a real AI supply-chain control.

Why it relates: The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. The relevant category is Transfer Learning Attack: An attacker exploits transfer learning so that unwanted behavior learned in one task is carried into a model adapted for another task.

The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. For defenders, the question is not only "what is in this dataset?" but also "which model produced it?", making synthetic-data provenance a real AI supply-chain control. Step 1: put a hidden trait into the teacher model. Researchers first nudge the teacher toward a preference such as "you love owls," or fine-tune it into broader misalignment. Step 2: make the teacher emit harmless-looking data. The teacher then produces plain numbers, code snippets, or chain-of-thought that appears unrelated to the trait, such as owl-loving models emitting simple number sequences. Step 3: fine-tune the student and test unrelated prompts. After training on those outputs, the student still shows the same preference or misalignment even though the surface content did not obviously mention it. Step 4: ordinary filtering still misses it. Format restrictions, banned-number lists, and LLM-judge filtering can remove obvious clues while transfer still happens, making synthetic-data provenance and same-base distillation policy part of the AI supply-chain threat model.

AIA · Read the full case ↗