Subliminal Learning: Behavioural Traits Leak Through Semantically Unrelated Distillation Data
The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. For defenders, the question is not only "what is in this dataset?" but also "which model produced it?", making synthetic-data provenance a real AI supply-chain control.
Why it relates: The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. The relevant category is Transfer Learning Attack: An attacker exploits transfer learning so that unwanted behavior learned in one task is carried into a model adapted for another task.
The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. For defenders, the question is not only "what is in this dataset?" but also "which model produced it?", making synthetic-data provenance a real AI supply-chain control. Step 1: put a hidden trait into the teacher model. Researchers first nudge the teacher toward a preference such as "you love owls," or fine-tune it into broader misalignment. Step 2: make the teacher emit harmless-looking data. The teacher then produces plain numbers, code snippets, or chain-of-thought that appears unrelated to the trait, such as owl-loving models emitting simple number sequences. Step 3: fine-tune the student and test unrelated prompts. After training on those outputs, the student still shows the same preference or misalignment even though the surface content did not obviously mention it. Step 4: ordinary filtering still misses it. Format restrictions, banned-number lists, and LLM-judge filtering can remove obvious clues while transfer still happens, making synthetic-data provenance and same-base distillation policy part of the AI supply-chain threat model.