ML08:2023 · Model Skewing

Explore 1 source-supported AI security cases for ML08:2023 · Model Skewing in OWASP ML Top 10 (2023), with evidence, scope, and defensive analysis.

Risk category

An attacker changes the distribution of training data so that the model's learned decisions become biased toward the attacker's desired outcome. This can happen through a compromised feedback loop, where fabricated feedback is accepted as genuine and used in subsequent model updates. The risk concerns adversarial manipulation of what the model learns over time, rather than ordinary changes in real-world data alone.

Official framework source ↗ · Open interactive explorer ↗

Incident ·

Tay Poisoning

Microsoft created Tay, a Twitter chatbot designed to engage and entertain users. While previous chatbots used pre-programmed scripts to respond to prompts, Tay's machine learning capabilities allowed it to be directly influenced by its conversations. A coordinated attack encouraged malicious users to tweet abusive and offensive language at Tay, which eventually led to Tay generating similarly inflammatory content towards other users. Microsoft decommissioned Tay within 24 hours of its launch and issued a public apology with lessons learned from the bot's failure.

Why it relates: Microsoft created Tay, a Twitter chatbot designed to engage and entertain users. The relevant category is Model Skewing: An attacker changes the distribution of training data so that the model's learned decisions become biased toward the attacker's desired outcome.

Microsoft created Tay, a Twitter chatbot designed to engage and entertain users. While previous chatbots used pre-programmed scripts to respond to prompts, Tay's machine learning capabilities allowed it to be directly influenced by its conversations. A coordinated attack encouraged malicious users to tweet abusive and offensive language at Tay, which eventually led to Tay generating similarly inflammatory content towards other users. Microsoft decommissioned Tay within 24 hours of its launch and issued a public apology with lessons learned from the bot's failure.

SecureFlow · Read the full case ↗