Explore 3 source-supported AI security cases for AML.T0088 · Generate Deepfakes in MITRE ATLAS, with evidence, scope, and defensive analysis.
Risk category
Adversaries may use generative artificial intelligence (GenAI) to create synthetic media (i.e. imagery, video, audio, and text) that appear authentic. These "[deepfakes]( https://en.wikipedia.org/wiki/Deepfake)" may mimic a real person or depict fictional personas. Adversaries may use deepfakes for impersonation to conduct [Phishing](/techniques/AML.T0052) or to evade AI applications such as biometric identity verification systems (see [Evade AI Model](/techniques/AML.T0015)). Manipulation of media has been possible for a long time, however GenAI reduces the skill and level of effort required, allowing adversaries to rapidly scale operations to target more users or systems. It also makes real-time manipulations feasible. Adversaries may utilize open-source models and software that were designed for legitimate use cases to generate deepfakes for malicious use. However, there are some projects specifically tailored towards malicious use cases such as [ProKYC](https://www.catonetworks.com/blog/prokyc-selling-deepfake-tool-for-account-fraud-attacks/).
Facial biometric authentication services are commonly used by mobile applications for user onboarding, authentication, and identity verification for KYC requirements. The iProov Red Team demonstrated a face-swapped imagery injection attack that can successfully evade live facial recognition authentication models along with both passive and active liveness verification (https://en.wikipedia.org/wiki/Liveness_test) on mobile devices. By executing this kind of attack, adversaries could gain access to privileged systems of a victim or create fake personas to create fake accounts on banking or cryptocurrency apps.
ATLAS annotations follow the source SecureFlow steps; they are not a MITRE endorsement of cross-framework mappings.
Cato CTRL security researchers have identified ProKYC, a deepfake tool being sold to cybercriminals as a method to bypass Know Your Customer (KYC) verification on financial service applications such as cryptocurrency exchanges. ProKYC can create fake identity documents and generate deepfake selfie videos, two key pieces of biometric data used during KYC verification. The tool helps cybercriminals defeat facial recognition and liveness checks to create fraudulent accounts. The procedure below describes how a bad actor could use ProKYC's service to bypass KYC verification.
ATLAS annotations follow the source SecureFlow steps; they are not a MITRE endorsement of cross-framework mappings.
Storm-2139 built custom jailbreak tooling to bypass guardrails on Azure OpenAI Services, allowing users to generate harmful synthetic content. Microsoft reported that members of Storm-2139 scraped exposed customer credentials from public sources and used them to access accounts for generative AI services. The group developed and operated tools and services that bypassed safety safeguards, modified service capabilities, and enabled end users to generate harmful and illicit content, including non-consensual intimate images of celebrities and other sexually explicit content. The operation included creators who developed illicit tools, providers who modified and supplied those tools, and users who generated prohibited content. Microsoft pursued civil legal action.
ATLAS annotations follow the source SecureFlow steps; they are not a MITRE endorsement of cross-framework mappings.