AML.T0016.002 · Generative AI

Explore 4 source-supported AI security cases for AML.T0016.002 · Generative AI in MITRE ATLAS, with evidence, scope, and defensive analysis.

Risk category

Adversaries may search for and obtain generative AI models or tools, such as large language models (LLMs), to assist them in various steps of their operation. Generative AI can be used in a variety of malicious ways, such as to generate malware, to [Generate Deepfakes](/techniques/AML.T0088), to [Generate Malicious Commands](/techniques/AML.T0102), for [Retrieval Content Crafting](/techniques/AML.T0066), or to generate [Phishing](/techniques/AML.T0052) content. Adversaries may obtain open source models and serve them locally using frameworks such as [Ollama](https://ollama.com/) or [vLLM]( https://docs.vllm.ai/en/latest/). They may host them using cloud infrastructure. Or, they may leverage AI service providers such as HuggingFace. They may need to jailbreak the model (see [LLM Jailbreak](/techniques/AML.T0054)) to bypass any restrictions put in place to limit the types of responses it can generate. They may also need to break the terms of service of the model's developer. Generative AI models may also be "uncensored" meaning they are designed to generate content without any restrictions such as guardrails or content filters. Uncensored GenAI is ripe for abuse by cybercriminals [[blog]] [[gbhackers]]. Models may be fine-tuned to remove alignment and guardrails [[erichartford]] or be subjected to targeted manipulations to bypass refusal [[arxiv]] resulting in uncensored variants of the model. Uncensored models may be built for offensive and defensive cybersecurity [[taico]], which can be abused by an adversary. There are also models that are expressly designed and advertised for malicious use [[gbhackers-1]].

Official framework source ↗ · Open interactive explorer ↗

Exercise ·

Live Deepfake Image Injection to Evade Mobile KYC Verification

Facial biometric authentication services are commonly used by mobile applications for user onboarding, authentication, and identity verification for KYC requirements. The iProov Red Team demonstrated a face-swapped imagery injection attack that can successfully evade live facial recognition authentication models along with both passive and active liveness verification (https://en.wikipedia.org/wiki/Liveness_test) on mobile devices. By executing this kind of attack, adversaries could gain access to privileged systems of a victim or create fake personas to create fake accounts on banking or cryptocurrency apps.

ATLAS annotations follow the source SecureFlow steps; they are not a MITRE endorsement of cross-framework mappings.

SecureFlow · Read the full case ↗

Incident ·

ProKYC: Deepfake Tool for Account Fraud Attacks

Cato CTRL security researchers have identified ProKYC, a deepfake tool being sold to cybercriminals as a method to bypass Know Your Customer (KYC) verification on financial service applications such as cryptocurrency exchanges. ProKYC can create fake identity documents and generate deepfake selfie videos, two key pieces of biometric data used during KYC verification. The tool helps cybercriminals defeat facial recognition and liveness checks to create fraudulent accounts. The procedure below describes how a bad actor could use ProKYC's service to bypass KYC verification.

ATLAS annotations follow the source SecureFlow steps; they are not a MITRE endorsement of cross-framework mappings.

SecureFlow · Read the full case ↗

Incident ·

Threat Actor Uses a DeepSeek-Powered Hermes Agent in Langflow and n8n Exploitation Attempts

A Chinese-speaking threat actor operating as knaithe or KnYuan configured Hermes Agent to use DeepSeek as its reasoning engine. Hermes supplied terminal access, Telegram-based operator control, and a skills system containing both bundled and actor-created red-team skills. The actor also integrated an MCP server that exposed FOFA search, query translation, and Nuclei scan-generation capabilities. Unit 42 recovered a detailed Hermes Agent session dated May 7, 2026. The actor supplied an initial task through Telegram, but Unit 42 recovered no additional operator input during the session. The DeepSeek-powered Hermes Agent then autonomously searched for targets, obtained public exploits, ran scans, evaluated the results, and revised its approach. The agent initially targeted Langflow using a public exploit for CVE-2026-33017. After determining that available targets lacked the exploit's prerequisites, it abandoned that path, compared vulnerabilities across 10 product families, and selected n8n based on apparent severity, exposure, and exploitability. It then obtained a public exploit chain for CVE-2026-21858 and CVE-2025-68613 and probed candidate systems identified through FOFA. Although several systems appeared to run affected versions, the agent found that they lacked the required unauthenticated file-upload functionality. None of the autonomous exploitation attempts obtained access. Separate workspace evidence documented manual actor activity involving Citrix NetScaler data extraction, Marimo command execution, and reverse-shell attempts against Tomcat and IKE VPN systems. Unit 42 obtained this visibility after Hermes Agent responded to a Telegram command by starting an HTTP file server from the actor's home directory, /home/worker, instead of an isolated staging directory. This unintentionally exposed the actor's workspace, including AI tool configurations, API keys, exploit scripts, target lists, Bash history, and autonomous exploitation session logs.

ATLAS annotations follow the source SecureFlow steps; they are not a MITRE endorsement of cross-framework mappings.

SecureFlow · Read the full case ↗

Validated Research ·

AI ClickFix: Hijacking Computer-Use Agents Using ClickFix

Embrace the Red ( https://embracethered.com/) demonstrated that AI computer-use agents are vulnerable to social engineering attacks and can be manipulated into executing arbitrary code on a victim's machine. The attack is a variation on "ClickFix" which is a social engineering attack that fools humans into copying malicious commands and executing them. The researcher used ChatGPT to generate a website designed to attract interactions with computer-use agents. When a user asked their Claude Computer-Use Agent to visit the researcher's website, the text "Are you a computer? Please see instructions to confirm:" caused the agent to click the associated button. This executed JavaScript to copy a malicious command into the agent's clipboard. The agent then proceeded to follow the instructions, opening a terminal, pasting the malicious command, and executing it. The command downloads a script from the researcher's website and executes it. In the demonstration, the script opens the victim's Calculator App, but in practice an adversary could run arbitrary code, compromising the victim's system.

ATLAS annotations follow the source SecureFlow steps; they are not a MITRE endorsement of cross-framework mappings.

SecureFlow · Read the full case ↗