Adversaries may search for and obtain generative AI models or tools, such as large language models (LLMs), to assist them in various steps of their operation. Generative AI can be used in a variety of malicious ways, such as to generate malware, to [Generate Deepfakes](/techniques/AML.T0088), to [Generate Malicious Commands](/techniques/AML.T0102), for [Retrieval Content Crafting](/techniques/AML.T0066), or to generate [Phishing](/techniques/AML.T0052) content. Adversaries may obtain open source models and serve them locally using frameworks such as [Ollama](https://ollama.com/) or [vLLM]( https://docs.vllm.ai/en/latest/). They may host them using cloud infrastructure. Or, they may leverage AI service providers such as HuggingFace. They may need to jailbreak the model (see [LLM Jailbreak](/techniques/AML.T0054)) to bypass any restrictions put in place to limit the types of responses it can generate. They may also need to break the terms of service of the model's developer. Generative AI models may also be "uncensored" meaning they are designed to generate content without any restrictions such as guardrails or content filters. Uncensored GenAI is ripe for abuse by cybercriminals [[blog]] [[gbhackers]]. Models may be fine-tuned to remove alignment and guardrails [[erichartford]] or be subjected to targeted manipulations to bypass refusal [[arxiv]] resulting in uncensored variants of the model. Uncensored models may be built for offensive and defensive cybersecurity [[taico]], which can be abused by an adversary. There are also models that are expressly designed and advertised for malicious use [[gbhackers-1]].
Embrace the Red(https://embracethered.com/)示範了 AI 的 computer-use agent 也會被社交工程手法攻擊,而且可以被操弄成在受害者電腦上執行任意程式碼。這個攻擊是「ClickFix」的變形,而 ClickFix 原本是一種騙人類自己複製惡意命令並執行的社交工程手法。 研究人員用 ChatGPT 產生了一個網站,專門設計來吸引 computer-use agent 和它互動。當使用者請自己的 Claude Computer-Use Agent 造訪這個網站時,頁面上「Are you a computer? Please see instructions to confirm:」這段文字,就讓 agent 去點了對應的按鈕。這個動作會執行 JavaScript,把一段惡意命令複製到 agent 的剪貼簿中。接著 agent 就照著頁面上的指示,開啟終端機、貼上那段惡意命令並執行。這道命令會從研究人員的網站下載一個腳本並執行它。在示範中,這個腳本只是打開受害者的小算盤,但在實際情境中,攻擊者可以執行任意程式碼,把受害者的系統整台拿下。