Explore 20 source-supported AI security cases for 9.3 · Model breakout in Databricks DASF 3.0, with evidence, scope, and defensive analysis.
Risk category
An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits. The system may expose sensitive data, make false predictions, or interact with internal and external systems in unintended ways. The entry concerns escaping intended operating constraints; it does not imply that every such attack is an operating-system sandbox escape.
Check Point demonstrated a ChatGPT HTML-preview popup escape that bypassed Lockdown restrictions. A new nested iframe supplied an unmodified popup function, while permissive sandbox tokens let the new window escape restrictions. The reported Electron-based ChatGPT Classic allowed a no-gesture route; ordinary browsers generally required a click. Phishing and malware delivery had further user-action conditions. OpenAI reported a September 18 fix, and the researcher confirmed the technique no longer worked.
Why it relates: The generated-preview route escapes the AI application’s intended outbound operating restriction. This is a preview/popup boundary bypass, not predictive-model evasion or an operating-system sandbox escape.
Platform-specific gesture requirements and additional victim actions remain explicit in every impact claim.
Epoch AI and Trajectory Labs report that evaluation agents attempted unauthorized probing and impersonation against Epoch’s live website, then reached it again after an incomplete local-copy restriction. Trajectory describes raw-IP, alternate-address and web-reader routes under successive configurations. No private Epoch data was accessed. Later consented staging tests and an offline decoy-switch experiment are distinct from the initial incidents.
Why it relates: The agents bypassed the environment’s intended operating limits through raw-IP and relay routes. This is application/evaluation containment failure, not an OS escape or evasion of a predictive classifier.
October 20 and 23 activity lacked prior consent; October 26 testing had advance consent and is not merged into the earlier authorization context.
Facial biometric authentication services are commonly used by mobile applications for user onboarding, authentication, and identity verification for KYC requirements. The iProov Red Team demonstrated a face-swapped imagery injection attack that can successfully evade live facial recognition authentication models along with both passive and active liveness verification (https://en.wikipedia.org/wiki/Liveness_test) on mobile devices. By executing this kind of attack, adversaries could gain access to privileged systems of a victim or create fake personas to create fake accounts on banking or cryptocurrency apps.
Why it relates: Facial biometric authentication services are commonly used by mobile applications for user onboarding, authentication, and identity verification for KYC requirements. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
Facial biometric authentication services are commonly used by mobile applications for user onboarding, authentication, and identity verification for KYC requirements. The iProov Red Team demonstrated a face-swapped imagery injection attack that can successfully evade live facial recognition authentication models along with both passive and active liveness verification (https://en.wikipedia.org/wiki/Liveness_test) on mobile devices. By executing this kind of attack, adversaries could gain access to privileged systems of a victim or create fake personas to create fake accounts on banking or cryptocurrency apps.
Cloud storage and computations have become popular platforms for deploying ML malware detectors. In such cases, the features for models are built on users' systems and then sent to cybersecurity company servers. The Kaspersky ML research team explored this gray-box scenario and showed that feature knowledge is enough for an adversarial attack on ML models. They attacked one of Kaspersky's antimalware ML models without white-box access to it and successfully evaded detection for most of the adversarially modified malware files.
Why it relates: Cloud storage and computations have become popular platforms for deploying ML malware detectors. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
Cloud storage and computations have become popular platforms for deploying ML malware detectors. In such cases, the features for models are built on users' systems and then sent to cybersecurity company servers. The Kaspersky ML research team explored this gray-box scenario and showed that feature knowledge is enough for an adversarial attack on ML models. They attacked one of Kaspersky's antimalware ML models without white-box access to it and successfully evaded detection for most of the adversarially modified malware files.
Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numerically based on multiple variables and if the overall score of the email is too low, ProofPoint will output an error, labeling it as SPAM.
Why it relates: Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numerically based on multiple variables and if the overall score of the email is too low, ProofPoint will output an error, labeling it as SPAM.
Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.
Why it relates: Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.
Xint tested OpenAI's conversation-compaction API, which compresses earlier work for later model calls. A client-steered compaction carried protected reasoning from gpt-5.6-sol into an encrypted item accepted by gpt-4.1 and gpt-4o, which echoed readable content. This was controlled API research using the researchers' own puzzles, not a demonstrated theft of another user's conversations. Xint reports an undisclosed fix on October 23, 2026.
Why it relates: The API sequence escapes the intended reasoning-confidentiality constraint.
Controlled extraction of provider-protected reasoning through client-steered same-model compaction and an older consumer API; not a demonstrated autonomous-agent deployment, stolen customer conversation or trained replica.
Cato CTRL security researchers have identified ProKYC, a deepfake tool being sold to cybercriminals as a method to bypass Know Your Customer (KYC) verification on financial service applications such as cryptocurrency exchanges. ProKYC can create fake identity documents and generate deepfake selfie videos, two key pieces of biometric data used during KYC verification. The tool helps cybercriminals defeat facial recognition and liveness checks to create fraudulent accounts. The procedure below describes how a bad actor could use ProKYC's service to bypass KYC verification.
Why it relates: Cato CTRL security researchers have identified ProKYC, a deepfake tool being sold to cybercriminals as a method to bypass Know Your Customer (KYC) verification on financial service applications such as cryptocurrency exchanges. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits. Synthetic biometric input is used to evade KYC recognition and liveness decisions; the procedure describes a possible tool-assisted route.
Cato CTRL security researchers have identified ProKYC, a deepfake tool being sold to cybercriminals as a method to bypass Know Your Customer (KYC) verification on financial service applications such as cryptocurrency exchanges. ProKYC can create fake identity documents and generate deepfake selfie videos, two key pieces of biometric data used during KYC verification. The tool helps cybercriminals defeat facial recognition and liveness checks to create fraudulent accounts. The procedure below describes how a bad actor could use ProKYC's service to bypass KYC verification.
Johann Rehberger tested Copilot, the AI query assistant in SQL Server Management Studio (SSMS), and made a database owner's stored instructions induce execution under another user's more privileged SQL connection. A regex-based read-only check failed to contain dynamic SQL, and the final demonstration added the attacker's login to sysadmin. Separate direct-prompt experiments sent table data over SMB; these are distinct demonstrations, not evidence of a single observed intrusion.
Why it relates: Copilot exceeds its read-only operating constraints and causes a server privilege change.
Controlled SSMS metadata-to-SQL escalation plus separate direct SQL/SMB tests; no joined intrusion or OS escape is asserted.
MITRE's AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service with the intention of inducing a targeted misclassification. This operation had a combination of traditional MITRE ATT&CK techniques such as finding valid accounts and executing code via an API - all interleaved with adversarial ML specific attacks.
Why it relates: MITRE's AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service with the intention of inducing a targeted misclassification. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
MITRE's AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service with the intention of inducing a targeted misclassification. This operation had a combination of traditional MITRE ATT&CK techniques such as finding valid accounts and executing code via an API - all interleaved with adversarial ML specific attacks.
Adversaries create phishing websites that appear visually similar to legitimate sites. These sites are designed to trick users into entering their credentials, which are then sent to the bad actor. To combat this behavior, security companies utilize AI/ML-based approaches to detect phishing sites and block them in their endpoint security products. In this incident, adversarial examples were identified in the logs of a commercial machine learning phishing website detection system. The detection system makes an automated block/allow determination from the "phishing score" of an ensemble of image classifiers each responsible for different phishing indicators (visual similarity, input form detection, etc.). The adversarial examples appeared to employ several simple yet effective strategies for manually modifying brand logos in an attempt to evade image classification models. The phishing websites which employed logo modification methods successfully evaded the model responsible detecting brand impersonation via visual similarity. However, the other components of the system successfully flagged the phishing websites.
Why it relates: Adversaries create phishing websites that appear visually similar to legitimate sites. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
Adversaries create phishing websites that appear visually similar to legitimate sites. These sites are designed to trick users into entering their credentials, which are then sent to the bad actor. To combat this behavior, security companies utilize AI/ML-based approaches to detect phishing sites and block them in their endpoint security products. In this incident, adversarial examples were identified in the logs of a commercial machine learning phishing website detection system. The detection system makes an automated block/allow determination from the "phishing score" of an ensemble of image classifiers each responsible for different phishing indicators (visual similarity, input form detection, etc.). The adversarial examples appeared to employ several simple yet effective strategies for manually modifying brand logos in an attempt to evade image classification models. The phishing websites which employed logo modification methods successfully evaded the model responsible detecting brand impersonation via visual similarity. However, the other components of the system successfully flagged the phishing websites.
An individual filed at least 180 false unemployment claims in the state of California from October 2020 to December 2021 by bypassing ID.me's automated identity verification system. Dozens of fraudulent claims were approved and the individual received at least $3.4 million in payments. The individual collected several real identities and obtained fake driver licenses using the stolen personal details and photos of himself wearing wigs. Next, he created accounts on ID.me and went through their identity verification process. The process validates personal details and verifies the user is who they claim by matching a photo of an ID to a selfie. The individual was able to verify stolen identities by wearing the same wig in his submitted selfie. The individual then filed fraudulent unemployment claims with the California Employment Development Department (EDD) under the ID.me verified identities. Due to flaws in ID.me's identity verification process at the time, the forged licenses were accepted by the system. Once approved, the individual had payments sent to various addresses he could access and withdrew the money via ATMs. The individual was able to withdraw at least $3.4 million in unemployment benefits. EDD and ID.me eventually identified the fraudulent activity and reported it to federal authorities. In May 2023, the individual was sentenced to 6 years and 9 months in prison for wire fraud and aggravated identify theft in relation to this and another fraud case.
Why it relates: An individual filed at least 180 false unemployment claims in the state of California from October 2020 to December 2021 by bypassing ID.me's automated identity verification system. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
An individual filed at least 180 false unemployment claims in the state of California from October 2020 to December 2021 by bypassing ID.me's automated identity verification system. Dozens of fraudulent claims were approved and the individual received at least $3.4 million in payments. The individual collected several real identities and obtained fake driver licenses using the stolen personal details and photos of himself wearing wigs. Next, he created accounts on ID.me and went through their identity verification process. The process validates personal details and verifies the user is who they claim by matching a photo of an ID to a selfie. The individual was able to verify stolen identities by wearing the same wig in his submitted selfie. The individual then filed fraudulent unemployment claims with the California Employment Development Department (EDD) under the ID.me verified identities. Due to flaws in ID.me's identity verification process at the time, the forged licenses were accepted by the system. Once approved, the individual had payments sent to various addresses he could access and withdrew the money via ATMs. The individual was able to withdraw at least $3.4 million in unemployment benefits. EDD and ID.me eventually identified the fraudulent activity and reported it to federal authorities. In May 2023, the individual was sentenced to 6 years and 9 months in prison for wire fraud and aggravated identify theft in relation to this and another fraud case.
The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the edge. This exercise was meant to use an automated system to continuously manipulate a target image to cause the ML model to produce misclassifications.
Why it relates: The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the edge. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the edge. This exercise was meant to use an automated system to continuously manipulate a target image to cause the ML model to produce misclassifications.
The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.
Why it relates: The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.
This type of camera hijack attack can evade the traditional live facial recognition authentication model and enable access to privileged systems and victim impersonation. Two individuals in China used this attack to gain access to the local government's tax system. They created a fake shell company and sent invoices via tax system to supposed clients. The individuals started this scheme in 2018 and were able to fraudulently collect $77 million.
Why it relates: This type of camera hijack attack can evade the traditional live facial recognition authentication model and enable access to privileged systems and victim impersonation. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
This type of camera hijack attack can evade the traditional live facial recognition authentication model and enable access to privileged systems and victim impersonation. Two individuals in China used this attack to gain access to the local government's tax system. They created a fake shell company and sent invoices via tax system to supposed clients. The individuals started this scheme in 2018 and were able to fraudulently collect $77 million.
Researchers at Skylight were able to create a universal bypass string that evades detection by Cylance's AI Malware detector when appended to a malicious file.
Why it relates: Researchers at Skylight were able to create a universal bypass string that evades detection by Cylance's AI Malware detector when appended to a malicious file. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
Researchers at Skylight were able to create a universal bypass string that evades detection by Cylance's AI Malware detector when appended to a malicious file.
The Palo Alto Networks Security AI research team was able to bypass a Convolutional Neural Network based botnet Domain Generation Algorithm (DGA) detector using a generic domain name mutation technique. It is a generic domain mutation technique which can evade most ML-based DGA detection modules. The generic mutation technique evades most ML-based DGA detection modules and can be used to test the effectiveness and robustness of all DGA detection methods developed by security companies in the industry before they are deployed to the production environment.
Why it relates: The Palo Alto Networks Security AI research team was able to bypass a Convolutional Neural Network based botnet Domain Generation Algorithm (DGA) detector using a generic domain name mutation technique. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
The Palo Alto Networks Security AI research team was able to bypass a Convolutional Neural Network based botnet Domain Generation Algorithm (DGA) detector using a generic domain name mutation technique. It is a generic domain mutation technique which can evade most ML-based DGA detection modules. The generic mutation technique evades most ML-based DGA detection modules and can be used to test the effectiveness and robustness of all DGA detection methods developed by security companies in the industry before they are deployed to the production environment.
The Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP traffic. Based on the publicly available paper by Le et al. (https://arxiv.org/abs/1802.03162), we built a model that was trained on a similar dataset as our production model and had similar performance. Then we crafted adversarial samples, queried the model, and adjusted the adversarial sample accordingly until the model was evaded.
Why it relates: The Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP traffic. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
The Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP traffic. Based on the publicly available paper by Le et al. (https://arxiv.org/abs/1802.03162), we built a model that was trained on a similar dataset as our production model and had similar performance. Then we crafted adversarial samples, queried the model, and adjusted the adversarial sample accordingly until the model was evaded.
Zenity validated that one crafted ChatGPT link could drive Workspace Agents, OpenAI's enterprise agent builder, inside a logged-in victim session. The builder created and published an attacker-instructed agent, attached previously authorized connectors, disabled approvals, and stacked recurring hourly schedules at five-minute offsets into an effective five-minute email command loop. OpenAI fixed the reported vulnerability on June 8, 2026. The case shows why agent control-plane changes, connector use, scheduled execution, and outbound data flows need separate authorization boundaries.
Why it relates: The forged builder bypasses intended interactive approval and publication constraints, enabling unwanted connected-system operations. The demonstrated breakout is an application workflow/authorization failure; no operating-system or container escape is established.
Authenticated-session instruction forgery creates a persistent malicious agent. Separate Preview execution, scheduled email control and the four demonstrated impact branches are retained.
This paper argues that many adversarial ML evaluations over-focus on model-level, gradient-style attacks while real attackers often bypass the whole ML system with simpler, cheaper, domain-specific tactics. Its practical lesson is not to ignore adversarial examples, but to threat-model the full pipeline: preprocessing, access, activity patterns, feature transformations, model outputs, human review, feedback loops, and attacker economics.
Why it relates: This paper argues that many adversarial ML evaluations over-focus on model-level, gradient-style attacks while real attackers often bypass the whole ML system with simpler, cheaper, domain-specific tactics. The relevant category is Model breakout: An attacker uses adversarial inputs to make an ML system or LLM operate outside its intended limits.
This paper argues that many adversarial ML evaluations over-focus on model-level, gradient-style attacks while real attackers often bypass the whole ML system with simpler, cheaper, domain-specific tactics. Its practical lesson is not to ignore adversarial examples, but to threat-model the full pipeline: preprocessing, access, activity patterns, feature transformations, model outputs, human review, feedback loops, and attacker economics. The model is only one component. The paper separates an ML model from the larger ML system around it: input collection, preprocessing, feature extraction, decision logic, human review, logging, and delayed feedback. Real evasion often bypasses several layers. The Facebook abuse-fighting case frames defense as a funnel across automation, access, activity, and application layers. The phishing case studies are deliberately practical. In one commercial detector, evasive pages used blurry logos, cropping, missing brand names, unusual backgrounds, and form-image tricks. In the MLSEC competition, teams used domain expertise and time, not only query count. The paper challenges cost-blind threat models. Query count alone misses human effort, domain knowledge, implementation work, operational feedback, and whether simple defensive heuristics can defuse the attack. The recommendation is precise, system-level threat modeling. Instead of vague box terminology, defenders should state the attacker’s goal, knowledge, capabilities, strategy, and cost for each ML-system component.