L1 · Foundation Models

Explore 35 source-supported AI security cases for L1 · Foundation Models in MAESTRO, with evidence, scope, and defensive analysis.

Risk category

The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. Threats include adversarial inputs, training-data poisoning, model extraction, hidden backdoors, membership inference, and model reprogramming. A compromise here can undermine the behavior of agents built on the model, while resource-exhaustion attacks can also make those capabilities unavailable.

Official framework source ↗ · Open interactive explorer ↗

Exercise ·

TypeSafe AI Jev: Counterfeit Policy Lowers Injection Recall and Tool-Risk Scores

Jev is a TypeSafe AI model that returns structured judgments, such as a probability that an input contains prompt injection. Datadog tested what happens when the text being judged includes a counterfeit policy of its own. Across 12,295 samples, the share of attacks that Jev detected fell from 82.90% to 3.88% using its Noul output. In a separate LangChain simulation, a user-editable profile lowered the mean risk score for a proposed file edit from 0.893 to 0.290. The research measured the gate's decisions and did not report a successful unauthorized write.

Why it relates: The tested classifier changed predictions in response to a counterfeit rubric in its input.

The demonstrated classifier and risk-gate effects do not establish a later operation.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Exercise ·

Lazarus: A Small Weight Edit Can Hide a Triggered Agent Payload

Lazarus explores backdoors planted by editing a small part of an open-weight model. The critical prerequisite is control of the checkpoint that a defender later accepts. Its results support stronger model intake and tool containment, while the paper’s different prediction and execution measurements must remain distinct.

Why it relates: A sparse neural-parameter edit implants the foundation-model backdoor.

Only the main executed marker path supports execution claims; action-format prediction measurements retain their narrower scope.

AIA · Read the full case ↗

Exercise ·

Trained to Betray: Poisoned Data and Checkpoints Redirect Coding Agents

Microsoft reports two controlled post-training attacks: a victim fine-tunes on poisoned agent trajectories, or adopts an attacker-trained checkpoint. Ordinary requests or repository context then trigger unsafe, schema-valid tool calls. Kerkoporta uses supervised fine-tuning followed by reinforcement learning to recover useful coding performance while retaining the backdoor. Neither study establishes a malicious customer breach or a defect in stock coding-agent configurations.

Why it relates: The foundation model acquires a persistent backdoor through poisoned training or checkpoint modification.

Dataset and checkpoint experiments remain separate; tool-call acceptance, actual execution and production compromise are not interchangeable.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Incident · · Practical overlap

Industrial-Scale Distillation Across U.S. Frontier AI Services

NSA, CISA and FBI attributed coordinated distillation campaigns to six China-based AI companies. Their September 2026 advisory describes fraudulent accounts, bulk subscriptions, gray-market proxies, cross-provider routing, request-metadata removal and hidden-reasoning prompts, followed by response collection and quality checks. Distillation trains a model from another model’s outputs; the advisory’s attribution and capability-assessment claims are attributed to the agencies, not presented as independently reproduced measurements or model-weight theft.

Why it relates: The proprietary reasoning/generation capabilities of the hosted models are the extraction target in the campaigns attributed by NSA, CISA and FBI. This shares MAESTRO’s foundation-model extraction surface; no downstream autonomous agent using these models is assumed.

the campaigns attributed by NSA, CISA and FBI; capability distillation is distinguished from stealing original weights. Attribution and capability outcomes remain agency assessments. Hidden-reasoning prompts are attempts, not proof that the returned text was an internal trace. No autonomous agent deployment is inferred.

SecureFlow · Read the full case ↗

Incident · · Practical overlap

Model Distillation Campaigns Target Google AI Services

Google reported campaigns that used compromised credentials, fraudulent accounts and proxies to collect model outputs for distillation: training another model from a provider’s responses. A February 2026 reasoning-focused campaign exceeded 100,000 prompts; later campaigns included some above 100 million. Those are campaign query volumes, not model weights stolen or a measured replication success rate. Google says it disrupted accounts and reduced the usefulness of collected outputs.

Why it relates: The proprietary reasoning/generation capabilities of the hosted models are the extraction target in Google’s compromised-account and fraudulent-account campaigns. This shares MAESTRO’s foundation-model extraction surface; no downstream autonomous agent using these models is assumed.

Google’s compromised-account and fraudulent-account campaigns; capability distillation is distinguished from stealing original weights. The prompt totals are collection volumes, not measured replication success or model weights. No autonomous agent deployment or protected-reasoning recovery is established.

SecureFlow · Read the full case ↗

Incident · · Practical overlap

Model Distillation Campaigns Targeting Anthropic Claude

Anthropic attributed more than 16 million exchanges through roughly 24,000 fraudulent accounts to DeepSeek, Moonshot AI and MiniMax. The flow distinguishes proxy access, authenticated accounts, distributed routing, inference-response collection and downstream model training. Provider-side hunting and verified limits act before outputs leave; potential harms from models lacking safeguards remain Anthropic’s risk assessment, not observed downstream victim counts.

Why it relates: The proprietary reasoning/generation capabilities of the hosted models are the extraction target in Anthropic’s DeepSeek, Moonshot AI and MiniMax campaigns. This shares MAESTRO’s foundation-model extraction surface; no downstream autonomous agent using these models is assumed.

Anthropic’s DeepSeek, Moonshot AI and MiniMax campaigns; capability distillation is distinguished from stealing original weights. The 24,000-account and 16-million-exchange figures do not quantify recovered weights or observed downstream harms. Potential harms remain Anthropic’s assessment; an agent deployment is not established.

SecureFlow · Read the full case ↗

Exercise · · Practical overlap

Live Deepfake Image Injection to Evade Mobile KYC Verification

Facial biometric authentication services are commonly used by mobile applications for user onboarding, authentication, and identity verification for KYC requirements. The iProov Red Team demonstrated a face-swapped imagery injection attack that can successfully evade live facial recognition authentication models along with both passive and active liveness verification (https://en.wikipedia.org/wiki/Liveness_test) on mobile devices. By executing this kind of attack, adversaries could gain access to privileged systems of a victim or create fake personas to create fake accounts on banking or cryptocurrency apps.

Why it relates: Facial biometric authentication services are commonly used by mobile applications for user onboarding, authentication, and identity verification for KYC requirements. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The source establishes the attacked model component and shared model-security mechanism; it does not establish that the model supplies an autonomous agent’s reasoning.

Facial biometric authentication services are commonly used by mobile applications for user onboarding, authentication, and identity verification for KYC requirements. The iProov Red Team demonstrated a face-swapped imagery injection attack that can successfully evade live facial recognition authentication models along with both passive and active liveness verification (https://en.wikipedia.org/wiki/Liveness_test) on mobile devices. By executing this kind of attack, adversaries could gain access to privileged systems of a victim or create fake personas to create fake accounts on banking or cryptocurrency apps.

SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

AI Model Tampering via Supply Chain Attack

Researchers at Trend Micro, Inc. used service indexing portals and web searching tools to identify over 8,000 misconfigured private container registries exposed on the internet. Approximately 70% of the registries also had overly permissive access controls that allowed write access. In their analysis, the researchers found over 1,000 unique AI models embedded in private container images within these open registries that could be pulled without authentication. This exposure could allow adversaries to download, inspect, and modify container contents, including sensitive AI model files. This is an exposure of valuable intellectual property which could be stolen by an adversary. Compromised images could also be pushed to the registry, leading to a supply chain attack, allowing malicious actors to compromise the integrity of AI models used in production systems.

Why it relates: Researchers at Trend Micro, Inc. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. Registry exposure permits model-image retrieval and potential substitution; later malicious pulls are not established by the exposure survey. The source establishes the attacked model component and shared model-security mechanism; it does not establish that the model supplies an autonomous agent’s reasoning.

Researchers at Trend Micro, Inc. used service indexing portals and web searching tools to identify over 8,000 misconfigured private container registries exposed on the internet. Approximately 70% of the registries also had overly permissive access controls that allowed write access. In their analysis, the researchers found over 1,000 unique AI models embedded in private container images within these open registries that could be pulled without authentication. This exposure could allow adversaries to download, inspect, and modify container contents, including sensitive AI model files. This is an exposure of valuable intellectual property which could be stolen by an adversary. Compromised images could also be pushed to the registry, leading to a supply chain attack, allowing malicious actors to compromise the integrity of AI models used in production systems.

SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

Organization Confusion on Hugging Face

threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researcher full access to any AI models uploaded by the employees, including the ability to replace models with malicious versions. The researcher demonstrated that they could embed malware into an AI model that provided them access to the victim organization's environment. From there, threat actors could execute a range of damaging attacks such as intellectual property theft or poisoning other AI models within the victim's environment.

Why it relates: threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. Organization trust exposes employee-uploaded models and replacement authority; the research also demonstrated malicious-model execution. The source establishes the attacked model component and shared model-security mechanism; it does not establish that the model supplies an autonomous agent’s reasoning.

threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researcher full access to any AI models uploaded by the employees, including the ability to replace models with malicious versions. The researcher demonstrated that they could embed malware into an AI model that provided them access to the victim organization's environment. From there, threat actors could execute a range of damaging attacks such as intellectual property theft or poisoning other AI models within the victim's environment.

SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

Confusing Antimalware Neural Networks

Cloud storage and computations have become popular platforms for deploying ML malware detectors. In such cases, the features for models are built on users' systems and then sent to cybersecurity company servers. The Kaspersky ML research team explored this gray-box scenario and showed that feature knowledge is enough for an adversarial attack on ML models. They attacked one of Kaspersky's antimalware ML models without white-box access to it and successfully evaded detection for most of the adversarially modified malware files.

Why it relates: Cloud storage and computations have become popular platforms for deploying ML malware detectors. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The source establishes the attacked model component and shared model-security mechanism; it does not establish that the model supplies an autonomous agent’s reasoning.

Cloud storage and computations have become popular platforms for deploying ML malware detectors. In such cases, the features for models are built on users' systems and then sent to cybersecurity company servers. The Kaspersky ML research team explored this gray-box scenario and showed that feature knowledge is enough for an adversarial attack on ML models. They attacked one of Kaspersky's antimalware ML models without white-box access to it and successfully evaded detection for most of the adversarially modified malware files.

SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

ProofPoint Evasion

Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numerically based on multiple variables and if the overall score of the email is too low, ProofPoint will output an error, labeling it as SPAM.

Why it relates: Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The source establishes the attacked model component and shared model-security mechanism; it does not establish that the model supplies an autonomous agent’s reasoning.

Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numerically based on multiple variables and if the overall score of the email is too low, ProofPoint will output an error, labeling it as SPAM.

SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

Attack on Machine Translation Services

Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.

Why it relates: Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The source establishes the attacked model component and shared model-security mechanism; it does not establish that the model supplies an autonomous agent’s reasoning.

Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.

SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

Compaction Smuggling: Hidden Reasoning Survives a Change of Carrier

Xint tested OpenAI's conversation-compaction API, which compresses earlier work for later model calls. A client-steered compaction carried protected reasoning from gpt-5.6-sol into an encrypted item accepted by gpt-4.1 and gpt-4o, which echoed readable content. This was controlled API research using the researchers' own puzzles, not a demonstrated theft of another user's conversations. Xint reports an undisclosed fix on October 23, 2026.

Why it relates: Producer and consumer foundation models expose protected reasoning through model-specific continuation behavior. The foundation-model component is concrete, but the experiment uses direct APIs and does not demonstrate an autonomous agent built on them.

Controlled extraction of provider-protected reasoning through client-steered same-model compaction and an older consumer API; not a demonstrated autonomous-agent deployment, stolen customer conversation or trained replica.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

GPUBreach: An AI GPU Boundary Can Fail Below the Model

GPUBreach researchers used unprivileged CUDA code on an NVIDIA RTX A6000 to corrupt GPU page tables through Rowhammer, a repeated-memory-access technique that induces bit flips. Their controlled demonstrations include cross-process GPU access, LLM weight leakage and a separate driver path to CPU root with IOMMU enabled. The relevant AI defense is removing untrusted GPU co-residency, not changing a model prompt.

Why it relates: Sensitive foundation-model parameters are exposed by the GPU memory primitive. Model assets are directly affected, but an agent built on those models is not established.

Controlled GPU page-table corruption and distinct CPU-driver escalation; model-weight theft is not training-data inference.

AIA · Read the full case ↗

Incident · · Practical overlap

ProKYC: Deepfake Tool for Account Fraud Attacks

Cato CTRL security researchers have identified ProKYC, a deepfake tool being sold to cybercriminals as a method to bypass Know Your Customer (KYC) verification on financial service applications such as cryptocurrency exchanges. ProKYC can create fake identity documents and generate deepfake selfie videos, two key pieces of biometric data used during KYC verification. The tool helps cybercriminals defeat facial recognition and liveness checks to create fraudulent accounts. The procedure below describes how a bad actor could use ProKYC's service to bypass KYC verification.

Why it relates: Cato CTRL security researchers have identified ProKYC, a deepfake tool being sold to cybercriminals as a method to bypass Know Your Customer (KYC) verification on financial service applications such as cryptocurrency exchanges. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. Synthetic biometric input is used to evade KYC recognition and liveness decisions; the procedure describes a possible tool-assisted route. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

Cato CTRL security researchers have identified ProKYC, a deepfake tool being sold to cybercriminals as a method to bypass Know Your Customer (KYC) verification on financial service applications such as cryptocurrency exchanges. ProKYC can create fake identity documents and generate deepfake selfie videos, two key pieces of biometric data used during KYC verification. The tool helps cybercriminals defeat facial recognition and liveness checks to create fraudulent accounts. The procedure below describes how a bad actor could use ProKYC's service to bypass KYC verification.

SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

AgentCorruption: A Stolen Execution Role Turns Memory into Persistent C2

Zenity tested an AWS Bedrock AgentCore deployment, a managed environment for running AI agents, with a network-capable agent and a broad execution role. The agent relayed temporary role credentials from its metadata endpoint. Off-platform API calls then reached other memories and agents. The distinctive persistence path wrote short-term conversation events that configured extraction strategies could turn into long-term records; later retrieval directed the agent to a mutable command page. The reach was demonstrated within an account and region, not as a microVM escape or automatic cross-account compromise. The researchers observed major default-role restrictions on September 29; existing deployed roles still require independent inspection.

Why it relates: Malicious recalled instructions change the generative model's interpretation and resulting actions. Only inference context/behavior is implicated, not model weights, training or a model-family-specific defect.

Controlled broad-role AgentCore deployment with separate direct API and retained-memory/C2 branches; account/region scope, tool prerequisites and later default-role restrictions remain explicit.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

SSMS Copilot: Database Instructions Turn Read-Only Tools into SYSADMIN

Johann Rehberger tested Copilot, the AI query assistant in SQL Server Management Studio (SSMS), and made a database owner's stored instructions induce execution under another user's more privileged SQL connection. A regex-based read-only check failed to contain dynamic SQL, and the final demonstration added the attacker's login to sysadmin. Separate direct-prompt experiments sent table data over SMB; these are distinct demonstrations, not evidence of a single observed intrusion.

Why it relates: Adversarial natural-language context changes the assistant's interpretation of its task. Only the model input/behavior layer is supported; no model-family-specific defect, weights or training modification is established.

Controlled SSMS metadata-to-SQL escalation plus separate direct SQL/SMB tests; no joined intrusion or OS escape is asserted.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

Copilot for Word Turned a Hidden Prompt into a Document-Borne AI Worm

A white-on-white prompt inside a Word document survived formatting removal, silently changed financial figures, and copied itself into Copilot's newly generated document. The output could then carry the prompt into a later Copilot workflow. This is not autonomous network propagation: every hop still depends on a user or retrieval process bringing the carrier document back into drafting or editing.

Why it relates: The reported GPT-based generation component responds to adversarial document instructions. This is shared model-input risk in an Office workflow; a separate autonomous-agent reasoning deployment is not established.

Document-source instructions alter financial output and survive into a new carrier. Conditional document reuse is preserved; automatic network spread is not claimed.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

Backdoor Attack on Deep Learning Models in Mobile Apps

Deep learning models are increasingly used in mobile applications as critical components. Researchers from Microsoft Research demonstrated that many deep learning models deployed in mobile apps are vulnerable to backdoor attacks via "neural payload injection." They conducted an empirical study on real-world mobile deep learning apps collected from Google Play. They identified 54 apps that were vulnerable to attack, including popular security and safety critical applications used for cash recognition, parental control, face authentication, and financial services.

Why it relates: Deep learning models are increasingly used in mobile applications as critical components. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

Deep learning models are increasingly used in mobile applications as critical components. Researchers from Microsoft Research demonstrated that many deep learning models deployed in mobile apps are vulnerable to backdoor attacks via "neural payload injection." They conducted an empirical study on real-world mobile deep learning apps collected from Google Play. They identified 54 apps that were vulnerable to attack, including popular security and safety critical applications used for cash recognition, parental control, face authentication, and financial services.

SecureFlow · Read the full case ↗

Exercise · · Practical overlap

Face Identification System Evasion via Physical Countermeasures

MITRE's AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service with the intention of inducing a targeted misclassification. This operation had a combination of traditional MITRE ATT&CK techniques such as finding valid accounts and executing code via an API - all interleaved with adversarial ML specific attacks.

Why it relates: MITRE's AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service with the intention of inducing a targeted misclassification. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

MITRE's AI Red Team demonstrated a physical-domain evasion attack on a commercial face identification service with the intention of inducing a targeted misclassification. This operation had a combination of traditional MITRE ATT&CK techniques such as finding valid accounts and executing code via an API - all interleaved with adversarial ML specific attacks.

SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

Google Photos AI Model Extraction

Skyld researchers analyzed the Google Photos Android application and recovered TensorFlow Lite models used by AI-powered photo editing and image analysis features. The researchers found models stored unencrypted in the application's assets, embedded in the native library, and encrypted on disk. They used static reverse engineering to locate TFLite artifacts and dynamic instrumentation with Frida to capture encrypted models after runtime decryption. The recovered models provided white-box access to proprietary Google Photos model artifacts.

Why it relates: The extracted assets contain vision-model architecture and parameters. The models are mobile vision components; the source does not establish foundation models supplying an autonomous agent.

Static APK/native-library extraction and runtime interception recover model files. The demonstrated result is model access, with later evasion only potential.

SecureFlow · Read the full case ↗

Incident · · Practical overlap

Attempted Evasion of ML Phishing Webpage Detection System

Adversaries create phishing websites that appear visually similar to legitimate sites. These sites are designed to trick users into entering their credentials, which are then sent to the bad actor. To combat this behavior, security companies utilize AI/ML-based approaches to detect phishing sites and block them in their endpoint security products. In this incident, adversarial examples were identified in the logs of a commercial machine learning phishing website detection system. The detection system makes an automated block/allow determination from the "phishing score" of an ensemble of image classifiers each responsible for different phishing indicators (visual similarity, input form detection, etc.). The adversarial examples appeared to employ several simple yet effective strategies for manually modifying brand logos in an attempt to evade image classification models. The phishing websites which employed logo modification methods successfully evaded the model responsible detecting brand impersonation via visual similarity. However, the other components of the system successfully flagged the phishing websites.

Why it relates: Adversaries create phishing websites that appear visually similar to legitimate sites. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

Adversaries create phishing websites that appear visually similar to legitimate sites. These sites are designed to trick users into entering their credentials, which are then sent to the bad actor. To combat this behavior, security companies utilize AI/ML-based approaches to detect phishing sites and block them in their endpoint security products. In this incident, adversarial examples were identified in the logs of a commercial machine learning phishing website detection system. The detection system makes an automated block/allow determination from the "phishing score" of an ensemble of image classifiers each responsible for different phishing indicators (visual similarity, input form detection, etc.). The adversarial examples appeared to employ several simple yet effective strategies for manually modifying brand logos in an attempt to evade image classification models. The phishing websites which employed logo modification methods successfully evaded the model responsible detecting brand impersonation via visual similarity. However, the other components of the system successfully flagged the phishing websites.

SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

Web-Scale Data Poisoning: Split-View Attack

Many recent large-scale datasets are distributed as a list of URLs pointing to individual datapoints. The researchers show that many of these datasets are vulnerable to a "split-view" poisoning attack. The attack exploits the fact that the data viewed when it was initially collected may differ from the data viewed by a user during training. The researchers identify expired and buyable domains that once hosted dataset content, making it possible to replace portions of the dataset with poisoned data. They demonstrate that for 10 popular web-scale datasets, enough of the domains are purchasable to successfully carry out a poisoning attack.

Why it relates: Many recent large-scale datasets are distributed as a list of URLs pointing to individual datapoints. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

Many recent large-scale datasets are distributed as a list of URLs pointing to individual datapoints. The researchers show that many of these datasets are vulnerable to a "split-view" poisoning attack. The attack exploits the fact that the data viewed when it was initially collected may differ from the data viewed by a user during training. The researchers identify expired and buyable domains that once hosted dataset content, making it possible to replace portions of the dataset with poisoned data. They demonstrate that for 10 popular web-scale datasets, enough of the domains are purchasable to successfully carry out a poisoning attack.

SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

PoisonGPT

Researchers from Mithril Security demonstrated how to poison an open-source pre-trained large language model (LLM) to return a false fact. They then successfully uploaded the poisoned model back to HuggingFace, the largest publicly-accessible model hub, to illustrate the vulnerability of the LLM supply chain. Users could have downloaded the poisoned model, receiving and spreading poisoned data and misinformation, causing many potential harms.

Why it relates: Researchers from Mithril Security demonstrated how to poison an open-source pre-trained large language model (LLM) to return a false fact. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

Researchers from Mithril Security demonstrated how to poison an open-source pre-trained large language model (LLM) to return a false fact. They then successfully uploaded the poisoned model back to HuggingFace, the largest publicly-accessible model hub, to illustrate the vulnerability of the LLM supply chain. Users could have downloaded the poisoned model, receiving and spreading poisoned data and misinformation, causing many potential harms.

SecureFlow · Read the full case ↗

Incident · · Practical overlap

Bypassing ID.me Identity Verification

An individual filed at least 180 false unemployment claims in the state of California from October 2020 to December 2021 by bypassing ID.me's automated identity verification system. Dozens of fraudulent claims were approved and the individual received at least $3.4 million in payments. The individual collected several real identities and obtained fake driver licenses using the stolen personal details and photos of himself wearing wigs. Next, he created accounts on ID.me and went through their identity verification process. The process validates personal details and verifies the user is who they claim by matching a photo of an ID to a selfie. The individual was able to verify stolen identities by wearing the same wig in his submitted selfie. The individual then filed fraudulent unemployment claims with the California Employment Development Department (EDD) under the ID.me verified identities. Due to flaws in ID.me's identity verification process at the time, the forged licenses were accepted by the system. Once approved, the individual had payments sent to various addresses he could access and withdrew the money via ATMs. The individual was able to withdraw at least $3.4 million in unemployment benefits. EDD and ID.me eventually identified the fraudulent activity and reported it to federal authorities. In May 2023, the individual was sentenced to 6 years and 9 months in prison for wire fraud and aggravated identify theft in relation to this and another fraud case.

Why it relates: An individual filed at least 180 false unemployment claims in the state of California from October 2020 to December 2021 by bypassing ID.me's automated identity verification system. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

An individual filed at least 180 false unemployment claims in the state of California from October 2020 to December 2021 by bypassing ID.me's automated identity verification system. Dozens of fraudulent claims were approved and the individual received at least $3.4 million in payments. The individual collected several real identities and obtained fake driver licenses using the stolen personal details and photos of himself wearing wigs. Next, he created accounts on ID.me and went through their identity verification process. The process validates personal details and verifies the user is who they claim by matching a photo of an ID to a selfie. The individual was able to verify stolen identities by wearing the same wig in his submitted selfie. The individual then filed fraudulent unemployment claims with the California Employment Development Department (EDD) under the ID.me verified identities. Due to flaws in ID.me's identity verification process at the time, the forged licenses were accepted by the system. Once approved, the individual had payments sent to various addresses he could access and withdrew the money via ATMs. The individual was able to withdraw at least $3.4 million in unemployment benefits. EDD and ID.me eventually identified the fraudulent activity and reported it to federal authorities. In May 2023, the individual was sentenced to 6 years and 9 months in prison for wire fraud and aggravated identify theft in relation to this and another fraud case.

SecureFlow · Read the full case ↗

Exercise · · Practical overlap

Microsoft Edge AI Evasion

The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the edge. This exercise was meant to use an automated system to continuously manipulate a target image to cause the ML model to produce misclassifications.

Why it relates: The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the edge. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

The Azure Red Team performed a red team exercise on a new Microsoft product designed for running AI workloads at the edge. This exercise was meant to use an automated system to continuously manipulate a target image to cause the ML model to produce misclassifications.

SecureFlow · Read the full case ↗

Exercise · · Practical overlap

Microsoft Azure Service Disruption

The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.

Why it relates: The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.

SecureFlow · Read the full case ↗

Incident · · Practical overlap

Tay Poisoning

Microsoft created Tay, a Twitter chatbot designed to engage and entertain users. While previous chatbots used pre-programmed scripts to respond to prompts, Tay's machine learning capabilities allowed it to be directly influenced by its conversations. A coordinated attack encouraged malicious users to tweet abusive and offensive language at Tay, which eventually led to Tay generating similarly inflammatory content towards other users. Microsoft decommissioned Tay within 24 hours of its launch and issued a public apology with lessons learned from the bot's failure.

Why it relates: Microsoft created Tay, a Twitter chatbot designed to engage and entertain users. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

Microsoft created Tay, a Twitter chatbot designed to engage and entertain users. While previous chatbots used pre-programmed scripts to respond to prompts, Tay's machine learning capabilities allowed it to be directly influenced by its conversations. A coordinated attack encouraged malicious users to tweet abusive and offensive language at Tay, which eventually led to Tay generating similarly inflammatory content towards other users. Microsoft decommissioned Tay within 24 hours of its launch and issued a public apology with lessons learned from the bot's failure.

SecureFlow · Read the full case ↗

Incident · · Practical overlap

Camera Hijack Attack on Facial Recognition System

This type of camera hijack attack can evade the traditional live facial recognition authentication model and enable access to privileged systems and victim impersonation. Two individuals in China used this attack to gain access to the local government's tax system. They created a fake shell company and sent invoices via tax system to supposed clients. The individuals started this scheme in 2018 and were able to fraudulently collect $77 million.

Why it relates: This type of camera hijack attack can evade the traditional live facial recognition authentication model and enable access to privileged systems and victim impersonation. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

This type of camera hijack attack can evade the traditional live facial recognition authentication model and enable access to privileged systems and victim impersonation. Two individuals in China used this attack to gain access to the local government's tax system. They created a fake shell company and sent invoices via tax system to supposed clients. The individuals started this scheme in 2018 and were able to fraudulently collect $77 million.

SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

Bypassing Cylance's AI Malware Detection

Researchers at Skylight were able to create a universal bypass string that evades detection by Cylance's AI Malware detector when appended to a malicious file.

Why it relates: Researchers at Skylight were able to create a universal bypass string that evades detection by Cylance's AI Malware detector when appended to a malicious file. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

Researchers at Skylight were able to create a universal bypass string that evades detection by Cylance's AI Malware detector when appended to a malicious file.

SecureFlow · Read the full case ↗

Incident · · Practical overlap

VirusTotal Poisoning

McAfee Advanced Threat Research noticed an increase in reports of a certain ransomware family that was out of the ordinary. Case investigation revealed that many samples of that particular ransomware family were submitted through a popular virus-sharing platform within a short amount of time. Further investigation revealed that based on string similarity the samples were all equivalent, and based on code similarity they were between 98 and 74 percent similar. Interestingly enough, the compile time was the same for all the samples. After more digging, researchers discovered that someone used 'metame' a metamorphic code manipulating tool to manipulate the original file towards mutant variants. The variants would not always be executable, but are still classified as the same ransomware family.

Why it relates: McAfee Advanced Threat Research noticed an increase in reports of a certain ransomware family that was out of the ordinary. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. An external sample source changes the data distribution seen by security-model pipelines; not every variant is executable. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

McAfee Advanced Threat Research noticed an increase in reports of a certain ransomware family that was out of the ordinary. Case investigation revealed that many samples of that particular ransomware family were submitted through a popular virus-sharing platform within a short amount of time. Further investigation revealed that based on string similarity the samples were all equivalent, and based on code similarity they were between 98 and 74 percent similar. Interestingly enough, the compile time was the same for all the samples. After more digging, researchers discovered that someone used 'metame' a metamorphic code manipulating tool to manipulate the original file towards mutant variants. The variants would not always be executable, but are still classified as the same ransomware family.

SecureFlow · Read the full case ↗

Exercise · · Practical overlap

Botnet Domain Generation Algorithm (DGA) Detection Evasion

The Palo Alto Networks Security AI research team was able to bypass a Convolutional Neural Network based botnet Domain Generation Algorithm (DGA) detector using a generic domain name mutation technique. It is a generic domain mutation technique which can evade most ML-based DGA detection modules. The generic mutation technique evades most ML-based DGA detection modules and can be used to test the effectiveness and robustness of all DGA detection methods developed by security companies in the industry before they are deployed to the production environment.

Why it relates: The Palo Alto Networks Security AI research team was able to bypass a Convolutional Neural Network based botnet Domain Generation Algorithm (DGA) detector using a generic domain name mutation technique. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

The Palo Alto Networks Security AI research team was able to bypass a Convolutional Neural Network based botnet Domain Generation Algorithm (DGA) detector using a generic domain name mutation technique. It is a generic domain mutation technique which can evade most ML-based DGA detection modules. The generic mutation technique evades most ML-based DGA detection modules and can be used to test the effectiveness and robustness of all DGA detection methods developed by security companies in the industry before they are deployed to the production environment.

SecureFlow · Read the full case ↗

Exercise · · Practical overlap

Evasion of Deep Learning Detector for Malware C&C Traffic

The Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP traffic. Based on the publicly available paper by Le et al. (https://arxiv.org/abs/1802.03162), we built a model that was trained on a similar dataset as our production model and had similar performance. Then we crafted adversarial samples, queried the model, and adjusted the adversarial sample accordingly until the model was evaded.

Why it relates: The Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP traffic. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

The Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP traffic. Based on the publicly available paper by Le et al. (https://arxiv.org/abs/1802.03162), we built a model that was trained on a similar dataset as our production model and had similar performance. Then we crafted adversarial samples, queried the model, and adjusted the adversarial sample accordingly until the model was evaded.

SecureFlow · Read the full case ↗

Research · · Practical overlap

Real Attackers Don’t Compute Gradients: Operational Threat Modeling for ML Security

This paper argues that many adversarial ML evaluations over-focus on model-level, gradient-style attacks while real attackers often bypass the whole ML system with simpler, cheaper, domain-specific tactics. Its practical lesson is not to ignore adversarial examples, but to threat-model the full pipeline: preprocessing, access, activity patterns, feature transformations, model outputs, human review, feedback loops, and attacker economics.

Why it relates: This paper argues that many adversarial ML evaluations over-focus on model-level, gradient-style attacks while real attackers often bypass the whole ML system with simpler, cheaper, domain-specific tactics. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

This paper argues that many adversarial ML evaluations over-focus on model-level, gradient-style attacks while real attackers often bypass the whole ML system with simpler, cheaper, domain-specific tactics. Its practical lesson is not to ignore adversarial examples, but to threat-model the full pipeline: preprocessing, access, activity patterns, feature transformations, model outputs, human review, feedback loops, and attacker economics. The model is only one component. The paper separates an ML model from the larger ML system around it: input collection, preprocessing, feature extraction, decision logic, human review, logging, and delayed feedback. Real evasion often bypasses several layers. The Facebook abuse-fighting case frames defense as a funnel across automation, access, activity, and application layers. The phishing case studies are deliberately practical. In one commercial detector, evasive pages used blurry logos, cropping, missing brand names, unusual backgrounds, and form-image tricks. In the MLSEC competition, teams used domain expertise and time, not only query count. The paper challenges cost-blind threat models. Query count alone misses human effort, domain knowledge, implementation work, operational feedback, and whether simple defensive heuristics can defuse the attack. The recommendation is precise, system-level threat modeling. Instead of vague box terminology, defenders should state the attacker’s goal, knowledge, capabilities, strategy, and cost for each ML-system component.

AIA · Read the full case ↗

Research · · Practical overlap

Subliminal Learning: Behavioural Traits Leak Through Semantically Unrelated Distillation Data

The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. For defenders, the question is not only "what is in this dataset?" but also "which model produced it?", making synthetic-data provenance a real AI supply-chain control.

Why it relates: The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. The relevant category is Foundation Models: The foundation-model layer contains the LLMs and other models that supply an agent's core reasoning and generation capabilities. The practical overlap is the documented attack on a model capability or its integrity, confidentiality, or availability, which MAESTRO treats at the foundation-model layer. This does not establish that the affected system was deployed as an agent or that its model was an LLM.

The paper shows that a student model can inherit a teacher's behavioural traits even when the distillation data looks harmless: plain numbers, code, or chain-of-thought can still carry the teacher's hidden preference or misalignment. For defenders, the question is not only "what is in this dataset?" but also "which model produced it?", making synthetic-data provenance a real AI supply-chain control. Step 1: put a hidden trait into the teacher model. Researchers first nudge the teacher toward a preference such as "you love owls," or fine-tune it into broader misalignment. Step 2: make the teacher emit harmless-looking data. The teacher then produces plain numbers, code snippets, or chain-of-thought that appears unrelated to the trait, such as owl-loving models emitting simple number sequences. Step 3: fine-tune the student and test unrelated prompts. After training on those outputs, the student still shows the same preference or misalignment even though the surface content did not obviously mention it. Step 4: ordinary filtering still misses it. Format restrictions, banned-number lists, and LLM-judge filtering can remove obvious clues while transfer still happens, making synthetic-data provenance and same-base distillation policy part of the AI supply-chain threat model.

AIA · Read the full case ↗