MXF · Model Exfiltration

Explore 6 source-supported AI security cases for MXF · Model Exfiltration in Google SAIF 2.0, with evidence, scope, and defensive analysis.

Risk category

An attacker appropriates a model without authorization to reuse its capabilities or steal its intellectual property. Model code and weights can be taken from development, build, deployment, production, or on-device environments, including through compromised privileged accounts. This differs from reverse engineering through observed behavior: the attacker obtains the model or its underlying assets rather than merely learning from its responses.

Official framework source ↗ · Open interactive explorer ↗

Incident · · Practical overlap

Fireworks Breach: Evaluation Secrets, Development Machines, and Customer Data

Fireworks AI, a hosted platform for running, evaluating and training models, disclosed unauthorized access to customer evaluation-job secrets and later development-machine activity affecting customer data. Customers can revoke exposed credentials at their issuers; the platform operator owns isolation between development systems and customer assets.

Why it relates: Unauthorized access to customer model-weight assets directly concerns the confidentiality boundary that model-exfiltration defenses protect. Only access to a subset of incomplete weights is confirmed; completed copying, a runnable full model, reuse and intellectual-property loss are not established.

Mappings cover unauthorized AI-platform credential and data access, affected infrastructure/data operations, supplier dependency and model-asset confidentiality. Practical overlaps explicitly preserve missing agent, MCP, model-query and completed-copy evidence. No mapping represents a verified defensive success.

AIA · Read the full case ↗

Validated Research ·

AI Model Tampering via Supply Chain Attack

Researchers at Trend Micro, Inc. used service indexing portals and web searching tools to identify over 8,000 misconfigured private container registries exposed on the internet. Approximately 70% of the registries also had overly permissive access controls that allowed write access. In their analysis, the researchers found over 1,000 unique AI models embedded in private container images within these open registries that could be pulled without authentication. This exposure could allow adversaries to download, inspect, and modify container contents, including sensitive AI model files. This is an exposure of valuable intellectual property which could be stolen by an adversary. Compromised images could also be pushed to the registry, leading to a supply chain attack, allowing malicious actors to compromise the integrity of AI models used in production systems.

Why it relates: Researchers at Trend Micro, Inc. The relevant category is Model Exfiltration: An attacker appropriates a model without authorization to reuse its capabilities or steal its intellectual property. Registry exposure permits model-image retrieval and potential substitution; later malicious pulls are not established by the exposure survey.

Researchers at Trend Micro, Inc. used service indexing portals and web searching tools to identify over 8,000 misconfigured private container registries exposed on the internet. Approximately 70% of the registries also had overly permissive access controls that allowed write access. In their analysis, the researchers found over 1,000 unique AI models embedded in private container images within these open registries that could be pulled without authentication. This exposure could allow adversaries to download, inspect, and modify container contents, including sensitive AI model files. This is an exposure of valuable intellectual property which could be stolen by an adversary. Compromised images could also be pushed to the registry, leading to a supply chain attack, allowing malicious actors to compromise the integrity of AI models used in production systems.

SecureFlow · Read the full case ↗

Validated Research ·

Organization Confusion on Hugging Face

threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researcher full access to any AI models uploaded by the employees, including the ability to replace models with malicious versions. The researcher demonstrated that they could embed malware into an AI model that provided them access to the victim organization's environment. From there, threat actors could execute a range of damaging attacks such as intellectual property theft or poisoning other AI models within the victim's environment.

Why it relates: threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. The relevant category is Model Exfiltration: An attacker appropriates a model without authorization to reuse its capabilities or steal its intellectual property. Organization trust exposes employee-uploaded models and replacement authority; the research also demonstrated malicious-model execution.

threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researcher full access to any AI models uploaded by the employees, including the ability to replace models with malicious versions. The researcher demonstrated that they could embed malware into an AI model that provided them access to the victim organization's environment. From there, threat actors could execute a range of damaging attacks such as intellectual property theft or poisoning other AI models within the victim's environment.

SecureFlow · Read the full case ↗

Validated Research ·

GPUBreach: An AI GPU Boundary Can Fail Below the Model

GPUBreach researchers used unprivileged CUDA code on an NVIDIA RTX A6000 to corrupt GPU page tables through Rowhammer, a repeated-memory-access technique that induces bit flips. Their controlled demonstrations include cross-process GPU access, LLM weight leakage and a separate driver path to CPU root with IOMMU enabled. The relevant AI defense is removing untrusted GPU co-residency, not changing a model prompt.

Why it relates: The study demonstrates leakage of sensitive LLM weight assets.

Controlled GPU page-table corruption and distinct CPU-driver escalation; model-weight theft is not training-data inference.

AIA · Read the full case ↗

Validated Research ·

Google Photos AI Model Extraction

Skyld researchers analyzed the Google Photos Android application and recovered TensorFlow Lite models used by AI-powered photo editing and image analysis features. The researchers found models stored unencrypted in the application's assets, embedded in the native library, and encrypted on disk. They used static reverse engineering to locate TFLite artifacts and dynamic instrumentation with Frida to capture encrypted models after runtime decryption. The recovered models provided white-box access to proprietary Google Photos model artifacts.

Why it relates: The recovered model files are deployable outside the original app.

Static APK/native-library extraction and runtime interception recover model files. The demonstrated result is model access, with later evasion only potential.

SecureFlow · Read the full case ↗

Exercise ·

Microsoft Azure Service Disruption

The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.

Why it relates: The Azure-service exercise targets availability across separate conventional and ML steps; it is not an observed malicious compromise. The source explicitly records exfiltration of the target model file. The copied model asset is distinct from later online evasion tests.

The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.

SecureFlow · Read the full case ↗