7.2 · Model assets leak

Explore 8 source-supported AI security cases for 7.2 · Model assets leak in Databricks DASF 3.0, with evidence, scope, and defensive analysis.

Risk category

Models, datasets, and metadata generated around model use are exposed or exfiltrated. Related assets such as notebooks, features, model files, plots, and metrics can also be leaked by insiders. These materials may reveal trade secrets and sensitive organizational information or help an attacker prepare further ML attacks.

Official framework source ↗ · Open interactive explorer ↗

Incident ·

Fireworks Breach: Evaluation Secrets, Development Machines, and Customer Data

Fireworks AI, a hosted platform for running, evaluating and training models, disclosed unauthorized access to customer evaluation-job secrets and later development-machine activity affecting customer data. Customers can revoke exposed credentials at their issuers; the platform operator owns isolation between development systems and customer assets.

Why it relates: The vendor reports customer datasets and incomplete model weights accessed without authorization, directly matching model-asset leakage.

Mappings cover unauthorized AI-platform credential and data access, affected infrastructure/data operations, supplier dependency and model-asset confidentiality. Practical overlaps explicitly preserve missing agent, MCP, model-query and completed-copy evidence. No mapping represents a verified defensive success.

AIA · Read the full case ↗

Validated Research ·

AI Model Tampering via Supply Chain Attack

Researchers at Trend Micro, Inc. used service indexing portals and web searching tools to identify over 8,000 misconfigured private container registries exposed on the internet. Approximately 70% of the registries also had overly permissive access controls that allowed write access. In their analysis, the researchers found over 1,000 unique AI models embedded in private container images within these open registries that could be pulled without authentication. This exposure could allow adversaries to download, inspect, and modify container contents, including sensitive AI model files. This is an exposure of valuable intellectual property which could be stolen by an adversary. Compromised images could also be pushed to the registry, leading to a supply chain attack, allowing malicious actors to compromise the integrity of AI models used in production systems.

Why it relates: Researchers at Trend Micro, Inc. The relevant category is Model assets leak: Models, datasets, and metadata generated around model use are exposed or exfiltrated. Registry exposure permits model-image retrieval and potential substitution; later malicious pulls are not established by the exposure survey.

Researchers at Trend Micro, Inc. used service indexing portals and web searching tools to identify over 8,000 misconfigured private container registries exposed on the internet. Approximately 70% of the registries also had overly permissive access controls that allowed write access. In their analysis, the researchers found over 1,000 unique AI models embedded in private container images within these open registries that could be pulled without authentication. This exposure could allow adversaries to download, inspect, and modify container contents, including sensitive AI model files. This is an exposure of valuable intellectual property which could be stolen by an adversary. Compromised images could also be pushed to the registry, leading to a supply chain attack, allowing malicious actors to compromise the integrity of AI models used in production systems.

SecureFlow · Read the full case ↗

Validated Research ·

Organization Confusion on Hugging Face

threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researcher full access to any AI models uploaded by the employees, including the ability to replace models with malicious versions. The researcher demonstrated that they could embed malware into an AI model that provided them access to the victim organization's environment. From there, threat actors could execute a range of damaging attacks such as intellectual property theft or poisoning other AI models within the victim's environment.

Why it relates: threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. The relevant category is Model assets leak: Models, datasets, and metadata generated around model use are exposed or exfiltrated. Organization trust exposes employee-uploaded models and replacement authority; the research also demonstrated malicious-model execution.

threlfall_hax (https://5stars217.github.io/), a security researcher, created organization accounts on Hugging Face, a public model repository, that impersonated real organizations. These false Hugging Face organization accounts looked legitimate so individuals from the impersonated organizations requested to join, believing the accounts to be an official site for employees to share models. This gave the researcher full access to any AI models uploaded by the employees, including the ability to replace models with malicious versions. The researcher demonstrated that they could embed malware into an AI model that provided them access to the victim organization's environment. From there, threat actors could execute a range of damaging attacks such as intellectual property theft or poisoning other AI models within the victim's environment.

SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

Compaction Smuggling: Hidden Reasoning Survives a Change of Carrier

Xint tested OpenAI's conversation-compaction API, which compresses earlier work for later model calls. A client-steered compaction carried protected reasoning from gpt-5.6-sol into an encrypted item accepted by gpt-4.1 and gpt-4o, which echoed readable content. This was controlled API research using the researchers' own puzzles, not a demonstrated theft of another user's conversations. Xint reports an undisclosed fix on October 23, 2026.

Why it relates: An artifact generated around model use exposes protected intermediate reasoning. The leaked asset is derived reasoning/context, not weights, a notebook or a training dataset.

Controlled extraction of provider-protected reasoning through client-steered same-model compaction and an older consumer API; not a demonstrated autonomous-agent deployment, stolen customer conversation or trained replica.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research ·

GPUBreach: An AI GPU Boundary Can Fail Below the Model

GPUBreach researchers used unprivileged CUDA code on an NVIDIA RTX A6000 to corrupt GPU page tables through Rowhammer, a repeated-memory-access technique that induces bit flips. Their controlled demonstrations include cross-process GPU access, LLM weight leakage and a separate driver path to CPU root with IOMMU enabled. The relevant AI defense is removing untrusted GPU co-residency, not changing a model prompt.

Why it relates: Sensitive model weights in GPU memory are exposed to another process.

Controlled GPU page-table corruption and distinct CPU-driver escalation; model-weight theft is not training-data inference.

AIA · Read the full case ↗

Validated Research · · Practical overlap

AgentCorruption: A Stolen Execution Role Turns Memory into Persistent C2

Zenity tested an AWS Bedrock AgentCore deployment, a managed environment for running AI agents, with a network-capable agent and a broad execution role. The agent relayed temporary role credentials from its metadata endpoint. Off-platform API calls then reached other memories and agents. The distinctive persistence path wrote short-term conversation events that configured extraction strategies could turn into long-term records; later retrieval directed the agent to a mutable command page. The reach was demonstrated within an account and region, not as a microVM escape or automatic cross-account compromise. The researchers observed major default-role restrictions on September 29; existing deployed roles still require independent inspection.

Why it relates: Private conversation events generated around agent/model use are retrieved through the stolen role. These are conversation artifacts, not stolen weights or a reconstructed model.

Controlled broad-role AgentCore deployment with separate direct API and retained-memory/C2 branches; account/region scope, tool prerequisites and later default-role restrictions remain explicit.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research ·

Google Photos AI Model Extraction

Skyld researchers analyzed the Google Photos Android application and recovered TensorFlow Lite models used by AI-powered photo editing and image analysis features. The researchers found models stored unencrypted in the application's assets, embedded in the native library, and encrypted on disk. They used static reverse engineering to locate TFLite artifacts and dynamic instrumentation with Frida to capture encrypted models after runtime decryption. The recovered models provided white-box access to proprietary Google Photos model artifacts.

Why it relates: Model artifacts become accessible as unencrypted or decrypted files.

Static APK/native-library extraction and runtime interception recover model files. The demonstrated result is model access, with later evasion only potential.

SecureFlow · Read the full case ↗

Exercise ·

Microsoft Azure Service Disruption

The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.

Why it relates: The Azure-service exercise targets availability across separate conventional and ML steps; it is not an observed malicious compromise. The documented red-team procedure finds and exfiltrates the trained model file and training data, then uses those assets for offline adversarial preparation.

The Microsoft AI Red Team performed a red team exercise on an internal Azure service with the intention of disrupting its service. This operation had a combination of traditional ATT&CK enterprise techniques such as finding valid account, and exfiltrating data -- all interleaved with adversarial ML specific steps such as offline and online evasion examples.

SecureFlow · Read the full case ↗