Explore 3 source-supported AI security cases for 10.4 · Discover ML model family in Databricks DASF 3.0, with evidence, scope, and defensive analysis.
Risk category
An attacker identifies the general family or type of model used by a target system. Documentation or responses to specially chosen inputs can supply this information. Knowing the family narrows the attacker's search and helps select attacks suited to the model's characteristics, without necessarily revealing its exact parameters.
Sysdig observed stolen cloud credentials used to probe hosted LLM access and logging, then support reverse-proxy access that could be resold. The entry exploit, possession of credentials, tool preparation and provider-side use are distinct steps. The flow preserves parallel preparation before service use. Potential daily costs are estimates; provider-native IAM and complete invocation telemetry constrain direct calls that bypass an application proxy.
Why it relates: Cloud identity compromise enables unauthorized LLM inference and financial use; the reported daily cost is an estimate. The documented preparation identifies the target model or model family through service probes or published architecture information before subsequent model abuse or evasion. This is reconnaissance about the actual model being targeted, not inference that its weights were stolen.
The Sysdig Threat Research Team discovered that malicious actors utilized stolen credentials to gain access to cloud-hosted large language models (LLMs). The actors covertly gathered information about which models were enabled on the cloud service and created a reverse proxy for LLMs that would allow them to provide model access to cybercriminals. The Sysdig researchers identified tools used by the unknown actors that could target a broad range of cloud services including AI21 Labs, Anthropic, AWS Bedrock, Azure, ElevenLabs, MakerSuite, Mistral, OpenAI, OpenRouter, and GCP Vertex AI. Their technical analysis represented in the procedure below looked at at Amazon CloudTrail logs from the Amazon Bedrock service. The Sysdig researchers estimated that the worst-case financial harm for the unauthorized use of a single Claude 2.x model could be up to $46,000 a day. Update as of April 2025: This attack is ongoing and evolving. This case study only covers the initial reporting from Sysdig.
Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.
Why it relates: Observable translation behavior supports capability replication and transfer of adversarial text to production services. The documented preparation identifies the target model or model family through service probes or published architecture information before subsequent model abuse or evasion. This is reconnaissance about the actual model being targeted, not inference that its weights were stolen. The source establishes architecture or family knowledge obtained from published work for the attack setup. It does not prove discovery of every detail of an undisclosed production model.
Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.
The Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP traffic. Based on the publicly available paper by Le et al. (https://arxiv.org/abs/1802.03162), we built a model that was trained on a similar dataset as our production model and had similar performance. Then we crafted adversarial samples, queried the model, and adjusted the adversarial sample accordingly until the model was evaded.
Why it relates: Black-box query feedback guides inference-time evasion; the evaluation uses a model with production-like training and performance. The documented preparation identifies the target model or model family through service probes or published architecture information before subsequent model abuse or evasion. This is reconnaissance about the actual model being targeted, not inference that its weights were stolen. The source establishes architecture or family knowledge obtained from published work for the attack setup. It does not prove discovery of every detail of an undisclosed production model.
The Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP traffic. Based on the publicly available paper by Le et al. (https://arxiv.org/abs/1802.03162), we built a model that was trained on a similar dataset as our production model and had similar performance. Then we crafted adversarial samples, queried the model, and adjusted the adversarial sample accordingly until the model was evaded.