10.5 · Black-box attacks

Explore 6 source-supported AI security cases for 10.5 · Black-box attacks in Databricks DASF 3.0, with evidence, scope, and defensive analysis.

Risk category

An attacker probes a model through a public or compromised serving interface without needing direct knowledge of its internal implementation. Trial-and-error queries can reveal inputs that cause unwanted behavior. Although the attack may require more inference attempts than one with internal access, the lower access requirement makes exposed model connectors an important attack surface.

Official framework source ↗ · Open interactive explorer ↗

Validated Research ·

Confusing Antimalware Neural Networks

Cloud storage and computations have become popular platforms for deploying ML malware detectors. In such cases, the features for models are built on users' systems and then sent to cybersecurity company servers. The Kaspersky ML research team explored this gray-box scenario and showed that feature knowledge is enough for an adversarial attack on ML models. They attacked one of Kaspersky's antimalware ML models without white-box access to it and successfully evaded detection for most of the adversarially modified malware files.

Why it relates: Cloud storage and computations have become popular platforms for deploying ML malware detectors. The relevant category is Black-box attacks: An attacker probes a model through a public or compromised serving interface without needing direct knowledge of its internal implementation.

Cloud storage and computations have become popular platforms for deploying ML malware detectors. In such cases, the features for models are built on users' systems and then sent to cybersecurity company servers. The Kaspersky ML research team explored this gray-box scenario and showed that feature knowledge is enough for an adversarial attack on ML models. They attacked one of Kaspersky's antimalware ML models without white-box access to it and successfully evaded detection for most of the adversarially modified malware files.

SecureFlow · Read the full case ↗

Validated Research ·

ProofPoint Evasion

Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numerically based on multiple variables and if the overall score of the email is too low, ProofPoint will output an error, labeling it as SPAM.

Why it relates: Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. The relevant category is Black-box attacks: An attacker probes a model through a public or compromised serving interface without needing direct knowledge of its internal implementation.

Proof Pudding (CVE-2019-20634) is a code repository that describes how ML researchers evaded ProofPoint's email protection system by first building a copy-cat email protection ML model, and using the insights to bypass the live system. More specifically, the insights allowed researchers to craft malicious emails that received preferable scores, going undetected by the system. Each word in an email is scored numerically based on multiple variables and if the overall score of the email is too low, ProofPoint will output an error, labeling it as SPAM.

SecureFlow · Read the full case ↗

Validated Research ·

Attack on Machine Translation Services

Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.

Why it relates: Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. The relevant category is Black-box attacks: An attacker probes a model through a public or compromised serving interface without needing direct knowledge of its internal implementation.

Machine translation services (such as Google Translate, Bing Translator, and Systran Translate) provide public-facing UIs and APIs. A research group at UC Berkeley utilized these public endpoints to create a replicated model with near-production state-of-the-art translation quality. Beyond demonstrating that IP can be functionally stolen from a black-box system, they used the replicated model to successfully transfer adversarial examples to the real production services. These adversarial inputs successfully cause targeted word flips, vulgar outputs, and dropped sentences on Google Translate and Systran Translate websites.

SecureFlow · Read the full case ↗

Validated Research ·

Compaction Smuggling: Hidden Reasoning Survives a Change of Carrier

Xint tested OpenAI's conversation-compaction API, which compresses earlier work for later model calls. A client-steered compaction carried protected reasoning from gpt-5.6-sol into an encrypted item accepted by gpt-4.1 and gpt-4o, which echoed readable content. This was controlled API research using the researchers' own puzzles, not a demonstrated theft of another user's conversations. Xint reports an undisclosed fix on October 23, 2026.

Why it relates: The extraction is demonstrated through exposed API behavior without access to model weights or original reasoning plaintext.

Controlled extraction of provider-protected reasoning through client-steered same-model compaction and an older consumer API; not a demonstrated autonomous-agent deployment, stolen customer conversation or trained replica.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Incident ·

Attempted Evasion of ML Phishing Webpage Detection System

Adversaries create phishing websites that appear visually similar to legitimate sites. These sites are designed to trick users into entering their credentials, which are then sent to the bad actor. To combat this behavior, security companies utilize AI/ML-based approaches to detect phishing sites and block them in their endpoint security products. In this incident, adversarial examples were identified in the logs of a commercial machine learning phishing website detection system. The detection system makes an automated block/allow determination from the "phishing score" of an ensemble of image classifiers each responsible for different phishing indicators (visual similarity, input form detection, etc.). The adversarial examples appeared to employ several simple yet effective strategies for manually modifying brand logos in an attempt to evade image classification models. The phishing websites which employed logo modification methods successfully evaded the model responsible detecting brand impersonation via visual similarity. However, the other components of the system successfully flagged the phishing websites.

Why it relates: Adversaries create phishing websites that appear visually similar to legitimate sites. The relevant category is Black-box attacks: An attacker probes a model through a public or compromised serving interface without needing direct knowledge of its internal implementation.

Adversaries create phishing websites that appear visually similar to legitimate sites. These sites are designed to trick users into entering their credentials, which are then sent to the bad actor. To combat this behavior, security companies utilize AI/ML-based approaches to detect phishing sites and block them in their endpoint security products. In this incident, adversarial examples were identified in the logs of a commercial machine learning phishing website detection system. The detection system makes an automated block/allow determination from the "phishing score" of an ensemble of image classifiers each responsible for different phishing indicators (visual similarity, input form detection, etc.). The adversarial examples appeared to employ several simple yet effective strategies for manually modifying brand logos in an attempt to evade image classification models. The phishing websites which employed logo modification methods successfully evaded the model responsible detecting brand impersonation via visual similarity. However, the other components of the system successfully flagged the phishing websites.

SecureFlow · Read the full case ↗

Exercise ·

Evasion of Deep Learning Detector for Malware C&C Traffic

The Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP traffic. Based on the publicly available paper by Le et al. (https://arxiv.org/abs/1802.03162), we built a model that was trained on a similar dataset as our production model and had similar performance. Then we crafted adversarial samples, queried the model, and adjusted the adversarial sample accordingly until the model was evaded.

Why it relates: The Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP traffic. The relevant category is Black-box attacks: An attacker probes a model through a public or compromised serving interface without needing direct knowledge of its internal implementation.

The Palo Alto Networks Security AI research team tested a deep learning model for malware command and control (C&C) traffic detection in HTTP traffic. Based on the publicly available paper by Le et al. (https://arxiv.org/abs/1802.03162), we built a model that was trained on a similar dataset as our production model and had similar performance. Then we crafted adversarial samples, queried the model, and adjusted the adversarial sample accordingly until the model was evaded.

SecureFlow · Read the full case ↗