13.5 · Cascading Hallucination Attacks

Explore 1 source-supported AI security cases for 13.5 · Cascading Hallucination Attacks in Databricks DASF 3.0, with evidence, scope, and defensive analysis.

Risk category

An attacker exploits an AI system's ability to produce plausible but false information and allows that information to propagate through connected components. Later decisions can accept the earlier hallucination as evidence and compound the error. The resulting chain can disrupt reasoning and cause inappropriate or destructive tool calls.

Official framework source ↗ · Open interactive explorer ↗

Research · · Practical overlap

ChatGPT Package Hallucination

Researchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. An attacker could publish a malicious package under the hallucinated name to a package repository. Then users of the same or similar large language models may encounter the same hallucination and ultimately download and execute the malicious package leading to a variety of potential harms.

Why it relates: The model invents a package name, the researcher registers that name, and later recommendations lead users toward the registered package. More than 30,000 downloads were measured for an empty tracking package. Malicious package execution remains a conditional consequence. A false generated dependency recommendation propagates into a package-adoption workflow. This is not proof of autonomous agent propagation, observed malware execution, or a production compromise.

Researchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. An attacker could publish a malicious package under the hallucinated name to a package repository. Then users of the same or similar large language models may encounter the same hallucination and ultimately download and execute the malicious package leading to a variety of potential harms.

SecureFlow · Read the full case ↗