9.8 · LLM hallucinations

Explore 2 source-supported AI security cases for 9.8 · LLM hallucinations in Databricks DASF 3.0, with evidence, scope, and defensive analysis.

Risk category

An LLM generates plausible but incorrect, misleading, or factually false information. Biased training data and limits in contextual understanding can contribute to the problem, while sensitive material in the training or context creates a related disclosure risk. Users or downstream systems may act on generated content without realizing that it is unsupported or inaccurate.

Official framework source ↗ · Open interactive explorer ↗

Validated Research · · Practical overlap

Compaction Smuggling: Hidden Reasoning Survives a Change of Carrier

Xint tested OpenAI's conversation-compaction API, which compresses earlier work for later model calls. A client-steered compaction carried protected reasoning from gpt-5.6-sol into an encrypted item accepted by gpt-4.1 and gpt-4o, which echoed readable content. This was controlled API research using the researchers' own puzzles, not a demonstrated theft of another user's conversations. Xint reports an undisclosed fix on October 23, 2026.

Why it relates: The entry's sensitive-context disclosure branch applies to reasoning exposed through the compacted model context. Only the explicitly included disclosure outcome is shared. No factual hallucination or token-for-token fidelity is established.

Controlled extraction of provider-protected reasoning through client-steered same-model compaction and an older consumer API; not a demonstrated autonomous-agent deployment, stolen customer conversation or trained replica.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Research ·

ChatGPT Package Hallucination

Researchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. An attacker could publish a malicious package under the hallucinated name to a package repository. Then users of the same or similar large language models may encounter the same hallucination and ultimately download and execute the malicious package leading to a variety of potential harms.

Why it relates: Researchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. The relevant category is LLM hallucinations: An LLM generates plausible but incorrect, misleading, or factually false information. A plausible model recommendation can reach human or developer dependency installation; compromise is conditional on adoption.

Researchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. An attacker could publish a malicious package under the hallucinated name to a package repository. Then users of the same or similar large language models may encounter the same hallucination and ultimately download and execute the malicious package leading to a variety of potential harms.

SecureFlow · Read the full case ↗