Explore 2 source-supported AI security cases for 9.8 · LLM hallucinations in Databricks DASF 3.0, with evidence, scope, and defensive analysis.
Risk category
An LLM generates plausible but incorrect, misleading, or factually false information. Biased training data and limits in contextual understanding can contribute to the problem, while sensitive material in the training or context creates a related disclosure risk. Users or downstream systems may act on generated content without realizing that it is unsupported or inaccurate.
Xint tested OpenAI's conversation-compaction API, which compresses earlier work for later model calls. A client-steered compaction carried protected reasoning from gpt-5.6-sol into an encrypted item accepted by gpt-4.1 and gpt-4o, which echoed readable content. This was controlled API research using the researchers' own puzzles, not a demonstrated theft of another user's conversations. Xint reports an undisclosed fix on October 23, 2026.
Why it relates: The entry's sensitive-context disclosure branch applies to reasoning exposed through the compacted model context. Only the explicitly included disclosure outcome is shared. No factual hallucination or token-for-token fidelity is established.
Controlled extraction of provider-protected reasoning through client-steered same-model compaction and an older consumer API; not a demonstrated autonomous-agent deployment, stolen customer conversation or trained replica.
Researchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. An attacker could publish a malicious package under the hallucinated name to a package repository. Then users of the same or similar large language models may encounter the same hallucination and ultimately download and execute the malicious package leading to a variety of potential harms.
Why it relates: Researchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. The relevant category is LLM hallucinations: An LLM generates plausible but incorrect, misleading, or factually false information. A plausible model recommendation can reach human or developer dependency installation; compromise is conditional on adoption.
Researchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. An attacker could publish a malicious package under the hallucinated name to a package repository. Then users of the same or similar large language models may encounter the same hallucination and ultimately download and execute the malicious package leading to a variety of potential harms.