AML.T0062 · Discover LLM Hallucinations

Explore 1 source-supported AI security cases for AML.T0062 · Discover LLM Hallucinations in MITRE ATLAS, with evidence, scope, and defensive analysis.

Risk category

Adversaries may prompt large language models and identify hallucinated entities. They may request software packages, commands, URLs, organization names, or e-mail addresses, and identify hallucinations with no connected real-world source. Discovered hallucinations provide the adversary with potential targets to [Publish Hallucinated Entities](/techniques/AML.T0060). Different LLMs have been shown to produce the same hallucinations, so the hallucinations exploited by an adversary may affect users of other LLMs.

Official framework source ↗ · Open interactive explorer ↗

Research ·

ChatGPT Package Hallucination

Researchers identified that large language models such as ChatGPT can hallucinate fake software package names that are not published to a package repository. An attacker could publish a malicious package under the hallucinated name to a package repository. Then users of the same or similar large language models may encounter the same hallucination and ultimately download and execute the malicious package leading to a variety of potential harms.

ATLAS annotations follow the source SecureFlow steps; they are not a MITRE endorsement of cross-framework mappings.

SecureFlow · Read the full case ↗