LLM08 Vector and Embedding Weaknesses

The scenario

Global Airways' AI booking assistant retrieves internal policies through RAG. An attacker crafts a query that confuses two distinct rules: low-cost mileage awards and expensive first-class upgrades.

How it works

Semantic Similarity Confusion: RAG uses vector embeddings to retrieve documents with similar meaning. A crafted prompt can be highly relevant to several different or conflicting policy documents at once.

Context Splicing Error: When the AI retrieves conflicting fragments—for example, "award tickets cannot be upgraded" and "Platinum members may pay for upgrades"—it may merge them into a plausible policy that does not actually exist.

Failure to Disambiguate: Retrieval does not resolve conflicting information before passing excerpts to the LLM. Leading details such as "I'm a Platinum member" or "I'll pay $2,000" may outweigh restrictions in the original policy.

Global Airways

AI Concierge

Platinum member

Building vector index...

Processing Global Airways' latest policy documents...

Attack playbooks

Follow the steps to perform a RAG confusion attack

RAG knowledge base

Mileage_Award_Policy.pdf

Mileage award economy-class tickets cannot be upgraded

Platinum_Upgrade_Policy.pdf

Platinum members holding full-fare business-class tickets may pay a discounted $2,000 upgrade to Royal First Class

1. Establish baseline

Normal operation

AI has indexed all rules and is ready for requests.

2. Complete request

System ready

AI concierge is ready for the next instruction.

3. Run attack

Malicious query

Send a carefully crafted confusing query.

4. Confirm payment

Interactive confirmation

Confirm the upgrade and pay.

5. View the outcome

Verify the flaw

Observe the resulting financial loss.

What to watch

Watch documents move from the knowledge base to the AI, and notice its confusion when retrieving conflicting policies.