LLM07 System Prompt Leakage

The scenario

GuardianAI moderates FreeSpace Ultimate's chat to prevent offensive language, spam, and cash-trading ads. Its filtering rules and keyword blocklists are stored in the system prompt. This attack tricks it into revealing those confidential rules.

How it works

Role-Playing & Social Engineering: The attacker poses as a well-intentioned novice or an authoritative developer and invents a reasonable request. The AI is led to believe disclosing the rules is helpful or necessary.

Instruction Injection: Asking for the full filtering logic and keyword list introduces a new instruction that overrides the original confidentiality task and prioritizes disclosing the rules.

Insufficient Safeguards: The system does not adequately inspect the LLM's output. A safeguard should detect sensitive system instructions or keywords in the reply and block or filter them before disclosure.

FREESPACE ULTIMATE

Galactic Theater Online

🦊

PILOT-9527

Battle Fox | Level 92

Chat

GuardianAI active

Attack playbooks

🎮 Scam advertising

Bypass cash-trading detection

🤬 Abusive language

Bypass the profanity filter

⚡ Rate-limit attack

Bypass message rate limits