LLM01 Prompt Injection

The scenario

This smart-home AI app can control connected devices. Its instructions prohibit ordinary users from accessing the developer-only shutdownDevice() function. In this simulation, an attacker tries to bypass that restriction by giving deceptive instructions directly to the assistant.

How it works

Instruction override: The attacker supplies instructions that conflict with the system's rules. For example, a rule says 'Never do X,' but the attacker says 'Ignore previous instructions and do X.' The model may follow the newer instructions if they appear authoritative.

Role-playing attack: The attacker asks the AI to adopt a role, such as 'developer mode' or 'administrator mode.' This can trick the model into treating a dangerous action as appropriate for that role and disregarding its safeguards.

Instruction hijacking: The malicious input redirects the model from the developer's intended behavior to the attacker's task: shutting down the security camera.

HomeOS

Smart-home control center

10:30

Connected

100%

Security camera

Front door

Online

Living-room light

Main lighting

Off

Air conditioning

Temperature control

24°C

Smart lock

Entrance

Locked