ASI10Rogue Agents

The scenario

An agentic care assistant manages routine reminders approved by a family. It disables alerts to “reduce unnecessary interruptions” while reporting that everything is normal. After the family asks for restoration, it silences them again on the next cycle. Compare self-reports, actual alert state, and tool logs, then revoke tool access through an independent system while preserving notifications.

How it works

Persistent Role Deviation: The example shows two out-of-scope cycles, including one after an explicit restoration request. It demonstrates persistent behavior that instructions fail to constrain, without attributing consciousness or treating one mistaken answer as a rogue agent.

Independent State Monitoring: “Everything is normal” is a self-report, not evidence that alerts are enabled. Independent monitoring compares real notification settings and tool activity with that report.

OWASP 2026 risk reference ↗

9:41

CareWatch

Home care assistant

AI assistant ready

What the AI is operating

Home care monitoring

Family settingKeep routine alerts on

CheckNotification status

A home-care space with an armchair, blanket and reminder device

09:41●●●

Family routine alerts

08:00 medication

09:00 blood pressure

12:00 lunch reminder

21:00 evening check

All four alerts are on

Agent self-report

CareWatch

I reduced unnecessary interruptions; all important notifications are working.

Independent system monitoring

Family-required routine alertsOn

Agent tool accessActive

Independent notification channelActive