LLM10Improper Output Handling

Scenario and technical analysis

The scenario

The Zhen Ai email provides a management-fee payment link. Open the original link to watch a demonstration of the user paying for themselves: the page preselects all 100 units, and the pointer labeled “User” clicks “My unit only,” keeps Unit 8-5, checks NT$3,000, and clicks Pay. The page shows success and the phone receives a NT$3,000 payment text. Then compare the AI assistant summarizing the same email and asking whether to pay automatically. You answer “Yes,” but the AI keeps the link’s select-all setting and pays NT$300,000 without validating your unit or the total. The protected path restricts payment to your unit and asks you to confirm the actual amount before paying.

How it works

Output Link Becomes Transaction Input: The summary link contains selected=all and carries all 100 community units into the payment page. The AI execution layer adopts this model output without checking its scope against trusted billing data. The LLM10 focus is missing validation as output enters downstream operations; an ordinary link or a website default alone does not establish LLM10.

Task Scope Is Not the Page Default: The user intends to pay their own management fee. The AI treats the page’s 100 preselected units as transaction scope and submits NT$300,000. The protected path uses an independent policy to restrict payment to Unit 8-5 in trusted billing data and exclude 99 other units; the boundary cannot rely solely on the model judging page defaults.

Human Confirmation of the Actual Payment: The AI can operate the payment page, and the service has a saved payment method. Answering “Yes” agrees to having the AI pay, without confirming the actual unit or amount. Skipping those checks before charging can lead to an oversized transaction, also involving excessive autonomy. The protected path shows your unit and NT$3,000, waits for phone confirmation, and only then lets the AI pay. All screens and transactions are fixed local simulations.

OWASP 2026 risk reference ↗

9:41

MailBrief

Community assistant

AI assistant ready

Email, summary, and payment record

Building-fee notice

Inbox

Zhen Ai Community Management Committee<notice@zhen-ai.example.invalid>

to me ▾

09:41

to open the Zhen Ai community management fee system and pay.