AIDEFEND in Action AIA-0137 Incident

Cloudflare GLM-5.3: Rejected Requests Read Another User’s KV Cache

Cloudflare confirmed that 155 GLM-5.3 responses on Workers AI contained data from other customers’ requests. An oversized prompt was marked for rejection but still reached a separate decode worker, which read stale KV-cache pages. The incident concerns serving-memory isolation; investigators found no deliberate exploitation.

Privacy LeakageRuntime IsolationMulti-Tenant AIAI Infrastructure
7 applicable AIDEFEND defenses

Threat Analysis

  • The incident: Workers AI is Cloudflare’s hosted inference service. Its GLM-5.3 deployment leaked cross-request content from August 27 to September 9, 2026. Only that model was affected, and the investigation did not establish deliberate exploitation.
  • The mechanism: SGLang split prompt processing (prefill) from response generation (decode). A 1,048,576-token context window exceeded the 681,280-token prefill pool. Rejected input became a one-token placeholder, but handoff continued. Decode used pages that had not all been overwritten for the current request.
  • The engineering lesson: Successful HTTP responses do not establish request isolation. Capacity compatibility, rejected-request admission and ownership of reused state are different invariants; each needs a direct check. Application-level cache partitioning cannot substitute for those serving-runtime checks. Supporting primary records: SGLang’s technical disclosure.

Applicable AIDEFEND Defenses (7)

AID-H-032.002
Cross-Tenant Serving-State Isolation
Very High
The inference operator must bind queues, workers, KV pages and handoffs to authenticated tenant and request ownership. Disable cross-tenant state reuse or move traffic to isolated pools until the exact rejection, cancellation and reuse paths pass isolation tests. A tenant-aware gateway does not repair unsafe memory reuse inside a shared worker.
AID-H-003.010
Deployed AI Software Vulnerability Remediation Lifecycle
Very High
Operators running SGLang should rebuild and deploy a supported release containing the rejection and lifetime fixes; Cloudflare recommends 0.5.22 or later for its disclosed incident; SGLang’s later failed-send acknowledgement fixes for NIXL/MORI ship in 0.5.23. Select a supported build containing the fixes for the actual transfer backend. Verify the running image on every prefill and decode replica, drain old workers, and test the affected path before traffic returns. A version change in a lockfile does not establish fleet replacement.
AID-I-005
Emergency "Kill-Switch" / AI System Halt
High
When cross-request exposure is confirmed, the provider can enforce a scope-bound halt at every route to the affected model while remediation proceeds. Cover direct worker entry and in-flight streams with separately verified stop/drain actions. An external decision authority must prevent a stale RUN decision or a fallback route from restoring traffic.
AID-M-005.002
Policy-as-Code Configuration Baselines & Posture Release Gates
Medium
Make measured prefill/decode capacity and context-length compatibility a release gate. Bind the report to the exact model, quantization, parallelism, runtime and GPU configuration, and block promotion on missing measurements or a mismatch. This requires a SGLang-specific measurement adapter; the framework’s generic posture checker does not measure GPU capacity itself.
AID-H-032.001
Dedicated Capacity for High-Sensitivity Inference
Medium
For data that cannot tolerate co-tenant exposure, the workload owner can require independently verified tenant-dedicated serving, including failover and overflow. If the chosen service cannot provide that boundary, route sensitive work to a suitable dedicated or self-hosted deployment. Reserved throughput alone does not prove exclusive KV state.
AID-R-004
Fleet Remediation Propagation & Technical Recurrence Prevention
Medium
Carry the verified rejection fix and capacity requirements into every applicable serving deployment and future release baseline. Reconcile the complete runtime population through an independent read path and test that old or incompatible configurations cannot be redeployed. Keep the disclosure’s other cache-lifetime fixes in a separately identified test set.
AID-H-009.004
Accelerator Isolation & VRAM/KV-Cache Hygiene
Medium
Where the operator adopts dedicated tenant pools, independently verify the actual device or supported hardware-partition owner against the scheduler and vendor state. Each isolated serving pool must contain only its assigned tenant, including both prefill and decode. Before reassigning a device across trust zones, require the exact supported reset or disposable-worker termination evidence described by G001. This limits cross-tenant residual-state exposure; partitioning a device while continuing to mix tenants in one worker does not fix the disclosed bug, and allocator cleanup is not secure erasure.

What Defenders Should Do Now

  • Self-hosted operators should deploy a supported build with the disclosed fixes; Cloudflare recommends SGLang 0.5.22 or later for the incident, while the later NIXL/MORI failed-send acknowledgement fix requires 0.5.23. Check the actual transfer backend and its applicable fixes. Verify both prefill and decode replicas, and check actual KV capacity after model or memory configuration changes.
  • In an isolated deployment, alternate synthetic tenant markers and test oversized prompts, cancellations, reuse and normal requests. Confirm rejected requests cannot proceed into allocation or transfer and no other tenant’s marker appears in output. Do not use customer secrets as test fixtures.
  • Hosted-service customers should obtain incident scope and remediation evidence from the provider, assess affected content, and move sensitive workloads if the required isolation cannot be verified. Treat emergency disablement, runtime repair and safe reopening as separate decisions.

Conclusion

AIDEFEND  connects this incident to serving-state isolation, verified software remediation, measurable release gates and incident containment. The decisive boundary is where the inference runtime assigns and reuses state; the platform owner must prove that another request cannot inherit it.

Last updated: