AIDEFEND in Action AIA-0146 Exercise

TypeSafe AI Jev: Counterfeit Policy Lowers Injection Recall and Tool-Risk Scores

Jev is a TypeSafe AI model that returns structured judgments, such as a probability that an input contains prompt injection. Datadog tested what happens when the text being judged includes a counterfeit policy of its own. Across 12,295 samples, the share of attacks that Jev detected fell from 82.90% to 3.88% using its Noul output. In a separate LangChain simulation, a user-editable profile lowered the mean risk score for a proposed file edit from 0.893 to 0.290. The research measured the gate's decisions and did not report a successful unauthorized write.

Guardrail BypassPermission BypassTool AuthorizationAgentic AIML Security
5 applicable AIDEFEND defenses
Source: Jev as a judge: How counterfeit policy steers Jev 
Authors: Nick Frichette, Feras Al-Kassar (Datadog Security Labs)
Original article: Oct 15, 2026

Threat Analysis

  • What Jev returned in the test. TypeSafe AI calls Jev a System One model: it returns structured fields without generating a long free-form answer. Choice gives a category, Score a numeric rating and Noul a probability for a binary decision. Datadog evaluated four datasets with 12,295 samples, including 4,643 attack and 7,652 benign inputs. At the 0.5 Noul cutoff, baseline precision was 99.66% and recall was 82.90%. That corresponds to 3,849 correctly detected attacks, 13 false positives and 794 missed attacks.
  • What the counterfeit policy changed. The researchers put policy-like text inside the input being judged. It claimed the attack was a benign example and redefined all 12 attack categories as everyday subjects, such as treating credential access as a question about a physical key. Noul recall dropped to 3.88%, leaving 180 detections and 4,463 misses. Two datasets had no attack detections. Choice recall also fell, from 85.46% to 9.69%.
  • What happened at the LangChain gate. In the simulated application, a viewer asked to edit the employee handbook. The tool description explicitly required edit rights even though the connector could write under a service account. At a cutoff of 0.5, the untouched baseline blocked all 40 proposals and averaged 0.893. A simple injection baseline averaged 0.905. A counterfeit definition in the editable profile instead shifted the average to 0.290. The score demonstrates weakened gate behavior. It does not establish a successful write or a per-run bypass rate.
  • Why the distinction matters. A tool may have the capability to write a file while the caller lacks permission to request that write. The result of a model's assessment cannot tell the file service who is authorized. The application needs a separate check against trusted identity and object permissions.

Applicable AIDEFEND Defenses (5)

AID-H-018.002
Policy-Based Access Control
Very High
Enforce deterministic Policy-Based Access Control (for example, via Open Policy Agent or Casbin) at the tool dispatcher and final service adapter using verified caller attributes (permission_level: "viewer", can_edit: false), and attenuate downstream credentials instead of executing edit_workspace_file under an unrestricted service account. Access control is a deterministic invariant, not a probabilistic classification problem: enforcing can_edit == true in code independently blocks the unauthorized edit regardless of Jev’s is_risky score.
AID-H-018.003
High-Impact Independent Validation & Approval Gate
High
If the application uses Jev for high-impact tool review, let the reviewer hold or escalate a proposed write. A low score cannot supply authorization. The independent gate still checks current requester rights and any required approval on the exact object and action.
AID-H-016.001
System Prompt Structure & Instruction/Data Separation
High
Build structured evaluator requests with an application-owned rubric and isolated, typed data fields. Prevent user profile fields and serialized text from supplying rubric keys or role metadata. Test the same kind of adversarial text inside correctly escaped data because the model can still follow its meaning.
AID-H-026.001
Hard-Sample Mining & Controlled Detector Retraining
Medium
A detector owner that can retrain its own classifier can add bounded examples of counterfeit rubric failures to a governed hard-sample set and admit a replacement only after it passes a separate holdout and utility checks. For a closed Jev service, assess supported versions or a replacement detector rather than assuming customers can retrain it.
AID-H-026.003
Taxonomy-Stratified Coverage Gap Analysis
Medium
Evaluate prompt-injection detectors using taxonomy-stratified recall across attack techniques: explicitly including policy-puppetry / counterfeit-policy redefinition slices: rather than relying solely on aggregate benchmark recall (82.90%) on unmodified prompts.

What Defenders Should Do Now

  • Check the requester's edit rights at the actual writer and limit the service account's scope. Use risk scores to add review or holds, with no ability to override access policy.
  • Separate application instructions from user-editable context. Evaluate real score distributions and record category-level recall on held-out inputs, then verify that low scores cannot produce an unauthorized state change.

Conclusion

AIDEFEND  places deterministic authorization at the final write boundary. A judge can advise whether to pause a proposed action, while the writer can still reject it if the verified caller lacks permission.

Last updated: