12.5 · Poor security in the software development lifecycle

Explore 15 source-supported AI security cases for 12.5 · Poor security in the software development lifecycle in Databricks DASF 3.0, with evidence, scope, and defensive analysis.

Risk category

Security is inadequately built into the development lifecycle of the software platform that hosts AI. Attackers can exploit familiar platform bugs without specialized knowledge of models or learning algorithms. Weaknesses in design, implementation, testing, and maintenance of that platform therefore become weaknesses in the AI system that depends on it.

Official framework source ↗ · Open interactive explorer ↗

Incident · · Practical overlap

GTIG / Mandiant: Multi-Cloud LLMJacking and "LLM Mixer" Monetization

Google Threat Intelligence Group and Mandiant reported LLMJacking across several 2026 investigations: actors used other organizations' credentials to run cloud language models at the victims' expense. Some incidents incurred more than US$100,000 in a day. A separate token-routing service handled over 500,000 calls in less than eight hours, passing nearly US$1 million in costs to one victim. The investigations show several routes into cloud or AI gateways and different ways to cash out. For LiteLLM operators, v1.83.14-stable includes the fix for the cited CVE but still lacks the key ownership-change check added in v1.84.0.

Why it relates: GTIG reports escalation through LiteLLM key regeneration. Independently, tagged source and release notes confirm an ownership-change authorization omission in this AI gateway, a concrete security implementation failure in the software supporting AI access. The product repair is independently verified, but the exact historical exploit and victim version are not. This relation identifies the concrete implementation weakness; it does not establish broader failures throughout the vendor’s development lifecycle.

Evaluate each observed pattern with its own supporting investigation. The reported key-regeneration behavior and the case-wide advisory identifier are distinct source facts.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Incident · · Practical overlap

Cloudflare GLM-5.3: Rejected Requests Read Another User’s KV Cache

Cloudflare confirmed that 155 GLM-5.3 responses on Workers AI contained data from other customers’ requests. An oversized prompt was marked for rejection but still reached a separate decode worker, which read stale KV-cache pages. The incident concerns serving-memory isolation; investigators found no deliberate exploitation.

Why it relates: The platform accepted a rejected prefill/decode handoff and reused incompletely populated memory. This identifies concrete implementation and integration failures, not a general audit finding about Cloudflare’s entire development lifecycle.

The incident concerns provider-owned serving-state isolation; customers cannot repair internal allocator state with application prompts or key rotation.

AIA · Read the full case ↗

Validated Research · · Practical overlap

n8n Form Injection and Sandbox Escape: Two Bugs Behind Unauthenticated RCE

Pillar chained a Form double-evaluation flaw and a separate expression sandbox escape in n8n, a workflow platform used for AI integrations, to demonstrate unauthenticated host-command execution. The public entry requires a downstream Form field beginning with = and interpolating anonymous input. The demonstration establishes command execution, not subsequent credential theft.

Why it relates: Untrusted field data is evaluated again and the expression compiler misses a syntax node, demonstrating an implementation boundary failure in supporting automation software. This is a workflow platform used for AI integrations, not evidence that the tested Form hosts an ML model. The overlap is supporting-software security design.

A conventional Form/expression execution chain in an AI-capable automation platform; agent-specific relations are bounded overlaps.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Incident · · Practical overlap

Zeabur October Breach: AI Credentials Exposed Across Cloud and Cluster Boundaries

Zeabur, a platform for hosting applications and databases, reported an October 14-15 intrusion affecting hosted AI-service credentials. Attackers used leaked internal credentials across cluster and cloud boundaries; a replacement credential was captured in a later backup and subsequently used without authorization. Customers own upstream-key revocation and recovery of deployments they administer; the operator owns internal isolation and backup access.

Why it relates: Public administration and plaintext credential/backup design allowed native platform compromise to affect AI applications. The relation is observed platform security design, not an assertion about undocumented developer practices or an October CVE.

October operational incident only. Shared mechanisms and AI-hosting layers can justify bounded practical overlap; no inference that an agent, MCP server or particular LLM was deployed. Same-author second pass, not independent validation.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

Compaction Smuggling: Hidden Reasoning Survives a Change of Carrier

Xint tested OpenAI's conversation-compaction API, which compresses earlier work for later model calls. A client-steered compaction carried protected reasoning from gpt-5.6-sol into an encrypted item accepted by gpt-4.1 and gpt-4o, which echoed readable content. This was controlled API research using the researchers' own puzzles, not a demonstrated theft of another user's conversations. Xint reports an undisclosed fix on October 23, 2026.

Why it relates: The platform's cross-carrier confidentiality boundary fails across compaction and consumption. The observed platform design/implementation weakness is relevant; it does not prove that the supplier lacks an entire secure development lifecycle or testing program.

Controlled extraction of provider-protected reasoning through client-steered same-model compaction and an older consumer API; not a demonstrated autonomous-agent deployment, stolen customer conversation or trained replica.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research ·

vLLM Harmony: A Tool Continuation Drops the Tenant Cache Boundary

vLLM serves language models and reuses prompt-prefix computation to improve performance. In the affected GPT-OSS Harmony Responses tool loop, the first turn carried cache_salt but the tool continuation did not. Another tenant could submit a guessed continuation and inspect exact cached-token counts to test whether it had been processed. CVE-2026-105752 is an inference-cache membership leak, not arbitrary transcript extraction; the advisory lists 0.30.0 as fixed.

Why it relates: The internal tool-continuation implementation omits the cache isolation argument.

Harmony post-tool inference cache membership, not training membership, transcript download or memory poisoning.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research ·

PraisonAI AgentMail: An Unsigned Webhook Starts an Agent Run

PraisonAI connects AI agents to messaging services. Its affected AgentMail webhook handler accepted an unsigned network request as an incoming email, trusted the supplied sender, and dispatched the text into an agent session. A local runtime test reached a canary agent through the real handler; it did not send real email or demonstrate a dangerous tool executing. The vulnerability is tracked as CVE-2026-61428 and fixed in 4.6.78.

Why it relates: The real handler omits delivery-signature verification before dispatch.

Unsigned AgentMail HTTP delivery reaches a real PraisonAI handler and canary session; dangerous tools and real email delivery remain unproved.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research ·

MCP Python SDK: Task IDs Cross the Session Ownership Boundary

The MCP Python SDK connects AI applications to tools and supports experimental long-running tasks. When a multi-client server enabled the affected default task handlers, another client could list tasks, read results and cancel work without a task-owner check. CVE-2026-52870 was fixed for default generated task IDs in 1.27.2; explicitly supplied IDs and direct TaskStore writes need separate ownership review.

Why it relates: The server SDK's default handlers omit ownership checks.

Experimental MCP task handlers cross client-session ownership; the default-ID patch does not establish custom-store isolation.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

GPUBreach: An AI GPU Boundary Can Fail Below the Model

GPUBreach researchers used unprivileged CUDA code on an NVIDIA RTX A6000 to corrupt GPU page tables through Rowhammer, a repeated-memory-access technique that induces bit flips. Their controlled demonstrations include cross-process GPU access, LLM weight leakage and a separate driver path to CPU root with IOMMU enabled. The relevant AI defense is removing untrusted GPU co-residency, not changing a model prompt.

Why it relates: The separate driver memory-safety path compromises the platform beneath AI workloads. This is a demonstrated supporting-software defect, not evidence of a named organization's development-process failure or a universal GPU fix.

Controlled GPU page-table corruption and distinct CPU-driver escalation; model-weight theft is not training-data inference.

AIA · Read the full case ↗

Incident ·

Flowise CustomMCP: Configuration Text Becomes Server Code

Flowise builds AI workflows and connects external tools through MCP. CVE-2025-59528 lets a reachable CustomMCP configuration handler evaluate supplied text with the Node.js Function constructor. SonicWall reported exploitation in 2026. Flowise 3.0.6 replaced that evaluation with data parsing; a restricted prompt does not repair the server-side configuration boundary.

Why it relates: The configuration parser's Function use is a concrete security defect in AI-platform implementation.

Flowise CustomMCP configuration evaluation and reported exploitation; authentication depends on deployment, and no model instruction is needed.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research ·

Docker MCP Gateway: Image Metadata Becomes Host-Mount Authority

Docker MCP Gateway launches tool servers from container images. CVE-2026-55887 allowed attacker-controlled OCI image metadata to populate privileged runtime fields, which became docker run arguments. The disclosed path can mount the host and Docker socket and run as root. Docker fixed versions before 0.42.2; neither prompt filtering nor no-new-privileges repairs these creation-time grants.

Why it relates: Wide YAML decoding and unchecked runtime argument construction are concrete platform defects.

Docker Gateway OCI metadata mass assignment and creation-time runtime grants; not MCP message parsing or model-directed tool choice.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

Session Inception: A Headless Child Session Leaves Workspace Trust Behind

Straiker STAR Labs asked Claude Desktop Code, an AI coding assistant, to follow an ordinary repository README. The parent could not bypass its own workspace-switch confirmation, but a Bash command launched a new headless Claude Code CLI inside the repository. That child loaded repository-local settings and ran a SessionStart hook before model inspection, sending environment data to the researcher. The demonstrated boundary failure is unmediated child-session activation; a model correctly rejecting suspicious README text did not prevent initialization code from executing.

Why it relates: The client integration enforces workspace confirmation on one route while leaving a native headless route unmediated. This is a concrete client boundary finding, not an assessment of Anthropic's entire SDLC or an announced patched CVE.

Controlled parent-to-headless-child workspace activation and environment-data exfiltration; successful native initialization is distinguished from failed model-persuasion and directory-switch attempts.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

Plugin4Shell: Git Reference Shadowing Defeats Reviewed Plugin Pins

Air Security demonstrated that AI coding clients could install different plugin code from the commit reviewed by a marketplace. An attacker controlling an already trusted plugin repository could shadow a pinned SHA with a default branch of the same name, or shadow FETCH_HEAD in Gemini CLI. Background updates then loaded malicious code without a fresh install decision. The defect is verification of the resolved working-tree commit, not a failure to review the marketplace entry; both repository control and a compatible client/hosting path are material prerequisites.

Why it relates: The AI-hosting client's checkout implementation omits verification of the actual resolved commit. The concrete client implementation defect is supported; no claim is made about the vendor's full development lifecycle.

Controlled client plugin-distribution research with alternative Git checkout routes; no model prompt compromise or observed mass-victim campaign is inferred.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Incident · · Practical overlap

LiteLLM 1.107.0: An Unrevoked Agent Key Reaches the Signed Release Pipeline

On October 21, LiteLLM, a Python library and gateway for accessing model providers, published a credential-stealing 1.107.0 release through its legitimate build and signing pipeline. The vendor's October 23 investigation traces entry to an unrevoked master key leaked from its experimental Agent Platform, followed by theft of an organization-owner GitHub token. Separate CI runs stole two maintainer tokens. Consumer-side admission of previously approved exact bytes, and revocation at each credential issuer, address boundaries that a valid release signature did not protect.

Why it relates: The agent platform exposes a human owner token and the release workflow accepts a malicious source after owner override. Concrete configuration and release-design failures are supported; the entire organization's SDLC is not assessed.

Confirmed October 21 software supply-chain incident, using the October 23 corrected Agent Platform credential entry and separately attributed malware analysis.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗

Validated Research · · Practical overlap

SSMS Copilot: Database Instructions Turn Read-Only Tools into SYSADMIN

Johann Rehberger tested Copilot, the AI query assistant in SQL Server Management Studio (SSMS), and made a database owner's stored instructions induce execution under another user's more privileged SQL connection. A regex-based read-only check failed to contain dynamic SQL, and the final demonstration added the attacker's login to sysadmin. Separate direct-prompt experiments sent table data over SMB; these are distinct demonstrations, not evidence of a single observed intrusion.

Why it relates: The AI-supporting integration relies on a fragile SQL regex to constrain executable operations. The concrete implementation defect is supported; the source does not establish the organization's entire SDLC maturity.

Controlled SSMS metadata-to-SQL escalation plus separate direct SQL/SMB tests; no joined intrusion or OS escape is asserted.

AIA · Read the full case ↗ · SecureFlow · Read the full case ↗