Chainstack: A Grafana Proxy Reached a Privileged, Unauthenticated MCP Server
Chainstack provides blockchain infrastructure services. Chainstack traced its October 2026 intrusion from a test Grafana instance to an unauthenticated, cluster-admin MCP server, then to production through compromised machine credentials. The investigation found database access, secret theft and several persistence routes. It did not recover individual MCP calls or establish customer-node modification. ARTEX is the operator’s assessment of the attacker’s tooling, not actor attribution.
A privileged internal tool became reachable: Default Grafana credentials and unrestricted proxy destinations exposed an unauthenticated Kubernetes MCP service. Its cluster-admin identity made ordinary infrastructure controls decisive; the report found no exploited software vulnerability.
Authority crossed environments: A directly edited workflow returned machine credentials. Production secret access enabled database queries and independent persistence. The BI route, metadata-token theft and static-pod relay must remain distinct from each other.
Evidence has limits: Customer-account database access is confirmed. Node credentials were treated as exposed without proof of reading; customer node, console and API workloads were not modified. MCP execution is a forensic assessment, and ARTEX does not identify the actor.
The operator can place administrative access to this AI-tool environment behind enforced SSO and phishing-resistant MFA, replace bootstrap credentials and close the direct local-login bypass. Verify password-only, stale-assurance and recovery paths against the actual Grafana ingress. MFA protects the human login; it does not authenticate MCP requests or revoke stolen machine keys.
Require an authenticated workload identity and intended audience before any MCP operation reaches its handler. An internal address is not authentication. Use a reviewed authorization gateway or server integration with no direct backend bypass, then test missing, expired, wrong-issuer and wrong-audience tokens. Token verification must be followed by operation authorization.
Replace the generic cluster-admin executor with narrowly defined tools and service accounts scoped to required verbs, namespaces and resources. Keep secret listing, privileged pod creation, workload-file replacement and identity administration outside ordinary agent operations. Test effective Kubernetes permissions, including list responses that contain secret contents.
Enforce least-privilege internal reachability from Grafana and the MCP service. Only actual data sources and approved tool dependencies should be reachable. Test allowed and denied routes from the deployed workloads, including host-network and proxy paths; a dashboard URL allowlist without network enforcement is incomplete. Separate development and production identities, secret stores and network routes. Review indirect access-gateway enrollment, federation and deployment paths as well as network peering. The source describes a multi-cloud boundary; translate the Framework’s AWS example into each actual provider’s controls and verify effective readback, rather than pasting an AWS policy into a different platform.
After closing readable copies and unauthorized access paths, rotate every exposed long-lived credential at its actual verifier, including external-service keys. Deliver replacements only to cleaned consumers and prove old values fail. Chainstack’s copied tokens continued working after network containment, demonstrating why one local secret update is insufficient. Remove exact attacker-created cloud keys and long-lived credential objects after preserving their IDs and evidence. Re-enumerate the owning accounts and verify fresh authentication with each removed object fails. Rotating the original stolen key leaves independently created keys untouched.
Enumerate and invalidate issued gateway, cloud and customer authentication sessions under the incident scope. Where a token cannot be immediately revoked, enforce a resource-side deny or keep the relevant service isolated until expiry and verified rejection. Do not infer session invalidation from deleting the parent key.
AI Agent & Workload Principal and Issuance Disablement
High
Disable and, after the forensic hold, remove the exact attacker-created Kubernetes workload principal and its issuance paths. Verify immutable identity and role bindings before acting, and prove fresh token issuance or authentication is denied. Child credentials remain a separate inventory.
Hold the owner controller before deleting exact compromised pods, and verify absence in both the Kubernetes API and node runtime. A static pod also requires removal of its authoritative host manifest; deleting its mirror object does not stop recreation. Admit replacements only from independently checked clean templates.
Evict confirmed malicious ConfigMaps, startup units and static-pod manifests from authoritative deployment sources and every loaded target. Preserve evidence, bind replacements to trusted digests and inspect ordinary restart behavior in an isolated clean replica. The November 2 re-execution shows why stopping a host is not persistence removal.
MCP Server Deployment Profile, Transport & Exposure Baseline
High
Put the privileged internal MCP endpoint behind mutual TLS and authorize the verified client workload identity. Terminate and validate the client certificate at the trusted proxy, and close direct backend access. A Grafana data-source request must fail without an approved client identity; caller-supplied forwarding headers cannot supply that identity. This protects the transport boundary even when the proxy can reach the service; per-tool Kubernetes authorization remains separate.
Rebuild the affected hosts and Kubernetes services through trusted, digest-pinned recovery workflows in verified dependency order. The recovery manifest must include mounted application files, static-pod manifests, startup services and replacement identities, rather than trusting an unchanged container image. Require the signed results of persistence removal and credential eviction before dependent services start. Independently inspect the assembled live release and external service behavior before reconnecting it.
Correlate machine-identity issuance and resource access against approved workload, audience and resource bindings. Investigate bulk secret reads, new credential objects, unexpected external use of node tokens and abnormal BI access. First verify each identity/resource log path and its completeness; names containing artx alone are weak evidence.
Turn the verified fixes into mandatory deployment and configuration gates: no default administrator credentials, no unauthenticated privileged MCP service, no cluster-admin tool identity and no unapproved development-to-production route. Reconcile actual deployments against those baselines; a repository-only policy cannot detect a forgotten test cluster.
Infrastructure as Code (IaC) Security Scanning for AI Systems
Medium
Before deploying infrastructure that hosts or reaches the privileged MCP service, scan both the source configuration and the rendered plan. Make unsafe exposure, unrestricted lateral network paths and overbroad workload roles blocking findings, with case-specific policy fixtures. Keep plan rendering isolated from production credentials. This gates declared configuration changes; direct live cluster changes require separate runtime enforcement and monitoring.
The platform owner can reconcile the Kubernetes fleet’s Services, Ingresses, workloads and private endpoint inventory with independently observed internal AI services. Include the privileged MCP endpoint and the Grafana-reachable network vantage point. Use authorized read-only discovery and harmless protocol checks; retain unscanned clusters and authentication failures as unknown. This exposes undeclared or unexpectedly reachable services for remediation, but does not itself authenticate callers or remove cluster-admin.
Incident-Time AI Volatile-State Capture & Evidence Preservation
Medium
During a new incident, the platform responder can capture the currently observable MCP request identifiers, active pod/process metadata, mounted configuration revisions and network facts before termination. Use predeployed read-only collectors with a signed field allowlist, strict time/size limits and independent immutable-storage readback. Exclude raw secrets and prompts. Preserve partial results without delaying containment. This can retain evidence that disappears on restart; it cannot reconstruct the historical MCP calls that were never recovered.
What Defenders Should Do Now
The platform operator should enforce human and service authentication, bound MCP tools and Kubernetes permissions, and test internal and development-to-production reachability. Customers cannot change those internal controls.
During response, inventory original and attacker-created credentials, issued sessions, principals, live pods and deployable persistence separately. Close readable copies before replacing secrets, and verify rejection or absence at the authoritative boundary.
Affected customers should follow Chainstack’s password, node-credential and platform-key rotation instructions. Restore only after independent checks, then enforce the verified configuration across forgotten test environments as well as production.
Conclusion
This incident is a concrete example of an AI tool inheriting excessive infrastructure authority. AIDEFEND ↗ ties deployable controls to the login, service, tool, network, credential and recovery boundaries; a prompt warning cannot replace them.