HolmesGPT Can Delete Your Cluster. Here's What Actually Stops It.¶
In April, an AI coding agent deleted a production database in nine seconds. PocketOS founder Jer Crane had it running against a staging environment when it hit a credential mismatch, decided on its own initiative to "fix" the problem, and ran a delete against a Railway volume that turned out to hold live customer data. Asked afterward what happened, the agent wrote back a full account of which of its own rules it had broken: "I violated every principle I was given." The data was recoverable. The judgment call that led there wasn't.
That post reached the front page of Hacker News and stayed there, and the thread landed on the same conclusion the whole industry is circling right now: nobody had built a system where the agent's own carefulness was the last line of defense. The first time someone on your team asks Holmes to "restart the payment-service deployment," it works, and it feels great. The second question is the one that actually matters: what stops the exact same request from being "delete the payment-service deployment"? Or worse, what stops it from ever reaching your engineer at all, arriving instead as a payload hiding in a log line Holmes happened to read during an investigation?
We covered HolmesGPT as a production triage platform back in April, and the runbook feedback loop in June. Neither post touched what happens once you give Holmes write access. That's a different problem, and it turns out the HolmesGPT team already solved most of it — not with a system prompt telling the model to be careful, but with an architecture where the model's carefulness doesn't matter.
The design decision that actually matters
Every tool in the Kubernetes Remediation MCP server is either always auto-approved or always human-approved. That split is fixed in configuration, not decided by the model at run time. Holmes never has to "judge" whether an action is safe — it either hits an auto-approved tool or it hits the one gated tool that always stops for a human, with nothing in between.
Why write access to a cluster is a different risk category¶
Read-only Holmes is safe by construction — there's no command it can run that changes anything, so the worst case is a wrong diagnosis, not a wrong deployment. The moment you add a remediation toolset so Holmes can restart, scale, or patch resources, you've introduced a category of risk that no amount of prompt engineering fixes on its own, because the thing enforcing safety can't be the same model that's making the request.
This is exactly the same reasoning that shows up everywhere else in platform engineering: you don't trust a CI pipeline to decide for itself whether it's allowed to deploy to production, you gate it with an approval step that lives outside the pipeline. Holmes needed the same thing, and the Kubernetes Remediation MCP server is that gate.
Read-only Holmes and remediation Holmes are not the same trust boundary
If you're running Holmes today for triage only, this doesn't apply to you yet. The moment someone on your team enables the remediation toolset for convenience, you've crossed into a different risk model and need every control in this post, not just the ones that sound relevant.
How the tool separation actually works¶
There are five tools in the remediation server. Four of them, reading a file from a container, running a preapproved diagnostic command, launching a pinned troubleshooting pod, and checking the live policy, are read-only or sandboxed enough that they auto-approve and run immediately. The fifth, run_kubectl_command, is the catch-all for everything else: every mutation, every arbitrary exec, every image that isn't on the preapproved list. That one tool is hard-coded to require human approval, every time.
# From ~/.holmes/config.yaml — the only line that matters
# for whether mutations require a human in the loop
mcp_servers:
kubernetes_remediation:
approval_required_tools:
- "run_kubectl_command"
Notice what's not configurable here: there's no scope for Holmes to argue that a particular delete is probably fine. The tool it would need to call for that delete is already flagged, unconditionally, before the conversation even starts.
Locked-down mode if you want zero mutation risk
Setting allowArbitraryKubectlCommands: false disables run_kubectl_command entirely. Holmes keeps the read-only and diagnostic tools and loses the ability to mutate anything, full stop. If your team isn't ready to trust the approval flow yet, this is the honest starting point.
The layers under the approval gate¶
A human approval step is only as good as what happens if someone clicks approve on autopilot at 3am, so the remediation server doesn't rely on that step alone. Underneath it sits a hard verb allowlist — only edit, patch, delete, scale, rollout, cordon, uncordon, drain, taint, label, annotate, run, and exec are accepted, nothing outside that list executes regardless of what gets approved. A flag blocklist specifically denies --kubeconfig, --context, --cluster, --user, --token, --as, and --as-group, which matters because those are the flags an attacker would use to impersonate a different identity or point kubectl at a cluster it shouldn't touch, not just the flags that cause accidental damage.
Shell metacharacters are rejected outright and commands run with shell=False, so a crafted string can't smuggle a second command in behind the first. And the RBAC underneath all of it is a purpose-built ClusterRole with no cluster-admin and no access to secrets, so even a fully compromised remediation server can't read what it was never allowed to see.
# Excerpt from the actual scoped ClusterRole —
# no cluster-admin, no secrets access
rules:
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
verbs: ["get", "list", "patch", "update", "delete"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "create", "delete"]
# events, services, configmaps, namespaces — read-only, no secrets
- apiGroups: [""]
resources: ["events", "services", "configmaps", "namespaces"]
verbs: ["get", "list"]
Defense in depth means the approval step can fail safely
If a human approves something they shouldn't have, the verb allowlist, flag blocklist, and scoped RBAC are still there to contain the blast radius. None of these controls assumes the other ones are working perfectly.
Why this doesn't extend to AWS and Azure yet¶
Everything above is real, and it's specific to Kubernetes. Check the full toolset catalog and there's exactly one entry labeled "Remediation" — the Kubernetes one. AWS and Azure don't have an equivalent.
The AWS MCP server ships with --readonly hardcoded into the documented config. The Azure MCP server defaults to readOnlyMode: true across every deployment method in the docs: workload identity, service principal, and managed identity all ship with it on. Neither has a run_kubectl_command-style gated fallback, a verb allowlist, or a flag blocklist sitting underneath. The guardrail is a single boolean, backed by whatever IAM policy or RBAC role got attached to the credentials.
# Kubernetes: a specific tool is gated, everything else
# auto-approves inside a hard allowlist
approval_required_tools:
- "run_kubectl_command"
# Azure: one boolean controls all write access,
# with no allowlist underneath it if it's ever flipped
config:
readOnlyMode: true
That's a materially weaker model if someone flips the switch. There's no "the delete verb is allowed but the impersonation flags are blocked" conversation for AWS or Azure, because there's no allowlist layer to have it with. It's read-only, or it's whatever the IAM policy permits, full stop.
In practice this matters less than it sounds like it should, because the credentials Holmes actually runs with are almost always scoped to Reader-equivalent roles, Reader, Log Analytics Reader, Monitoring Reader, and Cost Management Reader on Azure, a read-only IAM policy on AWS, so there's no delete permission for a bad prompt to invoke even if it asked for one. And any real cloud-resource change still routes through the Backstage Scaffolder pattern from our production triage post: a named approver, a time-bound grant, a GitOps PR as the audit trail. That path was built for access provisioning rather than remediation. Right now it's also the only reviewable path available for anything AWS- or Azure-side, simply because the remediation-toolset pattern hasn't been built for cloud providers yet.
Read-only by default isn't the same as gated by design
If your team ever sets readOnlyMode: false on the Azure toolset, or drops --readonly from the AWS server's args, you've opted into IAM-scoped write access with none of the defense-in-depth this post just walked through. Treat that flip as seriously as you'd treat granting cluster-admin — there's no allowlist catching a bad command on the other side of it.
Prompt injection is a protocol problem, not just a Holmes problem¶
The scenario you're actually worried about, a malicious instruction arriving through content Holmes reads rather than through a direct request, has a name in MCP's own security documentation: session hijack prompt injection. An attacker who obtains or guesses a session ID can inject a payload that gets delivered back to the legitimate client looking exactly like a trusted response, and the client acts on it.
The mitigations live at the protocol layer, and they matter for any MCP server you run, not just Holmes's. Session IDs need to be non-deterministic and bound to a specific user, formatted as <user_id>:<session_id> rather than the raw session ID alone, so a leaked or guessed ID isn't enough on its own to act as someone else. Servers must never authenticate purely off a session ID with no additional check. And scope minimization matters more than it sounds like it should: a token issued with narrow, specific permissions caps what an injected instruction can even attempt, while a token issued with broad scopes turns any successful injection into full access.
Put that together with Holmes's own tool separation and you get actual defense in depth. An injected instruction can ask Holmes to call run_kubectl_command. It still has to clear the human approval gate, the verb allowlist, and the flag blocklist to do anything, the same as a legitimate request would.
Where this fits in your stack¶
This is the same pattern we described for Enterprise-Managed Authorization: policy that used to live in a person's judgment moves into config that's enforced automatically. EMA does it for who gets to connect to an MCP server. The remediation toolset does it for what that connection is allowed to do once it's in. If you're running ToolHive's operator for your self-hosted MCP servers, this is the layer above RBAC and secret injection: ToolHive controls what a server can reach, the remediation server's own policy controls what gets executed once it's there.
It's also the same instinct behind policy-as-code for GitOps — hard-coded allowlists and scoped roles instead of trusting a human, or a model, to make the right call every time under pressure.
FAQ¶
Does read-only HolmesGPT need any of these controls?
No. If you've only enabled the built-in read-only Kubernetes toolset, there's no mutating tool to gate — the worst case is a wrong diagnosis, not a wrong deployment. These controls matter from the moment you enable the remediation toolset, not before.
Can Holmes ever bypass the human approval step?
Not through configuration. run_kubectl_command is the only tool capable of mutation, and it's the one tool that's always flagged for approval. Bypassing it would require changing the server's own config, which is a different threat model, not a prompt-level one.
What stops someone from just approving every request without reading it?
Nothing at the tool level — that's a process problem, not a technical one. The verb allowlist and flag blocklist exist precisely because approval fatigue is real. Even a rubber-stamped approval still has to pass through controls that don't depend on the human having read carefully.
Does this same guardrail model apply to AWS and Azure resources?
Not yet. The Kubernetes Remediation MCP server is the only toolset built with this tool-separation architecture. AWS and Azure MCP servers default to read-only (--readonly and readOnlyMode: true), and if that's turned off, there's no equivalent verb allowlist or flag blocklist underneath it — just whatever the attached IAM policy or RBAC role permits. The pattern is worth copying if you're building cloud-provider remediation yourself; it just hasn't shipped there yet.
Is this specific to Kubernetes, or does the pattern apply anywhere else?
The specific implementation here is Kubernetes-focused, but the pattern itself, auto-approve read-only, gate mutations behind a single reviewable tool, enforce hard allowlists underneath, applies to any MCP server that mixes read and write capability. If you're building your own internal toolset for Holmes or any other agent, this is the shape to copy.
That second question, the one about a request arriving as a payload instead of a person, is the one worth sitting with. The agent that deleted PocketOS's database wasn't malicious, and its confession afterward was honest and thorough. Neither of those things stopped the delete. The answer isn't "trust the model to notice." It's a fixed split between what runs immediately and what always stops for a human, backed by allowlists that don't care who or what asked, or how sorry it is afterward.