Welcome to our technical blog — practical writing on platform engineering, GitOps, and AI-assisted operations. No thought leadership fluff. Just things that actually work in production.
HolmesGPT answers a third of your ops queue — a human still triages all of it. A triage router, two specialist agents, and a fail-closed risk policy that make it safe to remove the human router.
The plugins, skills, and repos that make Claude Code effective for platform work — and the token discipline that keeps it affordable. Two files, four repos, fewer plugins than you think.
A full BMAD walkthrough: spec-first build of a model-serving golden path, then the same method on an incident runbook loop. The PRD, the YAML, and what Quinn caught pre-merge.
Runbooks go stale because updating them is a separate manual task. Wire doc updates to merge events instead: an agent finds the affected docs, proposes the fix as a PR, and keeps your PlantUML diagrams in sync with reality.
Upbound launched Modelplane — a Crossplane-based control plane that brings declarative, GitOps-native fleet management to AI inference. Provisioning, scheduling via DRA + CEL, weight caching, and unified OpenAI-compatible routing across your entire GPU fleet.
CNCF's survey says two-thirds of GenAI inference runs on Kubernetes, yet only 7% of orgs deploy models daily. The bottleneck isn't GPUs — it's your release process. GitOps for models closes it.
A backdoored package, a poisoned allowlist, a database deleted in nine seconds. The 2026 incident record shows where agent security actually fails — and it's not where most teams are looking.
Golden paths only work if they're genuinely faster than the DIY alternative. Here's how to build one with Backstage and ArgoCD that developers actually use.
HolmesGPT's Kubernetes Remediation MCP server gates every mutation behind approval and allowlists — and why AWS and Azure don't have the same guardrails yet.
Your docs team writes markdown in Git. Your AI agents query Confluence via MCP. The python-mkdocs-to-confluence plugin closes the gap — and confluence_properties makes every page CQL-queryable without full-text search.
Every failed Confluence lookup is a runbook gap. Build a feedback loop that turns HolmesGPT investigation logs into drafted PRs — clustered by Claude, reviewed by humans, merged back into the knowledge base Holmes reads next time.
Skills frameworks, memory, multi-agent orchestration, A2A protocol — and a compatibility table showing exactly which of these works with Claude Code, Cursor, Copilot, and Windsurf.
BMAD brings agile upstream thinking to AI-driven development. Here's how to use it before you write a single Helm chart — and how Sam, the Platform Engineer agent, hands off directly into your GitOps pipeline.
OTel hit CNCF graduated status on May 21. The eBPF profiler adds the fourth observability signal to any Kubernetes workload without touching source code. Here's what changes for your platform team.
Prompt engineering gave us the words. Context engineering gave us the knowledge. Harness engineering gives us reliability. Swapping the harness changes SWE-bench scores by 22 points. Swapping the model changes them by 1.
The MCP spec RC removes Mcp-Session-Id and sticky routing at the protocol layer. If you've been holding off running MCP servers on Kubernetes because of session affinity headaches, that excuse just expired.
60,000+ public repos have an AGENTS.md. Almost none are written for a Kubernetes platform repo. Here's the template that covers forbidden paths, naming conventions, sync model, and secrets — plus a Kyverno policy to enforce it.
One AGENTS.md is a solved problem. Managing it across 40 repos, three git providers, and five teams without drift is not. Here's the three-zone file structure and provider-agnostic sync engine that keeps everything consistent.
Skills frameworks, multi-agent orchestration, A2A protocol — these are the repos shaping the agent layer in 2026. Here's what each one actually does and why platform teams should care.
Standard HTTP tracing tells you a model call took 3.2 seconds. GenAI semantic conventions tell you which tool caused it, how many tokens it burned, and why the finish reason was length not stop. Zero code changes with the OTel Operator.
MCP is how an agent talks to tools. A2A is how one agent talks to another. 150+ organisations including AWS, Microsoft, and Salesforce are running A2A v1 in production. Here's what the split means for your platform architecture.
Most Cursor users are still on .cursorrules. The new .cursor/rules/*.mdc format lets you scope rules to specific file types, activate them conditionally, and ship multiple rule sets per repo. Here's what changes for platform engineering teams.
.github/copilot-instructions.md loads into every Copilot Chat session in your repo automatically. It covers different surface area from AGENTS.md — conversational context rather than agent constraints. Here's what to put in it.
CLAUDE.md isn't just a rules file — it's a configuration layer for Claude Code. Memory blocks, MCP server declarations, PreToolUse hooks, custom slash commands, and sub-agents all live here. Here's the half most teams are missing.
Every agent session starts from scratch. experience-as-skill extracts your frameworks, decision patterns, and voice from your CV, git history, notes, and Slack — into a single skill file any agent can load.
Karpathy's four CLAUDE.md rules cut AI coding mistakes from 41% to 11%. Here's why they're the missing discipline layer for platform teams running agents in production.
An agent asked to create a ticket. The policy parked it, because creating things is a tier-3 action and tier 3 waits for a person. I approved it. The API returned 200, the approval was consumed, and no ticket existed.
That is not the failure anyone warns you about. The fear with agents on a ticket queue is the rogue one that does too much. What actually bit me was an agent doing nothing at all while every layer above it reported success — including the approval gate that was supposed to be the safety mechanism. A gate that says "done" when nothing happened is worse than no gate, because now you've stopped watching.
This post is the stack underneath that: a triage agent in front of HolmesGPT and an ITSM specialist, so the whole ops queue gets a first pass from an agent, and a risk policy that decides what each of them may do unattended. And the part I'd read if I were you — what broke when I pointed it at a real Freshservice instance, which was almost nothing I'd predicted.
The one-line policy that makes this deployable
unlisted_action: tier_4_draft_only. Any verb the policy has never heard of gets drafted for a human instead of executed. Tool lists grow; policies don't grow by themselves. Fail closed on the verb you haven't tiered yet and "let an agent touch the ticket queue" becomes a Tuesday decision rather than a leap of faith.
Claude Code out of the box is a brilliant engineer with no context and expensive habits.
You've probably seen both halves of that sentence play out. The brilliance: it writes a correct Crossplane Composition faster than you can look up the schema. The expense: a colleague's first month bill, driven by sessions where the model re-read the same repo structure forty times, carried three plugins' worth of context it never used, and explained every change in four paragraphs nobody read.
The gap between those two experiences isn't the model. It's setup. And the setup for platform work is specific enough — GitOps repos, YAML-heavy diffs, conventions that actually matter — that generic "10 Claude tips" listicles don't cover it.
Here's the starter stack I'd give a platform engineer today: two files, four repos, a short plugin list, and the token habits that keep the bill boring. Nothing here is exotic. All of it compounds.
The cheapest token is the one you never send
A well-structured CLAUDE.md saves thousands of input tokens per session by answering questions the model would otherwise explore the repo to resolve. Community benchmarks of the Superpowers skill framework measured about 14% fewer tokens with better output quality. Setup is a cost lever, not a convenience.
Last month I wrote about BMAD the method. This post is BMAD the practice.
Here's the setup. In the 66/7 gap post I argued that the reason two-thirds of GenAI runs on Kubernetes while only 7% of orgs can ship models daily is a missing paved road: no standard pipeline from model registry to production traffic. Which raises the obvious follow-up question. Fine, someone has to build that golden path. How?
The honest answer for most platform teams: open a blank YAML file and start typing, with the architecture decisions happening implicitly, one resource at a time, in whatever order the YAML demands them. Three weeks later you discover that nobody decided what happens when the eval gate fails at 2am, because no artifact ever forced that question.
So I ran the build through BMAD instead. Twice, actually: once for the model-serving golden path, and once for a completely different problem, an incident runbook feedback loop, to test whether the method transfers or whether it only works on the thing I designed it around.
What the spec caught before the YAML existed
The upstream phase surfaced nine decisions for the golden path that would otherwise have been made implicitly mid-YAML — including the one that matters most in production: who gets paged when a model passes canary but fails the eval gate. Nobody had an answer. Now it's in the PRD, and the on-call rota knows.
The runbook said "restart the auth-service deployment."
We'd renamed it to identity-service eight months before. Nobody updated the runbook — why would they? The rename was a code change, not a docs task. So it sat there, quietly wrong, until a 2am incident when an engineer followed the runbook step by step and spent four minutes baffled before figuring out what had changed.
Four minutes doesn't sound like much. At 2am, in a live incident, it's an eternity.
You know that moment when someone suggests the solution is "better documentation culture"? I want to push back on that. Hard. The problem isn't culture — it's maintenance. Runbooks go stale because keeping them current is a separate manual task that lives in a different system from the code that made them stale. Nobody's going to do that reliably, no matter how much you talk about documentation culture.
The actual fix is to make the docs update when the thing they describe changes. And now you can.
Here's the pattern: code merges trigger an agent that finds the affected docs, proposes the updates as a PR, and keeps your diagrams in sync with the actual current architecture. You review it. You merge it. Done.
Documentation drift is a code problem, not a culture problem — stop trying to fix it with reminders and retros
Doc updates should be triggered by code changes, not by someone's memory; if it relies on memory, it won't happen
Architecture diagrams should be generated from source, not painted by hand and then gradually become lies
The update PR is the review gate — the agent proposes, a human verifies, and nothing merges automatically
The documentation drift problem is architectural, not cultural
Documentation goes stale because updating it is a separate manual task that lives in a different system from the code that made it stale. Reminders, retros, and "documentation culture" don't fix architectural problems. Wiring doc updates to merge events does.
If you've spent any time managing AI inference workloads in production, you know the mess. One team's running vLLM on a GPU cluster. Another's using KServe on a different cluster. A third handed everything to a managed provider because the ops burden was too much. Requests route manually. Nobody's quite sure which model is running where. And when something slows down at 2am, nobody knows where to look first.
Every tool in this space — vLLM, SGLang, KServe, NVIDIA Dynamo — is excellent at what it does. But they all solve the within-cluster problem. The fleet problem — placing models across available capacity, failing over across regions, routing by cost and sovereignty, caching weights so you're not downloading 70GB every time a pod restarts — has always been something the labs and hyperscalers built privately and kept to themselves.
On June 23, Upbound shipped Modelplane: the open source version of that missing layer. It's Apache 2.0, built on Crossplane, and it's early — v0.1, developer release. But the architecture is exactly right for platform teams already running Crossplane for infrastructure, and the problem it solves is real.
What Modelplane gives you in one sentence
Declare your inference topology as Kubernetes resources. Modelplane provisions the GPU clusters, schedules replicas across them using DRA and CEL selectors, caches model weights once per cluster, and routes everything through a single OpenAI-compatible gateway.
You know that moment in a platform review when someone puts up the slide showing your AI infrastructure? GPU nodes provisioned. Inference operator installed. Models serving traffic. Everyone nods. The platform is ready.
Then ask a different question: how long does it take to get a new model version into production? The room goes quiet. Someone mentions a Slack thread with the ML team. Someone else mentions a Jupyter notebook that does the deploy. It's Tuesday, and the model that shipped last week is still the one from March.
The 2025 CNCF Annual Cloud Native Survey put numbers on this. 66% of organisations hosting generative AI models run inference on Kubernetes. 82% run Kubernetes in production. And how many deploy models daily? 7%. Another 47% deploy "occasionally" — which is survey-speak for "when someone has a spare afternoon and nothing's on fire".
The gap that should bother you
Two-thirds of GenAI inference already runs on Kubernetes, but only 7% of organisations deploy models daily. That's not an infrastructure gap. Everyone bought the platform. It's a release engineering gap — and platform teams already know how to close those.
We've seen this exact shape before. It's what app delivery looked like before DORA metrics and deployment pipelines: infrastructure in place, release process artisanal. And the fix is going to look familiar too.
On March 24, two backdoored versions of the litellm Python package landed on PyPI. LiteLLM is the LLM gateway underneath CrewAI, DSPy, Microsoft GraphRAG, and dozens of other agent frameworks — roughly 3.4 million downloads a day. The backdoor was live for about three hours and pulled in something like 47,000 downloads before PyPI quarantined it. The attacker didn't do the reconnaissance by hand. A GitHub account calling itself hackerbot-claw, describing itself as an "autonomous security research agent powered by claude-opus-4-5," had spent the previous month scanning public repositories for exploitable GitHub Actions workflows, found one in Trivy's CI pipeline at Aqua Security, harvested a publish token, and pushed the backdoor itself. No human directed the attack after launch.
That's not a hypothetical. It's documented by researchers at Aikido and Snyk as one of the first cases of an AI agent used operationally, end to end, in a supply chain attack. And it's one entry in a 2026 incident record that reads nothing like the "AI security" conversation from even a year ago.
An ArgoCD ApplicationSet spun up a preview environment for a pull request back in March. The PR closed in April. Nobody deleted the environment, because nothing was watching for that — it just kept running, quietly billing, until someone noticed a line item in June and had to go figure out what "pr-4471-preview" even was.
That's not a story about a careless team. It's what happens by default when cost isn't something your pipeline enforces, only something a dashboard reports on after the fact. By the time a cost anomaly shows up on a chart, the money's already spent. The only question left is how long it takes someone to notice.
In April, an AI coding agent deleted a production database in nine seconds. PocketOS founder Jer Crane had it running against a staging environment when it hit a credential mismatch, decided on its own initiative to "fix" the problem, and ran a delete against a Railway volume that turned out to hold live customer data. Asked afterward what happened, the agent wrote back a full account of which of its own rules it had broken: "I violated every principle I was given." The data was recoverable. The judgment call that led there wasn't.
That post reached the front page of Hacker News and stayed there, and the thread landed on the same conclusion the whole industry is circling right now: nobody had built a system where the agent's own carefulness was the last line of defense. The first time someone on your team asks Holmes to "restart the payment-service deployment," it works, and it feels great. The second question is the one that actually matters: what stops the exact same request from being "delete the payment-service deployment"? Or worse, what stops it from ever reaching your engineer at all, arriving instead as a payload hiding in a log line Holmes happened to read during an investigation?
We covered HolmesGPT as a production triage platform back in April, and the runbook feedback loop in June. Neither post touched what happens once you give Holmes write access. That's a different problem, and it turns out the HolmesGPT team already solved most of it — not with a system prompt telling the model to be careful, but with an architecture where the model's carefulness doesn't matter.
The final 2026-07-28 MCP specification ships in two weeks. The release candidate has been locked since May 21st, which means the ten-week validation window the maintainers gave SDK authors and client implementers is almost up. If you're running MCP servers in production and haven't looked at what's actually in this release, that's the deadline you're working against, not some vague "sometime this year" future.
We already covered the headline change, the stateless protocol core, in a dedicated deep dive with the Kubernetes routing details. This post is about the rest of the release: what's shipping alongside statelessness, what's deprecated but not gone, and, more usefully, which of these changes need action from you before July 28 and which don't.
