Welcome to our technical blog — practical writing on platform engineering, GitOps, and AI-assisted operations. No thought leadership fluff. Just things that actually work in production.
Published: 22 September 2026 | Author: Surjit Bains
Vector search over your notes wastes tokens on near-misses. A 2-hop traversal of the links you already drew by hand cuts the same retrieval from 949 tokens to 367 — a 62% reduction — with nothing riding along that isn't actually connected.
Published: 15 September 2026 | Author: Surjit Bains
Harness engineering made one agent reliable. Graph engineering is what happens when running three agents turns reliability problems into coordination problems — and why declaring the topology up front beats discovering it at runtime.
HolmesGPT answers a third of your ops queue — a human still triages all of it. A triage router, two specialist agents, and a fail-closed risk policy that make it safe to remove the human router.
The plugins, skills, and repos that make Claude Code effective for platform work — and the token discipline that keeps it affordable. Two files, four repos, fewer plugins than you think.
A full BMAD walkthrough: spec-first build of a model-serving golden path, then the same method on an incident runbook loop. The PRD, the YAML, and what Quinn caught pre-merge.
Runbooks go stale because updating them is a separate manual task. Wire doc updates to merge events instead: an agent finds the affected docs, proposes the fix as a PR, and keeps your PlantUML diagrams in sync with reality.
Upbound launched Modelplane — a Crossplane-based control plane that brings declarative, GitOps-native fleet management to AI inference. Provisioning, scheduling via DRA + CEL, weight caching, and unified OpenAI-compatible routing across your entire GPU fleet.
CNCF's survey says two-thirds of GenAI inference runs on Kubernetes, yet only 7% of orgs deploy models daily. The bottleneck isn't GPUs — it's your release process. GitOps for models closes it.
A backdoored package, a poisoned allowlist, a database deleted in nine seconds. The 2026 incident record shows where agent security actually fails — and it's not where most teams are looking.
Golden paths only work if they're genuinely faster than the DIY alternative. Here's how to build one with Backstage and ArgoCD that developers actually use.
HolmesGPT's Kubernetes Remediation MCP server gates every mutation behind approval and allowlists — and why AWS and Azure don't have the same guardrails yet.
Your docs team writes markdown in Git. Your AI agents query Confluence via MCP. The python-mkdocs-to-confluence plugin closes the gap — and confluence_properties makes every page CQL-queryable without full-text search.
Every failed Confluence lookup is a runbook gap. Build a feedback loop that turns HolmesGPT investigation logs into drafted PRs — clustered by Claude, reviewed by humans, merged back into the knowledge base Holmes reads next time.
Skills frameworks, memory, multi-agent orchestration, A2A protocol — and a compatibility table showing exactly which of these works with Claude Code, Cursor, Copilot, and Windsurf.
BMAD brings agile upstream thinking to AI-driven development. Here's how to use it before you write a single Helm chart — and how Sam, the Platform Engineer agent, hands off directly into your GitOps pipeline.
OTel hit CNCF graduated status on May 21. The eBPF profiler adds the fourth observability signal to any Kubernetes workload without touching source code. Here's what changes for your platform team.
Prompt engineering gave us the words. Context engineering gave us the knowledge. Harness engineering gives us reliability. Swapping the harness changes SWE-bench scores by 22 points. Swapping the model changes them by 1.
The MCP spec RC removes Mcp-Session-Id and sticky routing at the protocol layer. If you've been holding off running MCP servers on Kubernetes because of session affinity headaches, that excuse just expired.
60,000+ public repos have an AGENTS.md. Almost none are written for a Kubernetes platform repo. Here's the template that covers forbidden paths, naming conventions, sync model, and secrets — plus a Kyverno policy to enforce it.
One AGENTS.md is a solved problem. Managing it across 40 repos, three git providers, and five teams without drift is not. Here's the three-zone file structure and provider-agnostic sync engine that keeps everything consistent.
Skills frameworks, multi-agent orchestration, A2A protocol — these are the repos shaping the agent layer in 2026. Here's what each one actually does and why platform teams should care.
Standard HTTP tracing tells you a model call took 3.2 seconds. GenAI semantic conventions tell you which tool caused it, how many tokens it burned, and why the finish reason was length not stop. Zero code changes with the OTel Operator.
MCP is how an agent talks to tools. A2A is how one agent talks to another. 150+ organisations including AWS, Microsoft, and Salesforce are running A2A v1 in production. Here's what the split means for your platform architecture.
Most Cursor users are still on .cursorrules. The new .cursor/rules/*.mdc format lets you scope rules to specific file types, activate them conditionally, and ship multiple rule sets per repo. Here's what changes for platform engineering teams.
.github/copilot-instructions.md loads into every Copilot Chat session in your repo automatically. It covers different surface area from AGENTS.md — conversational context rather than agent constraints. Here's what to put in it.
CLAUDE.md isn't just a rules file — it's a configuration layer for Claude Code. Memory blocks, MCP server declarations, PreToolUse hooks, custom slash commands, and sub-agents all live here. Here's the half most teams are missing.
Every agent session starts from scratch. experience-as-skill extracts your frameworks, decision patterns, and voice from your CV, git history, notes, and Slack — into a single skill file any agent can load.
Karpathy's four CLAUDE.md rules cut AI coding mistakes from 41% to 11%. Here's why they're the missing discipline layer for platform teams running agents in production.
A few months ago I wrote about harness engineering: the discipline of building constraints and feedback loops around an agent so it stops making the same mistake twice. I closed that post with a line I meant sincerely: harness engineering is early, and it's just getting started. I didn't expect "started" to mean four months.
Here's what actually happened. Teams that got good at harnessing one agent did the obvious next thing: they ran three. A coding agent, a review agent, a deploy-gate agent, each one solid on its own, each one running its own loop. And then they hit a wall the harness was never built for. Nobody told the review agent it wasn't the one deciding whether to merge. Two agents grabbed the same file at the same time, and each one thought it owned the fix. The harness caught every mistake it was designed to catch. It caught none of these, because these aren't reliability problems. They're coordination problems.
The line that's going around
A post from July 2026 put it in one sentence that stuck: "Loop engineering was the last unlock. Graph engineering is the next one. Agents are graduating from while-loops to org charts." Anthropic's own multi-agent research architecture backs it with a number: a lead agent delegating to parallel sub-agents beat a single agent by 90.2% on their internal research eval.
If prompt, context, and harness were the first three eras of working with AI, this is the fourth one: graph engineering.
I wired an LLM into my Obsidian vault a while back, the PARA-plus-LLM setup I wrote about in Your Second Brain, Now With an AI Inside It. It's genuinely useful. It's also, if I'm honest about the mechanics, wasteful in a way I didn't notice until I started thinking in graphs for a different reason entirely.
Ask it a question and you've got two bad options. Paste in the relevant folder, or the whole vault if you're not sure which folder, and burn tokens on notes that turn out to have nothing to do with the answer. Or let a "chat with your notes" plugin do semantic search over embeddings, which retrieves whatever's nearby in vector space, and vector space doesn't know the difference between "these two notes discuss similar-sounding infrastructure concepts" and "these two notes are actually about the same thing."
The number that made me rebuild the retrieval step
Running the same anchor note through a plain 2-hop graph traversal instead of dumping the vault: 6 notes instead of 17, 367 tokens instead of 949 (a 62% reduction), and every note in that neighborhood was there because I linked it there on purpose, not because an embedding model thought it looked similar.
There's a better retrieval mechanism sitting in the vault already, and you built it yourself, one [[link]] at a time.
An agent asked to create a ticket. The policy parked it, because creating things is a tier-3 action and tier 3 waits for a person. I approved it. The API returned 200, the approval was consumed, and no ticket existed.
That is not the failure anyone warns you about. The fear with agents on a ticket queue is the rogue one that does too much. What actually bit me was an agent doing nothing at all while every layer above it reported success — including the approval gate that was supposed to be the safety mechanism. A gate that says "done" when nothing happened is worse than no gate, because now you've stopped watching.
This post is the stack underneath that: a triage agent in front of HolmesGPT and an ITSM specialist, so the whole ops queue gets a first pass from an agent, and a risk policy that decides what each of them may do unattended. And the part I'd read if I were you — what broke when I pointed it at a real Freshservice instance, which was almost nothing I'd predicted.
The one-line policy that makes this deployable
unlisted_action: tier_4_draft_only. Any verb the policy has never heard of gets drafted for a human instead of executed. Tool lists grow; policies don't grow by themselves. Fail closed on the verb you haven't tiered yet and "let an agent touch the ticket queue" becomes a Tuesday decision rather than a leap of faith.
Claude Code out of the box is a brilliant engineer with no context and expensive habits.
You've probably seen both halves of that sentence play out. The brilliance: it writes a correct Crossplane Composition faster than you can look up the schema. The expense: a colleague's first month bill, driven by sessions where the model re-read the same repo structure forty times, carried three plugins' worth of context it never used, and explained every change in four paragraphs nobody read.
The gap between those two experiences isn't the model. It's setup. And the setup for platform work is specific enough — GitOps repos, YAML-heavy diffs, conventions that actually matter — that generic "10 Claude tips" listicles don't cover it.
Here's the starter stack I'd give a platform engineer today: two files, four repos, a short plugin list, and the token habits that keep the bill boring. Nothing here is exotic. All of it compounds.
The cheapest token is the one you never send
A well-structured CLAUDE.md saves thousands of input tokens per session by answering questions the model would otherwise explore the repo to resolve. Community benchmarks of the Superpowers skill framework measured about 14% fewer tokens with better output quality. Setup is a cost lever, not a convenience.
Last month I wrote about BMAD the method. This post is BMAD the practice.
Here's the setup. In the 66/7 gap post I argued that the reason two-thirds of GenAI runs on Kubernetes while only 7% of orgs can ship models daily is a missing paved road: no standard pipeline from model registry to production traffic. Which raises the obvious follow-up question. Fine, someone has to build that golden path. How?
The honest answer for most platform teams: open a blank YAML file and start typing, with the architecture decisions happening implicitly, one resource at a time, in whatever order the YAML demands them. Three weeks later you discover that nobody decided what happens when the eval gate fails at 2am, because no artifact ever forced that question.
So I ran the build through BMAD instead. Twice, actually: once for the model-serving golden path, and once for a completely different problem, an incident runbook feedback loop, to test whether the method transfers or whether it only works on the thing I designed it around.
What the spec caught before the YAML existed
The upstream phase surfaced nine decisions for the golden path that would otherwise have been made implicitly mid-YAML — including the one that matters most in production: who gets paged when a model passes canary but fails the eval gate. Nobody had an answer. Now it's in the PRD, and the on-call rota knows.
The runbook said "restart the auth-service deployment."
We'd renamed it to identity-service eight months before. Nobody updated the runbook — why would they? The rename was a code change, not a docs task. So it sat there, quietly wrong, until a 2am incident when an engineer followed the runbook step by step and spent four minutes baffled before figuring out what had changed.
Four minutes doesn't sound like much. At 2am, in a live incident, it's an eternity.
You know that moment when someone suggests the solution is "better documentation culture"? I want to push back on that. Hard. The problem isn't culture — it's maintenance. Runbooks go stale because keeping them current is a separate manual task that lives in a different system from the code that made them stale. Nobody's going to do that reliably, no matter how much you talk about documentation culture.
The actual fix is to make the docs update when the thing they describe changes. And now you can.
Here's the pattern: code merges trigger an agent that finds the affected docs, proposes the updates as a PR, and keeps your diagrams in sync with the actual current architecture. You review it. You merge it. Done.
Documentation drift is a code problem, not a culture problem — stop trying to fix it with reminders and retros
Doc updates should be triggered by code changes, not by someone's memory; if it relies on memory, it won't happen
Architecture diagrams should be generated from source, not painted by hand and then gradually become lies
The update PR is the review gate — the agent proposes, a human verifies, and nothing merges automatically
The documentation drift problem is architectural, not cultural
Documentation goes stale because updating it is a separate manual task that lives in a different system from the code that made it stale. Reminders, retros, and "documentation culture" don't fix architectural problems. Wiring doc updates to merge events does.
If you've spent any time managing AI inference workloads in production, you know the mess. One team's running vLLM on a GPU cluster. Another's using KServe on a different cluster. A third handed everything to a managed provider because the ops burden was too much. Requests route manually. Nobody's quite sure which model is running where. And when something slows down at 2am, nobody knows where to look first.
Every tool in this space — vLLM, SGLang, KServe, NVIDIA Dynamo — is excellent at what it does. But they all solve the within-cluster problem. The fleet problem — placing models across available capacity, failing over across regions, routing by cost and sovereignty, caching weights so you're not downloading 70GB every time a pod restarts — has always been something the labs and hyperscalers built privately and kept to themselves.
On June 23, Upbound shipped Modelplane: the open source version of that missing layer. It's Apache 2.0, built on Crossplane, and it's early — v0.1, developer release. But the architecture is exactly right for platform teams already running Crossplane for infrastructure, and the problem it solves is real.
What Modelplane gives you in one sentence
Declare your inference topology as Kubernetes resources. Modelplane provisions the GPU clusters, schedules replicas across them using DRA and CEL selectors, caches model weights once per cluster, and routes everything through a single OpenAI-compatible gateway.
You know that moment in a platform review when someone puts up the slide showing your AI infrastructure? GPU nodes provisioned. Inference operator installed. Models serving traffic. Everyone nods. The platform is ready.
Then ask a different question: how long does it take to get a new model version into production? The room goes quiet. Someone mentions a Slack thread with the ML team. Someone else mentions a Jupyter notebook that does the deploy. It's Tuesday, and the model that shipped last week is still the one from March.
The 2025 CNCF Annual Cloud Native Survey put numbers on this. 66% of organisations hosting generative AI models run inference on Kubernetes. 82% run Kubernetes in production. And how many deploy models daily? 7%. Another 47% deploy "occasionally" — which is survey-speak for "when someone has a spare afternoon and nothing's on fire".
The gap that should bother you
Two-thirds of GenAI inference already runs on Kubernetes, but only 7% of organisations deploy models daily. That's not an infrastructure gap. Everyone bought the platform. It's a release engineering gap — and platform teams already know how to close those.
We've seen this exact shape before. It's what app delivery looked like before DORA metrics and deployment pipelines: infrastructure in place, release process artisanal. And the fix is going to look familiar too.
On March 24, two backdoored versions of the litellm Python package landed on PyPI. LiteLLM is the LLM gateway underneath CrewAI, DSPy, Microsoft GraphRAG, and dozens of other agent frameworks — roughly 3.4 million downloads a day. The backdoor was live for about three hours and pulled in something like 47,000 downloads before PyPI quarantined it. The attacker didn't do the reconnaissance by hand. A GitHub account calling itself hackerbot-claw, describing itself as an "autonomous security research agent powered by claude-opus-4-5," had spent the previous month scanning public repositories for exploitable GitHub Actions workflows, found one in Trivy's CI pipeline at Aqua Security, harvested a publish token, and pushed the backdoor itself. No human directed the attack after launch.
That's not a hypothetical. It's documented by researchers at Aikido and Snyk as one of the first cases of an AI agent used operationally, end to end, in a supply chain attack. And it's one entry in a 2026 incident record that reads nothing like the "AI security" conversation from even a year ago.
An ArgoCD ApplicationSet spun up a preview environment for a pull request back in March. The PR closed in April. Nobody deleted the environment, because nothing was watching for that — it just kept running, quietly billing, until someone noticed a line item in June and had to go figure out what "pr-4471-preview" even was.
That's not a story about a careless team. It's what happens by default when cost isn't something your pipeline enforces, only something a dashboard reports on after the fact. By the time a cost anomaly shows up on a chart, the money's already spent. The only question left is how long it takes someone to notice.
