Blog¶

April 10, 2026
10 min read

Repo‑Native AI Workflows: Keep AI Work in Git

We caught a configuration mistake because somebody ran the same AI query twice and got a different answer. The first run had recommended a change that was already applied. Neither the agent nor the engineer knew — because the first conversation had happened in a chat window three days earlier, and nobody had written it down.

That's the problem, right there. Not that the AI gave different answers. That's fine, models do that. The problem is that the first answer had nowhere to live. It wasn't in Git, it wasn't in a PR, it wasn't linked to the change it prompted. It existed in a chat thread that expired the moment the engineer closed the tab.

AI work in chat evaporates. The prompt, the reasoning, the decision, the output — all of it gone. The fix isn't to stop using AI. It's to stop treating AI work as a special category that doesn't need the same discipline as code work. Same thing: open an issue, work on a branch, open a PR, merge to main.

April 9, 2026
9 min read

AI Incident Triage: Faster Summaries, Safer Actions

It's 2:47am. The P0 alert just fired and you're squinting at your phone, still half-asleep, already dreading what comes next. You pull the logs. Then the dashboards — there are three of them, obviously. Then the last two deploy notifications. Then Slack, because maybe someone said something useful. By the time you've assembled a rough picture of what's actually happening, it's 3:05am.

Eighteen minutes gone. And honestly? You're not even sure you've found the right thing yet.

That's not on you — it's not a skills problem or an experience problem. Context gathering across multiple systems is just slow. Full stop. Even if you've been on-call for years and know the stack cold, you're still stitching together data points from half a dozen places under pressure at an ungodly hour.

AI doesn't fix incidents. But it absolutely can fix that first eighteen minutes.

April 9, 2026
9 min read

SLO‑Driven Automation: Closing the Loop from Alerts to Fixes

We had a 99.9% availability SLO for the API gateway. When it breached, we paged an engineer. And when the engineer showed up — bleary-eyed, possibly at 2am — the first three minutes were always identical. Check the error rate in Grafana. Check which pods were unhealthy. Restart the unhealthy pods. Every time. Same alert, same three steps, same outcome.

Think about what that actually means. A human being was pulled away from sleep to perform a task that had no variation. No judgement required. No creative problem-solving. Just: check, check, restart. The same three steps that a script could do in ten seconds.

That's the thing about mechanical work — it wears people down in a way that creative incidents don't. It's not just the lost sleep. It's the knowledge that you could have slept, because nothing you did tonight required you to be human.

SLO-driven automation is about drawing that line clearly: these remediations are always the same, so they happen without a person. Anything that isn't — anything that needs actual judgement — still pages a human. For the triage layer that runs before remediation kicks in — context gathering and structured summaries for on-call engineers — see AI Incident Triage.

April 8, 2026
10 min read

MCP in the Real World: Security, Permissions, and Operations

MCP is genuinely easy to demo. Spin up a server, wire it to Claude Desktop, watch it query your GitHub repo or list your Kubernetes pods. Five minutes, works perfectly, everyone's impressed.

Then five engineers are using the same MCP server. It's pointing at production. They have different levels of access and nobody's really thought about that yet. One tool call times out and the agent retries it three times against a slow Kubernetes API. A GitHub issue body contains text that looks a lot like an instruction, and it ends up in the model's context.

None of that is an edge case. That's just Tuesday for any team that's moved MCP out of their laptop and into shared infrastructure. This post is about what you actually need to build before that Tuesday arrives. If you're still getting your bearings with MCP fundamentals, start with Demystifying MCP first.

April 8, 2026
11 min read

Platform Scorecards: Automated Monthly Health Snapshots

Every month, the platform lead sent a status update that said "things are improving." And every month, nobody could tell if that was actually true.

No numbers. No trend. No comparison to last month. Just a paragraph of qualitative statements that everyone read politely and immediately forgot. And here's the thing — the platform team probably was doing good work. Lead time was down. MTTR had improved. The on-call burden had dropped significantly.

None of it was visible. And invisible good work doesn't build confidence, doesn't surface problems before they become incidents, and absolutely doesn't make the case for more investment. A monthly scorecard changes that — not because leadership suddenly starts caring about metrics, but because the conversation shifts from "how are things going?" to "why did MTTR spike in March?"

April 8, 2026
9 min read

Policy as Code + Agents: Guardrails That Actually Hold

Our platform agent opened a PR that would have exposed a database security group to 0.0.0.0/0. And here's the thing — it was doing exactly what we asked. The prompt said "make this service accessible from the VPN." The agent heard "open ingress from anywhere." Technically? Both solve the same problem. The agent wasn't wrong. It just didn't know why the constraint existed.

Without a policy gate, that change would have gone through. Nobody would have caught it until someone ran a security audit, or worse. With one, it didn't — the agent got the policy failure back as feedback and tried a more targeted fix instead.

That's what policy as code looks like in an agentic system. Not blocking humans from making mistakes — they've got code review for that — but catching what agents get wrong before a human ever sees the PR. For the change management tier model that determines which agent changes need policy checks in the first place, see Agentic Change Management.

April 7, 2026
9 min read

Agentic Change Management: Safer Automation at Scale

Week three. That's how long it took. We'd given our platform agent write access to the cluster, feeling pretty good about ourselves, and then a change landed in staging that nobody had reviewed. The agent had decided to scale down a deployment to save resources during low-traffic hours. Sensible logic. Terrible timing — a batch job was scheduled to run that night.

The deployment recovered quickly. But the moment it happened, something clicked: write access without a review gate isn't automation. It's a liability dressed up as automation.

So here's the pattern we settled on. Agents propose changes as PRs, humans decide whether to apply them, and rollback is always included before merge. Simple in principle. A bit more involved to actually build — which is what this post is about.

April 7, 2026
8 min read

Argo CD 3.4: Your Teams Notifications Just Broke

You know that moment when you set something up, it works perfectly, and you stop thinking about it? That's Teams notifications in Argo CD. You wired up the webhook, tested it, watched an alert land in the channel, and moved on to the next thing. Job done. Except it's not done anymore. As of 31 March 2026, Microsoft retired Office 365 Connectors — the underlying mechanism that Argo CD's Teams notifier was built on. No fanfare, no loud breakage. Your notification config still looks correct. The webhook URL is still there. Argo CD still thinks it's sending alerts. They're just not arriving.

Think about what that actually means in practice. Your sync failures are still happening. Your health degradations are still triggering. The alerts are being dispatched — they're just disappearing into the void before they ever reach your Teams channel. If your team's incident detection relies on those notifications, you've been operating with a silent gap since the end of March. Nobody panicked, because nothing obviously broke. That's the worst kind of failure.

Argo CD 3.4 ships a proper fix: a new Teams Workflows notification service that replaces the retired connector. It went GA on 4 May 2026. The migration takes about ten minutes. Let's walk through what happened, how to fix it, and what else is worth your attention in this release.

April 7, 2026
7 min read

Crossplane 2.2: What's New (and Why It Matters)

Here's a scenario that'll feel familiar if you've been writing Crossplane composition functions for a while. You've got a multi-step pipeline, something's failing, and you're staring at the Composite Resource conditions trying to figure out which function in the chain produced the wrong output. You can see that it failed. You can't see what was sent to the function or what came back. So you add debug logging, redeploy, reproduce the failure, squint at the logs, and eventually piece it together. It's not broken, exactly. It's just slower than it needs to be.

Crossplane 2.2 fixes that with the Pipeline Inspector. And honestly that's the headline — the rest of the release is the kind of solid incremental work that makes a platform more reliable without asking you to change much. Composition functions got smarter about version-awareness, a meaningful bug fix landed for function selectors, and the MRD controller got quieter in busy clusters. Worth knowing what changed before you upgrade.

April 7, 2026
9 min read

DevEx Metrics That Matter (and How to Automate Them)

Here's a scenario I find genuinely unsettling: a platform team that's shipping consistently, 95% test coverage, zero critical vulnerabilities, clean backlog. By every measure they're tracking, things are good.

Lead time is 14 days. Nobody's asking why.

Fourteen days from first commit to production isn't good — it's a quiet disaster. Developers are sitting on their changes for two weeks. Feedback loops are long, so rework costs compound. Every release is large, which makes deployment anxiety real and rollbacks terrifying. And the platform team has no visibility into any of it, because they're measuring the wrong things.

That's not a made-up story. It's distressingly common.

DORA metrics exist precisely to break this pattern. Four numbers — lead time, deployment frequency, change failure rate, and MTTR — that tell you whether software delivery is actually getting better or quietly getting worse. They're not perfect. But they're the closest thing the industry has converged on, and they measure outcomes instead of activity. That distinction matters enormously.