Skip to content

Blog

  • 8 min read

Platform engineering is a sociotechnical problem

Banner image Banner image

Platform engineering is a sociotechnical problem

This was one of the better platform talks at KubeCon because it dealt with a problem most teams eventually hit and rarely admit cleanly: the platform looks solid, the engineering is serious, the abstractions are tidy, and yet users still keep finding ways around it.

Sony Interactive Entertainment used that tension as the starting point for a much more honest discussion about internal platforms. The core lesson was simple: once a platform serves multiple teams with different constraints, architecture stops being the whole story. What matters just as much is how teams interact, how decisions are made, and how you measure whether any of it is actually helping.

Quick takeaways

  • Platform bypasses are usually feedback, not just non-compliance.
  • Better architecture helps, but it does not remove coordination failure.
  • Product thinking starts when you measure adoption and usability, not just delivery.
  • "Done" should mean relied on, not merely implemented.

Producer-consumer platform model Producer-consumer platform model

The producer-consumer framing cuts through coordination sprawl

Once a platform serves multiple teams, "who needs to talk to whom and when?" becomes the real bottleneck — not the architecture. Sony's producer-consumer framing is a practical tool for answering that: identify who owns a capability, who depends on it, and when interaction is actually necessary. Reduce implicit dependencies and make the contracts explicit. That's where coordination overhead shrinks.

What was getting in the way

Sony described a pattern that will sound familiar to a lot of platform teams.

They had already done the work people normally recommend:

  • operators
  • pipelines
  • standardised deployment paths
  • abstractions and golden paths
  • cross-functional platform teams

From the outside, the platform looked mature.

But inside the organisation, the signals told a different story:

  • backlog kept growing
  • teams kept asking for exceptions
  • some teams wanted direct access to lower-level infrastructure
  • other teams bypassed the platform entirely and rebuilt pieces themselves

That was not happening because users wanted chaos. It was happening because the platform did not always fit the reality of the teams consuming it.

That shift matters. Once you see those behaviours as product feedback instead of governance failure, the problem becomes much clearer.

Why architecture was necessary but not sufficient

One part of the talk focused on architecture patterns that still matter a lot:

  • controller logic
  • reconciliation loops
  • clear resource contracts
  • composable boundaries
  • producer-consumer relationships between teams

That is good platform engineering. It gives teams cleaner interfaces and a more reasonable mental model.

But Sony's experience was that architecture improvements did not automatically solve scaling problems between teams. As the number of consumers and capabilities grew, coordination overhead grew with it.

That is where a lot of platform teams get stuck. They keep refining the technical model while the real friction is increasingly organisational.

Centralising coordination doesn't solve coordination problems

Sony hit this directly: every dependency routed through the same enablement team looks like a solution but creates a bottleneck. If your team is the unavoidable review step for every capability change, you haven't reduced coordination overhead — you've just consolidated it somewhere it's easier to observe. The fix is clearer capability boundaries and more local decision-making, not a bigger central team.

What broke next: team interaction at scale

Sony described how a previously simple communication model stopped working once more teams entered the platform ecosystem.

In the earlier stage, a team that needed something could just speak directly to the one other team that owned it. Responsibilities were clear. Boundaries were understandable. Coordination cost was manageable.

As the platform expanded, that turned into a mesh of dependencies. A single capability could require several teams in the room at once. Release delays, hidden dependencies, and last-minute escalations became more common. Small changes could take weeks.

They responded by mapping team interaction paths and trying to make the delivery flow more understandable. That helped at first, but it also exposed a trap: too many interactions began converging through an enablement team.

That is a useful warning sign.

If every dependency ends up routed through the same group, you have not solved the coordination problem. You have centralised it.

The better question: who produces, who consumes, and when do they need to talk?

One of the most practical ideas in the talk was shifting the conversation towards producer-consumer relationships.

That framing helps answer three useful questions:

  • who owns this capability?
  • who depends on it?
  • when is interaction actually necessary?

That is especially important in globally distributed organisations, where you cannot afford to solve every ambiguity with another recurring meeting.

Sony used this thinking to define clearer capability boundaries and encourage more local decision-making by teams, while still aligning with higher-level organisational goals.

That is a good model for platform teams in general: fewer implicit dependencies, fewer permanent coordination channels, more explicit contracts.

Why product thinking changed the control loop

This was the strongest part of the talk.

Sony realised they had excellent operational observability of their infrastructure, but weak observability of whether the platform itself was working as a product.

They could answer questions like:

  • what is the CPU usage of this cluster?
  • what is the health of these nodes?
  • what is the state of these services?

But they could not answer the more important product questions:

  • are teams actually adopting the capability we built?
  • how quickly do users get value from it?
  • is it easy to use?
  • does it reduce or increase workarounds?

That is the shift from platform as project to platform as product.

The roadmap can no longer be measured only by milestones, scope, or backlog movement. Those metrics tell you whether you shipped something. They do not tell you whether anyone wanted it in the form you shipped.

Sony changed the control loop by looking at:

  • time to value
  • adoption
  • user satisfaction
  • operational efficiency

That is a much healthier set of signals for an internal platform.

Apply this: change what 'done' means on your next delivery

Try this for one capability in the next sprint: don't count it as done when it ships — count it as done when a team uses it without help and doesn't file an exception. That one change in definition will surface documentation gaps, UX friction, and missing support paths that the delivery milestone never would.

The most useful idea: change what "done" means

Their redefinition of done is worth copying.

Previously, done meant a feature or project had been completed and delivered.

After the shift, done meant something users could rely on.

That includes more than implementation:

  • integrated
  • documented
  • supported
  • adopted
  • useful enough that teams actually depend on it

That is a better bar for platform work.

Internal platforms accumulate debt very quickly when teams celebrate delivery before they prove usefulness.

What I would copy from this talk

If I were applying this in a real platform team, I would start with four things.

1. Treat platform bypasses as product research

If teams keep building side paths, do not start with policy. Start with diagnosis.

Which need is not being met? What is too rigid? What is too slow? What is too opaque?

2. Reduce dependency sprawl between teams

If five teams need to coordinate for one normal capability change, you probably have a boundary problem.

Make the interfaces clearer or move the ownership.

3. Add product signals to platform reviews

Ask about adoption, time to first value, satisfaction, and workarounds alongside the usual reliability and capacity metrics.

Without those signals, platform teams can be very busy and still strategically off course.

4. Raise the definition of done

Do not count a capability as complete just because it shipped.

Count it as complete when teams can use it confidently and stop reaching for alternatives.

The bigger point

The most important line in the talk was not about Kubernetes or controllers. It was that scaling the platform required changes across three layers:

  • architecture
  • team interaction
  • feedback loops

That is the right model.

Most internal platforms struggle because one of those three layers is improving while the others remain stuck.

You can have clean abstractions and weak feedback. You can have strong teams and poor boundaries. You can have good infrastructure and bad product thinking.

Platform engineering starts to look more effective when those three layers move together.

References

Frequently asked questions

Where can I watch the Sony KubeCon session?

The talk was presented by Hagen Eugenia at KubeCon EU 2026. Find it on the CNCF YouTube channel — search "Sony platform engineering KubeCon EU 2026". The conference PDF is also linked in the References section.

How do you identify whether your platform has a product problem vs a technical problem?

Ask two questions. First: do teams that understand the platform well still find workarounds? If yes, it's a product problem — the capability doesn't fit the use case. Second: are bypass rates going up while features are being shipped? If yes, you're building for the platform's vision rather than user needs. Technical problems disappear when you fix the code; product problems persist regardless of code quality.

Is this 'platform as product' approach only relevant for large organisations like Sony?

No — the feedback loop breakdown Sony described (building without measuring adoption) happens at 5 engineers as easily as at 500. The scale determines the complexity of the symptoms, not the underlying cause. Small platform teams benefit from product thinking even more, because wasted effort on unused capabilities is proportionally more damaging when the team is small.

How does the sociotechnical framing connect to day-to-day platform work?

Sociotechnical means the technical and organisational systems are coupled — you can't improve one without considering the other. In practice: every architectural decision has a team interaction implication. Adding a new capability changes who needs to coordinate and how. If you design the architecture without mapping the coordination impact, you'll ship clean code into a messy coordination pattern and wonder why adoption is slow.

What metric would you watch first to know if the product-thinking shift is working?

Exception volume — the count of teams requesting direct access, custom deployment paths, or policy exemptions. If the platform is working as a product, exceptions should decrease as capabilities improve. If exceptions are growing alongside feature delivery, the platform is solving problems teams don't have while missing ones they do. Track it quarterly.

  • 5 min read

TAG DevEx in action: a practical model for reducing developer friction

Banner image Banner image

TAG DevEx in action: a practical model for reducing developer friction

The TAG Developer Experience panel at KubeCon EU 2026 was valuable because it translated DevEx from broad rhetoric into a concrete execution model.

Instead of treating DevEx as a permanent umbrella topic, the group is running short, scoped initiatives with clear outputs and explicit community input channels.

The visual map

TAG DevEx initiative loop TAG DevEx initiative loop

The three-pillar DevEx framing

TAG DevEx described developer experience across three connected pillars:

  • developer tooling (inner loop and outer loop)
  • application runtime realities (communication patterns, topology, tenancy)
  • platform enablement interfaces (golden paths, policies, paved roads)

That framing is useful because it stops teams from reducing DevEx to UI polish while ignoring runtime and platform constraints that create most daily friction.

Short initiatives with explicit scope beat open-ended DevEx programs

The TAG DevEx operating model — 3-6 month cycles, limited scope, clear deliverables, community input — is directly transferable to internal platform teams. Long-running "DevEx transformation" programs become fuzzy and hard to defund even when they stop delivering. Short cycles create accountability and ship things people can evaluate. Borrow this model for your next platform improvement initiative.

The initiative model is the main contribution

A standout idea from the session was not a new framework. It was the operating model itself.

Initiatives are framed as:

  • short lifecycle (typically 3-6 months)
  • limited, explicit scope
  • clear deliverables
  • cross-TAG and community input

This is a strong pattern for platform organizations too. Long-running "DevEx transformation" programs often become fuzzy. Short cycles with explicit outputs create accountability and momentum.

What is active now

The panel highlighted several active or emerging initiative lines.

Security and compliance through a DevEx lens

The objective is to collect real examples where security guidance either reduced friction or increased it, then use that evidence to improve practical adoption patterns.

This is the right approach: measure both security outcome and workflow impact.

AI-assisted development in CNCF projects

This initiative is collecting maintainers' and contributors' real-world use patterns, pain points, and value areas for AI-assisted SDLC.

The expected artifact is practical guidance that helps teams adopt AI with fewer blind spots.

Application dependency specification

The dependency initiative targets a common pain point between app and platform teams: unclear dependency contracts in development-to-deployment handoffs.

The team is exploring both runtime-observed and code-declared models rather than locking into one mechanism too early.

AI inner-loop experience (emerging)

There is also active interest in standardizing better local AI development experience, with open calls for contributors and leads.

Apply this: map your top DevEx pain points to the three pillars

Before your next platform roadmap session, map your top five developer friction points to the three pillars: developer tooling, application runtime realities, and platform enablement interfaces. Most teams will find their friction clusters in one or two pillars — that tells you where to focus. If everything lands in "platform enablement," your paved roads need work. If it's "developer tooling," the inner loop is the problem.

Why this matters for enterprise platform teams

The panel was framed in CNCF terms, but the model transfers directly to internal platform teams:

  1. define DevEx scope across tooling, runtime, and platform interface layers
  2. run short initiatives with concrete outputs
  3. gather practitioner evidence before drafting standards or policy
  4. close the loop from output back into next iteration

This is how teams avoid strategy theater and actually reduce friction.

Collecting friction stories before prescribing controls is not optional

The TAG DevEx security initiative explicitly collects evidence of where security guidance reduced friction vs increased it before drafting any recommendations. Internal platform teams frequently skip this step — they know what the platform should do and build it without checking whether it matches how developers actually work. Controls prescribed without evidence of the problem they solve create the friction they're meant to prevent.

Immediate actions you can take

  • map your top five DevEx pain points to the three-pillar model
  • spin up one 90-day initiative with named owners and measurable outputs
  • collect friction stories from developers before prescribing new controls
  • evaluate security and AI initiatives by both outcome quality and developer cost
  • document app-platform dependency expectations as explicit contracts

References

Frequently asked questions

Where can I watch the TAG DevEx KubeCon session?

The panel is on the CNCF YouTube channel. Search "TAG Developer Experience KubeCon EU 2026" or "Next Chapter Developer Experience TAG DevEx". The CNCF TAG DevEx also has a public GitHub repository with initiative documentation and meeting notes.

How does the AI-assisted development initiative connect to what my team is building?

TAG DevEx is collecting real-world patterns from CNCF maintainers: where AI-assisted development creates genuine value, where it introduces new friction, and what guidance helps teams adopt it safely. If your platform team is evaluating AI tooling for developers, this initiative's output will be more useful than vendor positioning — it's practitioner evidence from open-source contributors who have no reason to oversell.

How long does a TAG DevEx initiative typically take?

3-6 months from kick-off to published output. The security and compliance initiative was scoped to evidence collection and pattern synthesis — not a new standard. The AI inner-loop initiative was in early formation at KubeCon. Shorter scopes produce more finished outputs; longer initiatives tend to drift. If you run similar initiatives internally, 90 days is a good maximum before reassessing scope.

Can enterprise teams contribute to TAG DevEx initiatives?

Yes — TAG DevEx is an open CNCF working group. Enterprise practitioners with real evidence of DevEx friction (or improvements) are exactly what the initiatives need. The security initiative specifically asked for examples from regulated environments where security requirements created developer friction. Check the CNCF TAG App Delivery and TAG DevEx GitHub repositories for how to get involved.

How does TAG DevEx relate to platform maturity models?

TAG DevEx produces guidance, not prescriptive maturity levels. The three-pillar framing (tooling, runtime, platform enablement) gives a vocabulary for describing DevEx problems without imposing a single measurement framework. If you want a maturity model, the CNCF Platform Engineering maturity model is the better reference — TAG DevEx focuses on reducing specific friction, not on rating platform completeness.

  • 6 min read

Using AGENTS.md for Platform Engineering

Banner image Banner image

Using AGENTS.md for Platform Engineering

Most platform incidents do not start with a complicated bug. They start with confusion.

One team follows one release path, another skips a check, and somebody quietly changes a naming rule without updating the docs. A few weeks later, nobody agrees on what "standard" means.

That is where AGENTS.md helps. It gives the team one shared playbook for planning, delivery, review, and release. Humans can follow it, and agents can follow it too.

The aim is straightforward: move changes from idea to production in a way that is calm, repeatable, and auditable.

Why AGENTS.md beats a wiki page

A wiki page gets out of date the moment the first person forgets to update it. AGENTS.md lives in the repo — it travels with the code, gets reviewed in PRs, and can be read by both humans and agents. The same file that guides your team guides Copilot and Claude Code.

Why AGENTS.md matters

Platform engineering gets expensive when every change follows a different path. AGENTS.md gives you a default path that people can trust.

In practice, it helps you:

  • Set clear scope before work starts
  • Keep delivery steps consistent across teams
  • Build quality checks into the workflow
  • Onboard new engineers faster
  • Let agents help without losing control

What to include in AGENTS.md

Keep it short and specific. If it reads like policy prose, people will ignore it.

These six parts are usually enough:

  1. Mission and scope
  2. Workflow steps
  3. Owners (human or agent)
  4. Quality gates
  5. Standards and conventions
  6. Output locations (docs, repos, diagrams)

Practical workflow for platform changes

Workflow overview Workflow overview AGENTS.md control plane container view AGENTS.md control plane container view

Use one repeatable flow for each meaningful platform change:

  1. Define
  2. Write the change goal, scope, and success criteria.
  3. Assess
  4. Capture current state, dependencies, and risks.
  5. Design
  6. Pick the approach and record trade-offs.
  7. Approve
  8. Confirm quality gates before implementation.
  9. Implement
  10. Make the change with tests and automation.
  11. Document
  12. Update runbooks, diagrams, and rollback steps.
  13. Validate
  14. Run operational checks in non-production first.
  15. Release
  16. Ship with owner sign-off and monitoring.

This sequence reduces rework because decisions are made early, before implementation starts to drift.

Agent collaboration flow

Agent collaboration flow Agent collaboration flow

AGENTS.md enforces one rule that matters: no phase moves forward without the required output from the previous phase. That single rule prevents a lot of avoidable regressions.

A minimal AGENTS.md you can copy

Start with this and adapt it:

# AGENTS.md

## Mission
- Keep platform delivery consistent and low risk

## Workflow
1. Plan → define scope, success criteria, risks
2. Survey → assess current state + gaps
3. Ideate → propose options + trade-offs
4. Review → approve approach + gates
5. Build → implement + automate
6. Document → runbooks + diagrams
7. Validate → operational review
8. Ship → release readiness

## Quality gates
- Every change has an owner
- Risks documented before build
- Docs updated before release

## Standards
- Naming: <team>-<service>-<env>
- Environments: dev → staging → prod
- Tooling: Helm + ArgoCD

If this reads like a checklist, that is intentional. Checklists are easier to follow when teams are busy or under pressure.

Practical examples

Each example maps directly to the workflow above.

1) Platform release checklist

  • Why: reduce release risk and keep gates consistent
  • What: release plan, rollback plan, communication checklist
  • How: run one fixed checklist and require sign-off at each gate

2) Incident runbook

  • Why: improve incident response speed and clarity
  • What: roles, steps, and post-mortem template
  • How: trigger the runbook and record actions as structured output

3) Infrastructure bootstrap

  • Why: create new environments without one-off setup work
  • What: baseline stack (ArgoCD, secrets, policies, monitoring)
  • How: define the exact bootstrap sequence in AGENTS.md

4) SLO review

  • Why: make reliability reviews repeatable
  • What: SLO attainment, regressions, and next actions
  • How: generate monthly SLO summaries from metrics

5) Sprint review deck (Marp)

  • Why: remove manual slide preparation
  • What: request volume, average handling time, top requester, top request type
  • How: use agents to populate a Marp deck from metrics scripts

Companion repo

Walkthroughs and scripts live here: https://github.com/polarpoint-io/ai-capabilities

How OpenClaw uses it (optional)

In OpenClaw, AGENTS.md acts as a shared source of truth for multiple agents. It defines:

  • Which agent handles each phase
  • The outputs required for each phase
  • The gates a lead agent must approve

This keeps multi-agent execution safe and predictable.

Common anti‑patterns

Avoid these mistakes:

The vague gate tell

"Review done" is not a quality gate. Neither is "approved." If someone has to ask what done means at 2am during an incident, the gate failed. Write gates as specific, verifiable conditions: "All readiness checks pass in staging" or "Rollback steps tested against previous image tag."

  • Too long: AGENTS.md should be a few screens, not a wiki
  • Vague gates: "review done" is not a gate - define what done means
  • No ownership: every phase needs a named owner
  • Stale rules: update it when tooling or workflows change

What you get

  • Predictable platform delivery
  • Lower operational risk
  • Faster onboarding
  • Consistent documentation
  • A workflow that scales as agent use grows

Apply this: start with the minimal template

Copy the minimal AGENTS.md template in this post verbatim. Run it for one sprint. You'll naturally identify what's missing from your specific context — team names, environment names, tooling specifics. Add those in. Don't try to write the perfect AGENTS.md upfront; iterate toward it.

Final note

AGENTS.md only works when teams treat it as part of delivery, not an afterthought. Keep it current, keep it short, and make it the first thing people check before they start work.

Frequently asked questions

How long should an AGENTS.md file be?

Short enough to read in under five minutes. If it runs beyond two or three screenfuls, split it: put the core workflow in AGENTS.md at the root, and push domain-specific detail into subdirectory AGENTS.md files. A 500-line AGENTS.md is a wiki page in disguise — nobody reads it under pressure.

Does AGENTS.md replace runbooks?

No. AGENTS.md defines the process — how changes move from idea to production. Runbooks define the procedure for specific operations: how to restart a service, how to rotate a secret, what to do when the database connection pool exhausts. Both belong in the repo; they answer different questions.

Can agents actually follow AGENTS.md without human help?

Yes, for well-defined, bounded tasks — querying metrics, updating diagrams, populating sprint review data. For anything involving judgment calls (is this risk acceptable? should we proceed?) the agent should pause and ask. The workflow in this post builds those confirmation gates in explicitly.

What's the difference between AGENTS.md and CLAUDE.md?

AGENTS.md is the open standard supported by VS Code Copilot, Cursor, and Claude Code (via import). CLAUDE.md is Anthropic's format, specific to Claude Code, with features like @path imports and .claude/rules/ for path-scoped overrides. Start with AGENTS.md for maximum portability; add a CLAUDE.md that imports it if you want Claude-specific features.

How do you keep AGENTS.md from going stale?

Treat it like production code. Every PR that changes a workflow, tool, or standard should include an AGENTS.md update if relevant. Add it to your PR template as a checkbox: "Does this change require an AGENTS.md update?" That single prompt catches most drift before it accumulates.

  • 9 min read

The future of IDPs: Agentic Backstage

Banner image Banner image

The future of IDPs: Agentic Backstage

This talk hit a nerve. Most of us have seen a developer portal with good intentions and mixed adoption. The interesting bit here was not just "add AI", but where and how the AI showed up in the workflow.

Quick takeaways

  • A conversational layer helps only when catalogue metadata is reliable.
  • Keep self-service actions close to existing engineering workflows.
  • Focus on a few high-frequency tasks first: scaffold, search, deploy, troubleshoot.
  • Measure reduced cognitive load with real usage metrics, not feature counts.

Catalogue quality is the prerequisite

A conversational interface that surfaces stale metadata faster than a form wizard is worse than no interface at all. Before investing in the AI layer, audit catalogue completeness: owner annotations, up-to-date TechDocs, explicit deprecation markers, and relationship metadata (provides/consumes). The AI amplifies whatever quality is already there — good or bad.

What was getting in the way

At BackstageCon EU 2026 (co-located with KubeCon), Sam Nixon from Roadie presented a fundamental challenge with Internal Developer Portals: complexity breeds abandonment. Traditional Backstage requires developers to:

  • Navigate sprawling service catalogs with thousands of entries
  • Fill complex scaffolding forms with dozens of fields
  • Click through multi-step workflows
  • Search documentation across fragmented sources
  • Context-switch between portal and actual coding tools

Result: Cognitive overload, low adoption, platform teams scratching their heads wondering why developers avoid the portal.

What we actually wanted

An IDP that meets developers where they think - natural language interactions that reduce friction, provide contextual help, and feel more like pair programming than form-filling.

Architecture: conversational IDP

Agentic Backstage container view Agentic Backstage container view

Core concept: agentic software catalogue

Transform Backstage from a static catalogue into an intelligent agent that:

  1. Understands intent from natural language queries
  2. Queries the catalogue programmatically via Backstage APIs
  3. Takes actions (scaffold projects, trigger workflows, update metadata)
  4. Provides context (documentation, examples, related services)

The technical stack

Backstage Foundation: - Software catalogue (services, APIs, docs, teams) - Plugin ecosystem (scaffolder, search, TechDocs) - Service ownership metadata - Golden paths and templates

AI Layer: - LLM integration (Claude, GPT-4, local models) - Model Context Protocol (MCP) for Backstage APIs - Natural language → structured queries - Contextual prompt engineering with catalogue knowledge

Real-world use cases

1. Natural Language Scaffolding

Traditional Backstage: 

1. Navigate to Create Component
2. Select template from dropdown (50+ options)
3. Fill form: Name, Description, Owner, Repo URL, Branch, Pipeline Type, Deployment Target, Resource Limits, Monitoring Config...
4. Click through 3-4 review screens
5. Submit and hope you got everything right

Agentic Backstage: 

Developer: "Create a new Node.js microservice called 'payment-processor' 
            owned by the payments team, deployed to production with 
            standard monitoring"

AI Agent: "Creating payment-processor microservice:
           - Template: nodejs-microservice-standard
           - Owner: team:payments  
           - Deployment: production (via ArgoCD)
           - Monitoring: Prometheus + Grafana dashboards enabled
           - Repository: github.com/myorg/payment-processor

           Proceed? (yes/no)"

Developer: "yes"

AI Agent: "✓ Repository created
           ✓ CI/CD pipeline configured
           ✓ Service registered in catalogue
           ✓ Monitoring dashboards provisioned

           Next steps: Clone repo and see CONTRIBUTING.md"

Time saved: 15 minutes → 90 seconds

Traditional Backstage: 

Search: "authentication"
Results: 247 matches
- auth-service-v1 (deprecated)
- auth-service-v2 (deprecated)  
- auth-service-v3 (current? unclear)
- authentication-lib-java
- authentication-lib-python
- auth-docs-v1
- ... 241 more

Agentic Backstage: 

Developer: "Which authentication service should I use for my Python API?"

AI Agent: "Use auth-service-v3 (authentication.company.com/api)

           Why:
           - Only supported version (v1/v2 deprecated)
           - Python SDK: pip install company-auth-client
           - Owner: platform-security team
           - SLA: 99.9% uptime
           - Docs: [link to integration guide]

           Example integration:
           ```python
           from company_auth import AuthClient
           client = AuthClient(api_key=os.getenv('AUTH_API_KEY'))
           user = client.verify_token(request.headers['Authorization'])
           ```

           Need help with a specific integration? Just ask!"

3. Contextual documentation

Traditional Backstage:
Developer searches "how to deploy" → gets generic TechDocs → still doesn't know team-specific process

Agentic Backstage: 

Developer: "How do I deploy my service to production?"

AI Agent: "For YOUR service (payment-processor, owned by payments team):

           1. Merge PR to main branch
           2. CI creates Docker image: ghcr.io/myorg/payment-processor:v{VERSION}
           3. Update image tag in gitops/production/payment-processor/deployment.yaml
           4. Create PR to gitops repo
           5. Payments team lead approves (required per policy)
           6. ArgoCD auto-deploys within 5 minutes

           Monitoring: 
           - Grafana: [team-specific dashboard]
           - Alert channel: #payments-alerts

           Rollback:
           - Revert gitops PR
           - Or use: kubectl rollout undo deployment/payment-processor -n payments

           Want me to create the deployment PR for you?"

4. Self-service actions

Beyond Querying: Taking Action

Developer: "Add staging environment to my service"

AI Agent: "I'll scaffold the staging configuration:

           Creating:
           - gitops/staging/payment-processor/namespace.yaml
           - gitops/staging/payment-processor/deployment.yaml (2 replicas, smaller resources)
           - ArgoCD Application for auto-sync
           - Monitoring: staging-payment-processor Grafana dashboard

           Database: Do you need a staging database? (yes/no)"

Developer: "yes, PostgreSQL"

AI Agent: "✓ Terraform PR created for RDS staging instance
           ✓ Connection secret stored in Vault
           ✓ Added secret reference to deployment
           ✓ Updated migration job

           Staging environment ready in ~10 minutes.
           Access: https://payment-processor.staging.company.com"

Plugin ecosystem evolution

Traditional plugin challenges

  • Fragmentation: 150+ plugins, unclear which to use
  • Configuration: Each plugin requires YAML setup
  • Discovery: Hidden features, poor discoverability
  • Maintenance: Breaking changes, version conflicts

Agentic plugin management

Query Plugins Conversationally: 

Developer: "Show me deployment history for my service"

AI Agent: "I'll use the ArgoCD plugin to fetch deployment history for payment-processor:

           Recent deployments:
           - v2.3.1: Mar 24, 2026 14:30 - Healthy (current)
           - v2.3.0: Mar 24, 2026 09:15 - Rolled back (database migration issue)
           - v2.2.8: Mar 23, 2026 16:45 - Healthy (previous stable)

           Want details on the v2.3.0 roll back? Or see full sync status?"

Recommend Plugins Based on Context: 

Developer: "I want to see what's deployed in production"

AI Agent: "You'll need the ArgoCD plugin. I can:
           a) Enable it for your Backstage instance (requires admin approval)
           b) Show you the plugin configuration
           c) Connect you with platform team for access

           Which would you like?"

Implementation patterns

For platform teams

1. Catalogue quality matters more With AI agents querying programmatically, incomplete metadata becomes immediately obvious: - Ensure owner annotations on all components - Maintain up-to-date TechDocs - Tag deprecated services explicitly - Document relationships (provides/consumes APIs)

2. Golden paths as training data Your scaffolder templates become AI agent examples: 

# bad: generic template
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: nodejs-service

# good: rich context for AI
apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: nodejs-microservice-standard
  title: Node.js Microservice (Production-Ready)
  description: >
    Standard Node.js microservice template with:
    - Express.js REST API framework
    - PostgreSQL database integration
    - Prometheus metrics endpoint
    - Grafana dashboard provisioning
    - ArgoCD GitOps deployment
    - Jest testing scaffold

    Use for: customer-facing services, internal APIs
    Don't use for: batch jobs (use nodejs-job template), frontend (use react-app)

  tags:
    - nodejs
    - microservice
    - production
    - recommended

3. Structured prompts for common tasks Create reusable prompt templates:

# prompts/scaffold-service.md
When a developer asks to create a new service:

1. Determine programming language preference
2. Ask about deployment environment (dev/staging/prod)
3. Identify owning team from their Backstage profile
4. Suggest appropriate template based on:
   - Language → matching template
   - Team → team-specific standards
   - Service type → microservice vs batch job vs frontend
5. Confirm configuration before executing
6. After creation, provide:
   - Link to new repository
   - Link to CI/CD pipeline
   - Link to catalogue entry
   - Next steps for local development

4. Measure cognitive load reduction Track metrics before/after agentic features: - Time to complete scaffolding - Catalogue search → action completion rate - Support channel questions (should decrease) - Developer satisfaction surveys

For developers

5. Treat AI agent like a senior developer Best practices for prompts: - Be specific: "Create production Node.js API" not "make service" - Provide context: "I'm on payments team working on checkout flow" - Ask follow-ups: "Why that template?" - Request explanations: "Explain the monitoring setup"

6. Validate AI-generated actions Always review: - Repository configurations - Resource limits and scaling policies - Network and security settings - Database connection strings

The AI accelerates, but you verify.

The new interface tell

Intuit tried this with an ArgoCD UI extension — built it, shipped it, got poor adoption because experts went straight to logs and novices never opened Argo at all. The lesson: don't build new interfaces. Put AI where engineers already are (Slack, Teams, their IDE). A conversational layer in yet another portal tab will have the same adoption problem as the portal it's trying to fix.

Panel insights: ecosystem perspective

At the Backstage Plugin Ecosystem panel, maintainers from major organisations (Spotify, Red Hat, Roche, SAP, VMware) discussed:

Healthy ecosystem characteristics: - standardisation: Common patterns for plugin structure - Documentation: Clear setup guides and API docs - Versioning: Semantic versioning with migration guides - Community: Active Discord, GitHub discussions - Quality gates: Automated testing, security scanning

Agentic future: - AI agents will prefer well-documented plugins (better API understanding) - Natural language plugin invocation bypasses UI complexity - Contextual plugin recommendations based on catalogue metadata - Automated plugin configuration from conversational setup

What changed in practice

Before: Static catalogue requiring manual navigation, complex forms, hidden documentation
After: Conversational interface that understands intent, takes action, provides context

The shift from "Portal as Documentation Hub" to "Portal as Intelligent Assistant" fundamentally changes developer experience. Agentic Backstage doesn't replace the catalogue - it makes it accessible.

By reducing cognitive load through natural language, platform teams can achieve the original IDP promise: increasing developer velocity while maintaining standards.

Apply this: measure cognitive load reduction with real metrics

Track time-to-complete-scaffolding before and after. Track support channel question volume (it should decrease). Track catalogue search-to-action completion rate. "Developers seem happier" is not a metric. If you can't show a 50%+ reduction in time-to-first-deployment, the conversational layer isn't working yet.

Getting started

Experiment today

  1. Enable Backstage search API for programmatic queries
  2. Integrate LLM client (OpenAI, Anthropic, local Ollama)
  3. Create prompt templates for common tasks (scaffolding, catalogue search)
  4. Build simple chatbot using Backstage APIs
  5. Iterate based on developer feedback

Production considerations

  • Access control: AI inherits user permissions via tokens
  • Audit logging: Track AI-generated actions
  • Rollback procedures: Easy undo for AI mistakes
  • Prompt injection protections: Validate user inputs
  • Cost management: LLM API rate limits and budgets

Frequently asked questions

Does agentic Backstage require a specific LLM?

No — the architecture is model-agnostic. The AI layer sits between the user and the Backstage APIs, translating natural language to structured catalogue queries and scaffolder actions. You can use Claude, GPT-4, or a local model via Ollama. The choice affects response quality and latency, not the fundamental architecture.

How does the AI agent inherit user permissions in Backstage?

Via token passthrough. The AI agent operates with the authenticated user's Backstage token — it can only see and act on resources the user already has access to. There's no privilege escalation. This is the same model MCP uses: the protocol delegates auth to the underlying service.

What's the minimum viable first use case for agentic Backstage?

Natural language catalogue search. It's read-only, has no side effects, and delivers immediate value to developers who struggle with sprawling catalogues. Once that's working and trusted, layer in scaffolding (higher stakes, confirmation gate required) and then self-service actions (highest stakes, full audit logging required).

Can agentic Backstage create PRs or trigger pipelines directly?

Yes, within the scaffolder's existing action framework. Any action available in a Backstage template (create repo, trigger CI, register component) can be invoked by the AI agent. The key control: always present a confirmation step before executing irreversible actions. "Proceed? (yes/no)" is not optional for anything that creates or modifies external resources.

How do you prevent prompt injection in a public-facing IDP?

Validate all user inputs before passing them to the LLM. Treat user-supplied strings as untrusted data — don't interpolate them directly into system prompts. Use structured output formats (JSON schema validation) so the LLM can't be instructed to output arbitrary text that gets executed downstream. Audit log every AI-generated action with the original user input alongside it.

References

Presented at BackstageCon EU 2026 (co-located with KubeCon) by Sam Nixon (Roadie) and ecosystem maintainers

  • 7 min read

Building self-service platforms with Crossplane v2.0

Banner image Banner image

Building self-service platforms with Crossplane v2.0

I liked this session because it skipped the fluff and went straight to the awkward bit most platform teams know too well: developers waiting days, sometimes weeks, for fairly standard infrastructure requests.

Quick takeaways

  • Expose a smaller platform API to developers, keep complexity behind compositions.
  • Use Crossplane Projects to keep API, function code, and tests versioned together.
  • Treat metrics as an operations product, not just raw counters.
  • Start with one self-service path, then expand once usage is stable.

What was getting in the way

At KubeCon EU 2026, Jared Watts and Adam Wolfe Gordon (Upbound) presented a universal challenge in platform engineering: developers often wait weeks to deploy services due to infrastructure complexity, compliance requirements, and DevOps bottlenecks. Creating a database, configuring networking, setting up monitoring - each step requires coordination across multiple teams and tools.

What we actually wanted

A control plane framework that extends Kubernetes to orchestrate everything beyond containers - enabling platform teams to expose curated, self-service APIs to developers while maintaining guardrails and organisational best practices.

Crossplane v2 shifts the mental model from 'provisioning tool' to 'platform API'

CNCF graduation plus 3,000+ contributors signals this is infrastructure — not a bet. The Projects workflow is the feature that matters most for platform teams day-to-day: one repo for API definitions, composition logic, and tests means you stop fighting multi-repo synchronisation and start shipping platform capabilities at software velocity.

CNCF graduation milestone

Crossplane achieved CNCF graduation status with over 3,000 community contributors, cementing its position as the foundational framework for platform engineering. This maturity brings:

  • Production-proven stability across enterprises
  • Broad ecosystem support (900+ AWS services as Kubernetes APIs)
  • Active governance and security practices
  • Multi-cloud abstraction layer built on Kubernetes patterns

Architecture: control plane for everything

Crossplane v2.0 container view Crossplane v2.0 container view

Core concepts

1. Composite Resource Definitions (XRDs)
Define the shape of your platform API - what developers see and interact with: 

apiVersion: example.com/v1
kind: App
spec:
  image: my-container:v2.0
  database: postgres
  storage: 100Gi

Platform teams curate this experience, constraining options while maintaining flexibility.

2. Compositions
Implement the logic and transformation - how developer requests fan out into actual infrastructure: - Functions pipeline (gRPC-based, language-agnostic) - Python, Go, TypeScript, or simple Go templates - Transform XR → Deployment, Service, RDS instance, networking, scaling policies

3. Managed Resources
Represent cloud provider services as reconciled Kubernetes API objects: - S3 buckets, EKS clusters, RDS databases become kind: Bucket, kind: EKSCluster - Continuous reconciliation fixes drift automatically - Status conditions reflect real-world state

The Promise: From Weeks to Seconds

Before Crossplane:
Developer → DevOps ticket → Infrastructure team → Compliance review → Manual provisioning → Weeks elapsed

After Crossplane:
Developer applies simple App CR → Platform automatically provisions deployment + database + networking + monitoring → Minutes elapsed

Crossplane v2.0: developer experience improvements

The multi-repo problem

Traditional Crossplane development required juggling: - Repository for functions (Python/Go code) - Repository for configurations (XRDs, Compositions) - Dependencies spanning multiple repos - Manual synchronization on every update

Result: High cognitive load, brittle workflows, coordination overhead.

Crossplane Projects: unified development artifact

The v2.0 release introduces Projects - a single source repository containing: - API definitions (JSON Schema → XRDs) - Composition logic (functions) - Dependencies (providers, CRDs) - Tests (X-prin framework) - Versioning (unified releases)

Think of it like a modern application repository but for your platform APIs.

Live demo walkthrough

Adam demonstrated the new workflow at KubeCon:

1. Initialize Project

crossplane beta project init my-platform
cd my-platform

Creates structure: 

crossplane.yaml  # Project metadata, OCI registry
apis/            # API definitions
functions/       # Function code
compositions/    # Composition templates

2. Define API with JSON Schema

{
  "type": "object",
  "properties": {
    "image": { "type": "string" },
    "port": { "type": "integer" },
    "database": { "type": "string", "enum": ["postgres", "mysql"] }
  }
}

Generate XRD: 

crossplane beta project xrd generate api.json

3. Generate Composition and Function

crossplane beta project composition generate
crossplane beta project function add my-app-function

Creates Python function template with auto-ready baseline.

4. Write Function Logic

# functions/my-app-function/main.py
def compose(xr, observed, desired):
    # Extract values from XR
    image = xr.spec.image
    port = xr.spec.port
    db = xr.spec.database

    # Compose Kubernetes resources
    deployment = {
        "apiVersion": "apps/v1",
        "kind": "Deployment",
        "spec": {
            "template": {
                "spec": {
                    "containers": [{
                        "image": image,
                        "ports": [{"containerPort": port}]
                    }]
                }
            }
        }
    }

    service = {/* ... */}
    database_instance = {/* ... */}

    return [deployment, service, database_instance]

5. Test Locally

crossplane beta project render # Dry-run function pipeline
crossplane beta project run    # Spin up local kind cluster

Creates complete local environment with Crossplane + your project installed.

6. Validate with X-prin

# tests/app-test.yaml
xr:
  spec:
    image: nginx:1.21
    port: 80
    database: postgres

assertions:
  - deployment.spec.template.spec.containers[0].image == "nginx:1.21"
  - service.spec.ports[0].port == 80
  - database.spec.engine == "postgres"

Run tests: 

xprin test tests/app-test.yaml

7. Deploy to Production

crossplane beta project build
crossplane beta project push

Packages everything into OCI artifact, pushes to registry.

Resource State Metrics: granular observability

The second major v2.0 feature addresses operational visibility at scale.

The old problem

Traditional Crossplane metrics: - "15 EKS clusters are unhealthy" - But which clusters? Which teams affected? What's the scope?

The new solution: Resource State Metrics

Built on upstream Resource State Metrics project with Cel expressions:

apiVersion: metrics.crossplane.io/v1alpha1
kind: ResourceMetricsMonitor
metadata:
  name: eks-cluster-health
spec:
  resources:
    - apiVersion: ec2.aws.crossplane.io/v1beta1
      kind: Cluster

  metrics:
    - name: cluster_health
      help: "EKS cluster health by team and environment"
      labels:
        team: 'object.metadata.labels["team"]'
        environment: 'object.metadata.labels["environment"]'
        xr_name: 'object.metadata.labels["crossplane.io/claim-name"]'

      cel: |
        object.status.conditions.exists(c, c.type == "Ready" && c.status == "True") ? 1 : 0

Result: Prometheus metrics with team/environment/XR labels for precise troubleshooting.

Cardinality Management

cardinalityLimit: 100  # Prevent Prometheus explosions

Status shows current usage: 

status:
  observedCardinality: 12
  withinLimit: true

Grafana Dashboards

Query by team or environment: 

cluster_health{team="platform", environment="prod"} == 0

Answers: "Show me unhealthy clusters for the platform team in production."

The cardinality explosion tell

Resource State Metrics with unconstrained label cardinality will crash your Prometheus. Every unique combination of team, environment, and xr_name labels creates a new time series. Set cardinalityLimit on every ResourceMetricsMonitor and monitor status.observedCardinality before you hit the ceiling. The default demo config in the KubeCon talk had no limit — don't copy it into production.

Implementation blueprint

Apply this: start with one self-service path, validate usage, then expand

The most common mistake with Crossplane is building ten Compositions before anyone has used one. Define the API for your highest-frequency infrastructure request — usually a database or a deployment target — ship it to one team, measure time-to-first-deployment, and gather friction feedback. Expand only after the first path is trusted and well-understood.

For platform teams

  1. Adopt Crossplane Projects
  2. Migrate from multi-repo to unified project structure
  3. Version APIs + functions together

  4. Simplify developer onboarding

  5. Define Curated APIs

  6. Start with JSON Schema (familiar tooling)
  7. Constrain options (database sizes, instance types)

  8. Use familiar abstractions (App, Database, Queue)

  9. Write Functions in Preferred Language

  10. Python for data transformation
  11. Go for performance-critical logic

  12. TypeScript for web team expertise

  13. Deploy Metrics Monitoring

  14. Create ResourceMetricsMonitor for critical resources
  15. Extract team/environment labels
  16. Set cardinality limits per monitor

For developers

  1. Use Platform APIs 

    apiVersion: example.com/v1
kind: App
metadata:
  name: my-service
spec:
  image: my-org/my-service:v2.0
  database: postgres
  storage: 50Gi

  2. Self-Service Without Tickets

  3. No DevOps coordination
  4. No weeks-long waits
  5. Guardrails prevent misconfigurations

For organisations

  1. Measure Platform Success
  2. Track time-to-first-deployment
  3. Monitor ticket reduction

  4. Survey developer satisfaction

  5. Scale Incrementally

  6. Start with one team/use case
  7. Validate platform-market fit
  8. Iterate based on feedback

What changed in practice

Before: Infrastructure as code spread across Terraform, CloudFormation, Helm charts - manual coordination, weeks-long cycles
After: Unified Kubernetes API for everything - self-service with guardrails, minutes-to-deployment

Crossplane v2.0 shows platform engineering maturity: standardised patterns, better developer experience, and practical operational observability. With CNCF graduation and over 3,000 contributors, momentum in the ecosystem is strong.

The shift from "Platform as Code" to "Platform as API" fundamentally changes how organisations scale infrastructure operations.

References

Presented at KubeCon + CloudNativeCon Europe 2026 by Jared Watts & Adam Wolfe Gordon (Upbound)

Frequently asked questions

Is Crossplane v2.0 ready for production use?

Crossplane as a project is CNCF-graduated and production-proven at enterprise scale. The specific v2.0 features shown at KubeCon — the Projects CLI workflow and Resource State Metrics — were in beta at conference time. Check the Crossplane v2 release notes for GA status before adopting specific features in production workloads.

Where can I watch the KubeCon session?

The full session by Jared Watts and Adam Wolfe Gordon is on the CNCF YouTube channel. Search "Building self-service platforms Crossplane v2 KubeCon EU 2026".

How long does it take to go from zero to a working self-service API?

The KubeCon demo showed a working local environment in under 10 minutes using crossplane beta project init and crossplane beta project run. A production-grade Composition with tests, RBAC, and monitoring is a different scope — budget 1-2 sprints for a first domain. The local development loop being fast enough to use daily is what makes iteration tractable.

Do I need to rewrite existing Crossplane v1 Compositions for v2?

Not necessarily. v2 is backwards-compatible with v1 Compositions. The Projects workflow is additive — you can adopt it for new work without migrating existing Compositions. Namespace-scoped managed resources are the biggest operational change; plan that migration separately using the phased approach RBI described.

What's the X-prin testing framework mentioned in the demo?

X-prin is an open-source composition testing framework that lets you write declarative test cases: given this XR spec, assert these Kubernetes resources are generated. It's the Crossplane equivalent of unit tests for your composition logic. Find it at github.com/crossplane-contrib/xprin.

  • 10 min read

KubeCon EU 2026: what actually mattered

Banner image Banner image

KubeCon EU 2026: what actually mattered

KubeCon had no shortage of announcements, but the interesting part was not the volume. It was the convergence.

Across keynotes, maintainer updates, end-user sessions, and lightning talks, the same themes kept reappearing from different angles: platform teams acting more like product teams, AI moving into operational workflows, policy and governance becoming more automated but also more bounded, and internal platforms shifting away from ticket queues toward self-service APIs and reusable capability marketplaces.

If I strip away the noise, this is what I would actually take back to a platform team.

AI-driven GitOps visual AI-driven GitOps visual

Crossplane v2.0 visual Crossplane v2.0 visual

Agentic Backstage visual Agentic Backstage visual

The convergence pattern is the real signal

KubeCon EU 2026 wasn't defined by any single announcement. It was defined by the same themes appearing independently across keynotes, end-user talks, and maintainer updates: platform teams as product teams, AI in operational loops, policy becoming a workflow, and self-service replacing ticket queues. When six different speakers from different organisations reach the same conclusion, that's a durable signal — not a trend cycle.

The biggest takeaways

1. AI moved from novelty to operations

The strongest examples were not "AI writes code" demos. They were operational workflows where AI reduced real toil.

In the MCP + Argo CD talk, the standout point was this: teams got better adoption when they put AI in existing support channels, not in yet another portal tab. That sounds obvious in hindsight, but it is easy to miss when teams are excited by shiny UI work.

The same pattern showed up elsewhere too:

  • Backstage is moving toward a multi-surface operating model across UI, CLI, and MCP tooling.
  • Kyverno and Kagent showed policy operations becoming agent-assisted and closed-loop.
  • TAG DevEx is explicitly studying AI-assisted development based on real usage data rather than hype.
  • Multiple sessions treated agent experience as a real platform design concern, not just a gimmick.

If your engineers already troubleshoot in Slack, keep them there. Bring the capability to their workflow.

2. Platform APIs beat ticket queues

The Crossplane session made this painfully clear. Most teams still lose days waiting for routine infra changes because requests are encoded as tickets, not APIs.

Crossplane v2.0’s project model is a pragmatic step forward: - one project for API definitions, composition logic, and tests - local development loop that is fast enough to use daily - clearer path from platform intent to production behaviour

The big shift is cultural as much as technical: platform teams stop shipping "internal services" and start shipping products.

That same lesson was reinforced even more directly by the self-service platform session from DigitalOcean. Their core point was sharp: Kubernetes was not the bottleneck, the operating model was. Once environment creation moved from ticket queue to Backstage + Argo Events + Argo Workflows + virtual clusters + Kyverno guardrails, provisioning dropped from days to minutes.

This was one of the clearest signals of the conference: self-service is no longer a vague aspiration. It is a concrete architecture pattern.

3. Developer portals are becoming conversational and multi-surface

Backstage sessions pointed in one direction: discovery and self service are better when they feel like a conversation, not a form wizard.

That does not mean deleting structure. It means using structure behind the scenes while exposing a simpler interface to engineers.

The practical takeaway is to improve catalogue quality first. If metadata is stale, any agent layer will surface the mess faster.

The more mature version of this idea came from the Backstage maintainer update: platform teams should stop thinking in terms of a single portal UI and start thinking in terms of one platform model exposed through multiple operating surfaces. That is a much stronger design direction than just adding chat to an existing screen.

4. Governance is becoming a workflow, not just a rule set

One of the stronger late-conference themes was that policy is no longer just admission control.

Kyverno’s direction, especially in the multi-cluster governance talk, was toward full policy lifecycle management:

  • validation
  • mutation
  • generation
  • image verification
  • reporting
  • exemptions
  • cleanup

What made the session useful was the framing around operator workflow. The real problem was not only writing policies. It was checking them, testing them, troubleshooting them, and doing that across many clusters without drowning in manual work.

That is where agentic tooling becomes genuinely useful: not replacing deterministic policy enforcement, but reducing the human overhead around it.

5. Observability is getting more contextual

The Crossplane metrics discussion landed well because it moved beyond aggregate counts.

"15 clusters unhealthy" is not enough. Teams need: - which clusters - which team owns them - how long they have been degraded - what changed around the same time

This is where label strategy and cardinality discipline matter. Better questions, not just more dashboards.

6. Platform engineering has clearly become a measurement problem

The Norwegian public-sector platform maturity talk was one of the better reality checks of the week.

Platform adoption has clearly won. Internal developer platforms and Kubernetes are widespread, and tooling is converging even without central mandates. But measurement still lags. Teams know they are building platforms; they are less clear on how to prove those platforms are successful.

That gap matters. If teams cannot define what success looks like, they cannot prioritise the right improvements, justify investment, or distinguish real self-service from internal process theater.

7. The edge story is no longer theoretical

The CERN electric glider keynote reminded everyone that cloud native tooling now runs in places where power, network and weather are all hostile constraints.

That is relevant even if you do not run aircraft projects. The same patterns apply to factories, field devices and remote operations where assumptions from datacentre life break down quickly.

The telecom keynote made the same point from another angle: cloud native is no longer confined to datacentres and web apps. It is becoming part of how large, geographically distributed, highly specialised infrastructure is operated.

Apply this: pick one action from the 'What to do on Monday' list and start this week

The action plan at the end of this post isn't aspirational — each item is directly sourced from a KubeCon session with a concrete before/after. Pick the one that matches your biggest current pain point. Reduce one high-friction workflow, expose one capability as an API, or tighten one piece of catalog metadata. Small, boring improvements compound.

These were the trends that looked durable rather than fashionable.

1. Governed autonomy is replacing both central control and platform anarchy

The strongest sessions were not advocating fully autonomous systems. They were describing bounded autonomy:

  • AI in support channels with approval gates
  • action registries and standardised execution layers
  • policy engines underneath intelligent orchestration
  • self-service with declarative guardrails

This is a much more credible model than either “humans must approve everything manually forever” or “agents will just run the platform.”

2. Internal platforms are becoming capability marketplaces

Abby Bangser’s keynote was especially clear on this point. Platform teams cannot scale simply by adding more platform engineers. The longer-term answer is to let domain experts publish reusable platform capabilities while the platform team owns the standards, interfaces, and operating model.

That is a significant evolution from the older internal-platform-as-a-central-team model.

3. Kubernetes is expanding from container runtime substrate to broader systems operating layer

This showed up in multiple places:

  • accelerated and AI workloads
  • multiplayer gaming with Agones
  • edge and telecom environments
  • virtual clusters for developer environments

Kubernetes is no longer interesting only because it schedules containers. It is interesting because it provides a stable operational contract across increasingly different workload types.

4. The quality of metadata and interfaces is becoming a first-order platform concern

Backstage, agentic tooling, software catalog discussions, and even policy skills all pointed to the same issue: if metadata is stale, interfaces are inconsistent, or documentation is only designed for humans, the platform becomes harder to automate safely.

That is why catalog quality, policy reports, ownership labels, and machine-readable contracts kept resurfacing.

Exciting developments worth watching

These were the developments that felt most likely to compound over the next 12-18 months.

1. Backstage as a true multi-surface control plane

This is more interesting than “Backstage with AI.” If the action registry, catalog, permissions, CLI, and MCP surfaces keep converging, Backstage becomes a much more serious operational layer for internal platforms.

2. Closed-loop policy governance

Kyverno plus agentic orchestration is still early, but the direction is right: operators asking for reports, checks, installs, and remediation in one bounded flow rather than moving manually across ten tools.

3. Ticketless self-service environments with hard guardrails

The DigitalOcean session was one of the most practically useful examples at the conference because it showed a clear before-and-after operating model. It is easy to imagine many teams copying that architecture directly.

4. Platform engineering standards work getting more operational

TAG DevEx and the wider platform engineering community are moving from broad theory into scoped, measurable initiatives and reusable models. That is a healthier sign than another year of generic “platforms are products” slogans.

5. AI infrastructure normalising as a platform concern

The accelerator-native keynote, GPU fragmentation lightning talk, and agent-experience keynote all reinforced the same thing: AI is no longer a special side track. It is becoming another platform domain that needs scheduling, governance, observability, and usable interfaces.

The 'AI does everything' trap

The sessions that landed worst at KubeCon were the ones positioning AI as a wholesale replacement for operational discipline. The sessions that landed best treated AI as a narrow reduction in specific toil: support channel bots, change review layers, policy audit aggregation. If your AI plan can't answer "what specific manual step does this replace?", it's not ready for production.

What to do on Monday

If I had to boil this down into an action plan:

  1. Pick one high-friction operational workflow and reduce it with bounded automation rather than adding a new portal surface.
  2. Define one platform capability your developers request repeatedly and expose it as a productised API or template instead of a ticket.
  3. Tighten service catalog metadata, ownership fields, and machine-readable platform contracts before layering in more agents.
  4. Measure platform success explicitly: time to environment, ticket volume, adoption, failure rate, and developer satisfaction.
  5. Package repeated troubleshooting and governance work into reusable workflows, skills, or paved roads.
  6. Audit where your current platform still depends on specialist memory rather than sharable operational capability.
  7. Treat AI platform integration as a governance and interface design problem, not only a tooling problem.

None of this needs a big-bang rewrite. Small, boring improvements compound fast.

Where to go deeper

Platform operating models

AI, agents, and governance

Platform APIs and delivery systems

Developer surfaces and control planes

Official repos and resources

These are the primary project links that map most directly to the talks and posts above.

Platform APIs, GitOps, and self-service

Policy, agents, and governed autonomy

Observability, performance, and runtime operations

Broader cloud-native operating patterns

Full source notes

Frequently asked questions

Where are the KubeCon EU 2026 session recordings?

All sessions are published on the CNCF YouTube channel typically 1-2 weeks after the event. The KubeCon EU 2026 playlist will contain keynotes, maintainer updates, and breakout sessions. Individual posts in this series link to specific talk titles you can search for.

What was the single most actionable session for platform teams?

The DigitalOcean self-service environment session stood out for its clear before/after operating model — provisioning moved from days to minutes using Backstage + Argo Events + Argo Workflows + virtual clusters + Kyverno guardrails. It showed a specific architecture rather than principles. Most platform teams could adapt that pattern directly.

Is 'governed autonomy' just a polite way of saying AI with human approvals?

Largely yes, but the framing matters. Governed autonomy is a design principle: autonomy is bounded by explicit decision tiers, encoded in the system rather than enforced by humans remembering to follow process. RBI's sharded Kargo model and the AI governance tier definitions are examples — the constraints are architectural, not procedural. That's what makes it durable under pressure.

How does the edge/telecom story connect to standard platform engineering work?

The CERN glider and telecom keynotes showed that cloud native patterns — GitOps, operator-based reconciliation, declarative configuration — are operating in environments with no reliable network, hostile power conditions, and real-time constraints. The same Kubernetes operator pattern that manages your ArgoCD Applications manages satellite ground control. The connection is the operational contract, not the specific workload.

Which KubeCon theme is most likely to still be relevant in 18 months?

Platform teams as product teams. It's the oldest theme in this list and the one most teams are still getting wrong. Every other theme — AI in operations, governed autonomy, self-service APIs — depends on platform teams having a product mindset: measuring adoption, treating bypasses as feedback, and defining done as "relied on" rather than "shipped". Get that right and the rest follows.

  • 7 min read

AI-driven GitOps with MCP and Argo CD

Banner image Banner image

AI-driven GitOps with MCP and Argo CD

This was one of those talks where the demo felt uncomfortably close to real life. You could see the practical value straight away: fewer support bottlenecks, faster incident triage, and less context-switching for developers.

Quick takeaways

  • Put AI where engineers already work (Slack/Teams), not in a new interface.
  • MCP makes Argo CD actions discoverable and reusable for different AI clients.
  • Start with operational tasks where feedback is fast: deploy, check health, roll back.
  • Capture troubleshooting logic once and reuse it through Agent Skills.

The Slack bot beat the UI extension — by a lot

Intuit built the AI troubleshooting feature twice: once as an Argo CD UI extension (poor adoption) and once as a Slack bot in existing support channels (dramatically higher engagement). The lesson isn't that UI extensions are bad. It's that adoption follows workflow proximity. If your engineers debug in Slack, that's where the AI assistance belongs.

What was getting in the way

At KubeCon EU 2026, Alexander Matyushentsev (Argo CD co-founder) and Leonardo Luz Almeida (Intuit) laid out the problem clearly: managing 350+ Kubernetes clusters with 3,000+ production services across 50,000+ namespaces creates massive support bottlenecks. Developers wait in Slack channels for troubleshooting help. Expert DevOps engineers drown in repetitive diagnostic work. Traditional UI extensions fail because they don't meet users where they already are.

What we actually wanted

An AI-powered GitOps experience where developers use natural language to deploy applications, troubleshoot production issues, and automatically roll back failures - all through the tools they already use daily (Slack, Claude, Copilot).

Architecture in one view

MCP + Argo CD container view MCP + Argo CD container view

Model Context Protocol (MCP): the universal connector

MCP is a standardised bridge between AI clients and services like Argo CD. Instead of building custom integrations for every LLM, MCP provides:

  • JSON-RPC protocol over stdio or HTTP (Server-Sent Events or polling)
  • Discovery API exposing tools (functions), resources (read-only data), and prompts (guidance text)
  • Token passthrough for authentication - MCP delegates auth to underlying services
  • One-to-one tool mapping with Argo CD CLI/UI capabilities (sync, inspect, logs, manifests)

Open Source MCP for Argo CD

Available at github.com/argoproj-labs/argocd-mcp with growing community adoption (#mcp-for-argocd on CNCF Slack with 16 members at conference time).

Three use cases that worked in practice

1. Natural Language Application Creation

Before: Fill out complex forms, specify manifest repos, branches, namespaces, sync policies
After: "Create an app called 'frontend' using the manifests from github.com/myorg/apps, main branch, deploy to production namespace"

The AI agent discovers available MCP tools, constructs the Argo CD Application resource, and deploys - faster than manual creation.

2. Batch Deployment from Git Directories

Prompt: "Connect to my GitOps repo and create an Argo CD application for each directory under /apps"

The agent: - Scans the Git repository structure - Identifies manifest directories - Creates multiple applications automatically - Implements app-of-apps patterns without manual intervention

This is genuinely faster than human assembly for repetitive structures.

3. Automated Deployment with Intelligent Rollback

Prompt: "Deploy version 2.0 of my service, monitor its health, and roll back to the previous version if it degrades"

The agent: 1. Syncs the new version 2. Continuously checks application health via Argo CD API 3. Analyzes degradation patterns from status conditions 4. Automatically reverts manifests on failure detection

This reduces Mean Time To Recovery (MTTR) from hours to minutes.

Intuit's production journey

Failed Experiment: Argo CD UI Extension

Intuit initially built an AI-powered troubleshooting extension directly in the Argo CD UI: - Extracted logs, Kubernetes events, live state, desired state via Argo CD API - Provided LLM-powered root cause analysis - Result: Poor adoption - experts went straight to logs, novices never opened Argo CD

Key Lesson: Don't build new interfaces; integrate where users already are.

Breakthrough: Slack Bot Integration

Moving AI troubleshooting into existing Slack support channels achieved dramatically higher engagement:

Example 1: Stack Trace Analysis
Developer: "I'm seeing warnings in argo logs, not sure if critical"
Bot: "Not critical. The class is attempting to cast byte array during schema validation. Check implementation at line 76, ensure serialization order is: byte array → deserialized logic → schema validation."

Example 2: Multi-Failure Triage
Developer: "Production deployment issue"
Bot: "I see TWO failures: (1) Recent deployment can't reach config server, (2) Current running version is degraded. Which should I investigate?"
Developer: "Current failure"
Bot (2.5 min later): "Application cannot retrieve configuration from Spring config server due to connection issue. Root cause: App configured with e2e environment URL while this is production. Update URL from config-e2e.company.com to config-prod.company.com"

Reverse Proxy Pattern

Single MCP server acts as facade for 40+ Argo CD instances, simplifying: - Developer access management - Service-to-service communication - Security boundaries and token distribution

Agent Skills: The Next Evolution

To avoid duplicating troubleshooting logic across UI extensions, bots, and CLI tools, Intuit is experimenting with Agent Skills - reusable markdown-based diagnostic recipes.

Skills define: - How to extract Argo CD base URLs and application names - Step-by-step troubleshooting procedures for degraded apps - Which API calls to make and how to interpret responses

This creates a library of operational knowledge that any agent can consume.

Apply this: document one troubleshooting flow as an Agent Skill before the next incident

Pick the most common Argo CD failure your team handles (image pull errors, resource limit violations, sync hook timeouts) and write it up as a SKILL.md. The next time that failure pattern appears, the agent has the diagnostic steps rather than someone having to remember them at 2am. One skill, written once, reused indefinitely.

Implementation Guide

For platform teams

  1. Deploy MCP for Argo CD from argo-cd-labs
  2. Configure HTTP transport for remote access (OAuth 2.0 or token passthrough)
  3. Integrate with existing support channels (Slack, Teams) rather than building standalone UIs
  4. Create prompts that instruct AI on common failure patterns:
  5. Image pull errors from Kubernetes events
  6. Resource limit violations from quota checks
  7. Config server misconfigurations from environment mismatches

For developers

  1. Test free-form application creation with natural language
  2. Experiment with batch operations via Git directory scanning
  3. Implement automated health monitoring with roll back conditions
  4. Measure engagement: track before/after metrics for bot interactions vs UI usage

For organisations

  1. Document troubleshooting workflows as Agent Skills in markdown format
  2. Share diagnostic logic across tools to eliminate duplication
  3. Consider MCP for other GitOps ecosystem tools (Flux, Argo Rollouts, External Secrets Operator)

The 'new interface' adoption trap

Intuit's first attempt failed because it built yet another UI tab — this time inside Argo CD. Experts bypassed it (they went straight to logs) and novices never opened Argo CD in the first place. Before building an AI interface, ask: where do engineers currently go when something breaks? Build there, not where you wish they'd go.

What changed in practice

Before: Weeks-long support backlogs, expert knowledge bottlenecks, manual deployments
After: Conversational GitOps operations, automated diagnostics, self-service at scale

The convergence of GitOps and AI through standardised protocols like MCP changes how platform teams operate. By meeting users where they are and encoding operational knowledge in reusable formats, teams move from "Platform as Code" to "Platform as Conversation".

References

Presented at KubeCon + CloudNativeCon Europe 2026 by Alexander Matyushentsev (Akuity) and Leonardo Luz Almeida (Intuit)

Frequently asked questions

Is the MCP for Argo CD project production-ready?

At KubeCon EU 2026, github.com/argoproj-labs/argocd-mcp had 16 members in the CNCF Slack channel — community adoption was early-stage. Intuit's production Slack bot predated the open-source MCP server and used custom integrations. Treat the open-source MCP server as a strong starting point for experimentation; validate it against your specific Argo CD version and scale before putting it on a critical support path.

Where can I watch the KubeCon session?

The talk by Alexander Matyushentsev (Akuity) and Leonardo Luz Almeida (Intuit) is on the CNCF YouTube channel. Search "AI-driven GitOps MCP Argo CD KubeCon EU 2026".

Does the reverse proxy pattern work with Argo CD OIDC authentication?

Yes — the MCP server acts as a facade and handles token distribution to the underlying Argo CD instances. OIDC tokens are passed through to each Argo CD instance, which enforces its own RBAC. The key benefit is that developers interact with one MCP endpoint rather than managing credentials for 40+ Argo CD instances separately.

How do Agent Skills relate to SKILL.md files?

They're the same concept. Intuit calls them Agent Skills; the open standard uses SKILL.md. A skill packages a diagnostic or operational procedure as a reusable markdown file any compliant agent can follow. The Kyverno session at KubeCon used the same pattern for policy operations. It's the same idea independently converging on the same solution — which is a strong signal the pattern is right.

Can this pattern be applied to other GitOps tools like Flux?

Yes — the MCP protocol is tool-agnostic. The ArgoCD MCP server exposes Argo CD-specific tools, but the same architecture applies to Flux: an MCP server that wraps Flux CLI operations and the Flux API would give AI agents the same natural language access to Flux-managed resources. The MCP session at KubeCon explicitly noted this as a direction for the broader GitOps ecosystem.

  • 5 min read

Setting Up OpenClaw: Skills, Tailscale, GitHub Config Sync, and Copilot

Banner image Banner image

Setting Up OpenClaw: Skills, Tailscale, GitHub Config Sync, and Copilot

A few months ago we onboarded a new environment “the fast way.” It worked… until it didn’t. No clear roll back, no consistent skills, and no shared way to answer the basic question: what’s running, where, and why?

This guide fixes that. It’s a clean Day‑1 setup that gets OpenClaw running securely, reproducibly, and in a way you can scale.

Principles (the why)

  • Secure access by default (no exposed ports)
  • Reproducible configuration (GitHub‑backed)
  • Productive workflows (skills + Copilot)

Architecture at a glance

OpenClaw system context OpenClaw container view

1) Install OpenClaw (quick path)

Use the official docs for your platform. If you want a guided setup with scripts, use the companion repo:

https://github.com/polarpoint-io/ai-capabilities

Ubuntu (reference setup)

sudo apt-get update
sudo apt-get install -y git curl ca-certificates

Install OpenClaw via the official docs, then verify:

openclaw status
openclaw gateway status
openclaw gateway start

Tailscale is the right default for remote access

Exposing OpenClaw's gateway port publicly is the fastest path to a compromised assistant. Tailscale gives you private, encrypted access without opening any firewall rules — the node is only reachable from devices on your tailnet. It takes about 5 minutes to set up and eliminates the most obvious attack surface for a process that has access to your files, SSH hosts, and messaging channels.

2) Bring up Tailscale (secure access)

Tailscale gives you private, encrypted access without opening ports publicly.

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up

Tip: tag the host and lock it down with ACLs.

3) Use the setup scripts (fast, repeatable)

The ai‑capabilities repo includes setup scripts for both Linux and macOS:

# Linux
./scripts/setup-linux.sh

# macOS
./scripts/setup-macos.sh

These scripts: - ensure Tailscale is installed and online - start the OpenClaw gateway

Snippet (Linux):

openclaw gateway start
openclaw gateway status

4) Enable your first skills

Installed skills right now: - self-improving-agent - captures learnings from failures and feedback - ddg-web-search - web search without an API key - github - GitHub ops via gh - kubernetes - cluster operations and manifests

List skills:

clawhub list

Install a new skill:

clawhub install <skill-name>

5) Sync config with GitHub

Store non‑secret config in GitHub for reproducibility.

Suggested layout:

openclaw-config/
  README.md
  openclaw.json (sanitized)
  skills/
  notes/

Never commit secrets. Use env vars or a secret manager instead.

Apply this: never commit secrets to your openclaw-config repo

The GitHub config sync layout in this post keeps non-secret configuration version-controlled and reproducible. The critical constraint: API tokens, SSH keys, and webhook secrets go in environment variables or a secret manager — never in the repo. The openclaw.json file in the repo should be a sanitised version with all credential fields replaced by $ENV_VAR references. Document this explicitly in the README so the next person doesn't wonder why the config file doesn't work as-is.

6) Pair with GitHub Copilot

Copilot helps with: - docs/blog drafts - refactors and cleanup - repetitive config work

Workflow: 1) OpenClaw automates 2) Copilot speeds edits 3) You review + ship

Skills missing after reinstall is the most common setup pain

When you reinstall OpenClaw or set up on a new machine, clawhub list will show zero skills even though you have them configured in your GitHub repo. Skills are installed to the local instance, not synced automatically from your config repo. Add a clawhub install step to your setup scripts and document which skills you depend on in openclaw-config/README.md. Otherwise you'll discover missing skills mid-task.

7) Validate the setup

Run these checks:

openclaw status
tailscale status
clawhub list

Quick checklist

  • ✅ OpenClaw running
  • ✅ Tailscale connected
  • ✅ Skills installed
  • ✅ Config synced (sanitized)
  • ✅ Copilot ready

Common pitfalls (and fixes)

  • Gateway not running → openclaw gateway start
  • Tailscale offline → tailscale up
  • Skills missing → clawhub list, reinstall
  • Secrets in Git → move to env vars/secret manager
  • No roll back → add a roll back note to openclaw-config/README.md

Next steps

Explore runnable examples and scripts in the companion repo: https://github.com/polarpoint-io/ai-capabilities

Frequently asked questions

What platforms does OpenClaw support?

OpenClaw runs on Linux (Ubuntu, Debian, and derivatives) and macOS. Windows is not officially supported — the setup scripts in the companion repo target Unix environments. If you need to run on Windows, a WSL2 environment works with the Linux scripts. The gateway process itself is a Node.js daemon that is theoretically cross-platform, but the skill ecosystem assumes Unix tooling.

How do I back up my OpenClaw configuration and memory?

Keep your workspace directory (~/.openclaw/workspace/ by default) in a private Git repo. Commit AGENTS.md, SOUL.md, USER.md, TOOLS.md, HEARTBEAT.md, and the memory/ directory. Exclude any files containing secrets. A daily cron job that commits and pushes gives you automatic version history of your agent's memory and personality evolution.

What's the difference between a skill and a built-in OpenClaw capability?

Built-in capabilities (messaging, file access, basic shell commands) are part of the OpenClaw gateway itself — they're always available. Skills are add-on packages from ClaWHub that provide higher-level domain capabilities: web search, GitHub operations, Kubernetes management. Skills follow the SKILL.md standard and can be installed, uninstalled, and updated independently via clawhub install and clawhub update.

Can OpenClaw access my local files and run shell commands?

Yes — with appropriate permissions. The gateway process runs with your user account's permissions and can read files, run commands, and access your SSH hosts (via the Kubernetes and GitHub skills). This is the power that makes it genuinely useful and also the reason Tailscale access control and careful skill selection matter. Grant skills only the access they need for the tasks you actually want to delegate.

How does OpenClaw authenticate to external services like GitHub?

Skill-level authentication varies by skill. The GitHub skill uses the gh CLI, which authenticates via gh auth login. The Kubernetes skill uses your local kubeconfig. OpenClaw itself doesn't store credentials — it delegates to the tools your account already has configured. This means setup is usually "do you have the CLI authenticated?" rather than "configure a new credential".

  • 4 min read

Secrets & GitOps: ArgoCD + External Secrets Done Right

Banner image Banner image

Secrets & GitOps: ArgoCD + External Secrets Done Right

What was getting in the way

GitOps worked — until secrets showed up. Teams either committed secrets or blocked releases waiting for manual secret creation. Neither option is sustainable once you have more than two clusters or three services rotating credentials.

What we actually wanted

A GitOps flow that keeps secrets outside of Git and still automates deployments end to end.

The GitOps secret pattern in one sentence

Git holds ExternalSecret manifests (just references, no values). ESO fetches values from your secret manager. ArgoCD syncs the manifests. Nothing sensitive ever touches your Git history.

How we approached it

External secrets context External secrets context Guardrails flow Guardrails flow

Apply this: migrate the most sensitive secrets first

Don't try to migrate everything at once. Start with database credentials, API keys, and service account tokens — the things where a leak would be most damaging. Once those are working with ESO, the rest follows the same pattern. You'll also learn the rough edges of your specific secret manager integration on lower-stakes items first.

The secure pattern

  • Git holds ExternalSecret manifests only
  • ESO pulls from Vault/SSM/Secrets Manager
  • ArgoCD syncs manifests, ESO resolves secrets

The ExternalSecret is what lives in Git. It has no sensitive values — just a reference to the secret path in your secret manager and a description of which keys to extract. Anyone with repo access can see which secrets are in use without being able to read the values. That is your audit story.

Files worth opening

  • repo/gitops/secrets/secretstore.yaml
  • repo/gitops/secrets/externalsecret.yaml

The refresh interval gotcha

ESO's default refreshInterval is 1h. If you rotate a secret in your secret manager expecting it to propagate immediately, it won't — the running pod keeps the old value until the next refresh cycle. For secrets that rotate frequently or in response to incidents, set refreshInterval: 15m or shorter. For long-lived credentials, 1h is fine.

What changed in practice

Teams can ship GitOps changes without ever touching sensitive data. Security teams gain control without blocking delivery. ArgoCD diffs stay clean because the ExternalSecret manifest — not the generated Kubernetes Secret — is what ArgoCD compares. Secret rotation is completely invisible to ArgoCD; it happens in the secret manager and ESO picks it up on the next refresh.

Frequently asked questions

What secret managers does ESO support?

ESO supports AWS Secrets Manager, AWS SSM Parameter Store, Google Secret Manager, Azure Key Vault, HashiCorp Vault, 1Password, Kubernetes secrets, and a growing list of others. The SecretStore provider field controls which backend you use — switching backends requires only a SecretStore change, not a rewrite of your ExternalSecret manifests.

Does ArgoCD treat ExternalSecret manifests as out-of-sync when secrets rotate?

No — this is one of the key benefits. ArgoCD compares the ExternalSecret manifest in Git against the ExternalSecret resource in the cluster. If neither has changed, ArgoCD reports the app as synced even when ESO quietly updates the underlying Kubernetes Secret with a rotated value. No spurious diff noise.

Can I use ESO in a multi-cluster setup?

Yes. Use ClusterSecretStore (cluster-scoped) for stores accessed by multiple namespaces, or SecretStore (namespace-scoped) for tighter isolation. In a multi-cluster ArgoCD setup, deploy ESO to each cluster and point each cluster's ClusterSecretStore at the appropriate path in your secret manager for that environment.

How do I handle secrets that need to exist before ESO is installed?

Bootstrap secrets — the credentials ESO itself uses to authenticate to your secret manager — have a chicken-and-egg problem. Most teams handle this with a SealedSecret or direct kubectl create secret for the bootstrap credential only, then manage everything else via ESO. Document this exception explicitly so the next person understands why one secret bypasses the ESO pattern.

What happens if the secret manager is unreachable?

ESO will surface an error condition on the ExternalSecret resource and the Kubernetes Secret won't be updated. Pods already running continue using their existing secret values (mounted at pod start). New pods that haven't yet received the secret will fail to start. Monitor ExternalSecret status conditions and alert on SecretSyncError to catch connectivity issues before they become incidents.

  • 7 min read

Multi‑Cluster GitOps with ArgoCD: The Operational Blueprint

Banner image Banner image

Multi‑Cluster GitOps with ArgoCD: The Operational Blueprint

One cluster is manageable. Two is fine. Five is where things start getting interesting. Somewhere around ten, you're not managing Kubernetes anymore — you're managing the management of Kubernetes, and that's a different job entirely.

The pattern most teams fall into is depressingly predictable. You start with one ArgoCD instance, one cluster, everything works great. Then a new environment lands — staging needs to be production-like, a regional cluster gets spun up, a client needs isolation. Each new cluster gets a copy of the configuration from the last one, tweaked by hand. Promotions happen by copying YAML between directories and hoping nothing was missed. Six months in, you've got seven clusters and no real confidence they're running the same thing.

Config drift isn't a discipline problem. It's an architectural one. You need a model that makes consistency the default, not something you have to manually enforce.