66% of GenAI Runs on Kubernetes. Only 7% Can Ship Daily.¶

You know that moment in a platform review when someone puts up the slide showing your AI infrastructure? GPU nodes provisioned. Inference operator installed. Models serving traffic. Everyone nods. The platform is ready.

Then ask a different question: how long does it take to get a new model version into production? The room goes quiet. Someone mentions a Slack thread with the ML team. Someone else mentions a Jupyter notebook that does the deploy. It's Tuesday, and the model that shipped last week is still the one from March.

The 2025 CNCF Annual Cloud Native Survey put numbers on this. 66% of organisations hosting generative AI models run inference on Kubernetes. 82% run Kubernetes in production. And how many deploy models daily? 7%. Another 47% deploy "occasionally" — which is survey-speak for "when someone has a spare afternoon and nothing's on fire".

The gap that should bother you

Two-thirds of GenAI inference already runs on Kubernetes, but only 7% of organisations deploy models daily. That's not an infrastructure gap. Everyone bought the platform. It's a release engineering gap — and platform teams already know how to close those.

We've seen this exact shape before. It's what app delivery looked like before DORA metrics and deployment pipelines: infrastructure in place, release process artisanal. And the fix is going to look familiar too.

C4 Architecture Diagram

The Gap Isn't Where You Think It Is¶

When model deployment is slow, the reflex is to blame the hard parts of AI infrastructure: GPU scarcity, scheduling complexity, model size. And those are real problems. But they're not why only 7% ship daily.

Here's the tell. The same survey says 47% of orgs deploy models occasionally. So the machinery works — models do get to production. What's missing is repeatability. Each deploy is a small project instead of a routine event. Someone coordinates it, someone watches it, someone knows the one weird step that isn't written down.

If you've been doing platform work for more than a few years, you've lived this movie. In 2015 the org had Jenkins, artifact repos, and config management, and still shipped quarterly, because the release process ran on tribal knowledge and fear. What changed wasn't the infrastructure. It was treating the path to production as a product: standard artifact, standard pipeline, standard rollback, and enough automated verification that a deploy stopped being a decision and became a non-event.

Model deploys in 2026 are exactly where app deploys were then. The cluster is ready. The paved road doesn't exist.

Where the numbers come from

All figures are from the 2025 CNCF Annual Cloud Native Survey, published January 2026. The survey also found Kubernetes production use at an all-time high of 82% — the "operating system for AI" framing is CNCF's own.

Why Your Existing Paved Road Rejects Models¶

The obvious move is to put models on the same golden path as applications. Teams try it, and it breaks in four specific places.

The artifact is wrong. Your pipeline moves 40MB container images. A model deploy moves a 40GB weights file, and "build, push, pull" semantics fall apart when a node pull takes twenty minutes and your rollout strategy assumes seconds. You need model-aware caching and warm pools, or every deploy eats your error budget on startup alone.

Rollout isn't a container swap. Replacing an inference pod isn't like replacing a stateless API pod. There's GPU capacity to reserve before the new version can start, KV caches to warm, and often a period where old and new versions need to run side by side on hardware you don't have spare. A naive rolling update either fails scheduling or doubles your GPU bill mid-deploy.

Rollback has no signal. kubectl rollout undo works when failure means error rates and latency. Models fail differently — the new version answers faster and cheaper, and three days later someone notices the answers got worse. Your CD pipeline has no gate that catches quality regression, because quality isn't in your metrics stack at all.

Nobody owns the pipeline. The ML team owns the model, the platform team owns the cluster, and the deploy lives in the gap between them — usually as a notebook or a runbook with a person's name in it.

The notebook deploy tell

If getting a model to production involves anyone running a notebook, you don't have a deployment process — you have a person. That person goes on holiday. This is the single most common reason "occasionally" never becomes "weekly".

GitOps for Models: The Pipeline That Closes It¶

The fix isn't a new platform. It's extending the GitOps loop you already run so that a model version is just another declarative change.

The shape looks like this. The model registry holds versioned, evaluated model artifacts — that's the ML team's interface, and it's the only place they need to touch. A Git commit pins a registry reference into an environment repo. ArgoCD syncs it into an inference resource: KServe's InferenceService if you want the batteries-included path, or a vLLM deployment via its operator if you're running open weights and want the knobs.

# environment repo — the model version is a Git-pinned declarative fact
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: support-summariser
spec:
  predictor:
    model:
      modelFormat:
        name: huggingface
      storageUri: "oci://registry.internal/models/support-summariser:v47"
      resources:
        limits:
          nvidia.com/gpu: "1"

The piece most teams miss is the analysis gate. Progressive delivery for models means the rollout controller shifts traffic gradually and evaluates quality (not just error rate) before promoting. Argo Rollouts lets you plug an eval metric in as an analysis template:

# analysis gate — promotion requires the eval score, not just HTTP 200s
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: eval-gate
spec:
  metrics:
    - name: eval-score
      interval: 5m
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: avg(model_eval_score{service="support-summariser-canary"})
      successCondition: result[0] >= 0.87

That model_eval_score metric is doing the heavy lifting: a shadow eval job replays a golden dataset against the canary and publishes the score. Now rollback is automatic and has a reason attached. The quality regression that used to take three days and an annoyed user to surface gets caught at 10% traffic.

Apply this in your platform

Start with the eval gate, not the full pipeline. Even if deploys stay manual for now, wiring a golden-dataset eval into a Prometheus metric gives every future automation step something to gate on. It's the piece with no prerequisite and the piece everything else depends on.

The Ladder From Occasional to Daily¶

You don't jump from "occasionally" to daily deploys. There's a ladder, and each rung removes one manual decision.

Getting to monthly means killing the notebook: any deploy happens by changing a registry reference in Git, even if a human still watches it end to end. Getting to weekly means trusting the machinery: ArgoCD owns the sync, the eval gate owns promotion, and a human's only job is reviewing the PR that bumps the version. Getting to daily means the PR itself is often automated — the registry publishes a new evaluated version, a bot opens the bump PR, and merge is the only human act left. The same shape as drift detection PRs, pointed at model versions instead of config.

Notice what's not on the ladder: a new platform, a new team, or an MLOps suite. Every rung is made of parts you already operate — Git, ArgoCD, Prometheus, a rollout controller. The GPU scheduling layer underneath (DRA and device taints if you're on recent Kubernetes) matters, but it's an enabler, not the bottleneck.

And if you're building the platform API layer above this, the model-as-declarative-resource pattern is exactly what a Crossplane-based inference control plane formalises: one claim, model version as a field, everything else composed behind it.

The one-metric version

Track "model deploys per week" the way you track DORA deployment frequency. It's the single number that tells you whether your AI platform is a platform or a very expensive science project.

FAQ¶

We only retrain quarterly — why would we need daily deploys?

Deployment frequency isn't about retraining cadence. Version bumps, prompt template changes, quantisation swaps, serving config tuning: these all ship through the same pipeline, and they happen weekly even when training doesn't. The 7% aren't retraining daily; they've made shipping anything model-related routine.

Does this require KServe, or does raw vLLM work?

The pattern is agnostic. KServe gives you the InferenceService abstraction, canary support, and model-format handling out of the box. A vLLM deployment managed by its operator works too — you'll wire more of the rollout mechanics yourself. Choose based on how much of the serving layer you want to own, not on the GitOps loop, which is identical either way.

What goes in the golden dataset for the eval gate?

Real production traffic, sampled and labelled, not synthetic benchmarks. A few hundred representative prompts with scored reference outputs beats ten thousand generic ones. Version the dataset in Git next to the environment config, because "which eval did this model pass" is a question you'll eventually be asked in an incident review.

Isn't this just MLOps rebranded?

The MLOps world built this for the training side — experiment tracking, feature stores, registries. What's been missing is the last mile: deployment as a platform concern, run by the same GitOps machinery as everything else. Call it what you like; the point is your inference deploys shouldn't need a different operating model than your app deploys.

The 66/7 gap is going to close over the next couple of years, if the survey trend lines are anything to go by. The question is whether your org closes it deliberately, with a paved road your platform team designed, or accidentally, with whatever pipeline grows out of the notebook that currently does the job.

You've closed this gap once before. Same playbook. Bigger artifacts.

For building this exact golden path spec-first, see BMAD in Practice. For the golden-path thinking behind it, see Platform as Product. For the GPU scheduling layer underneath, see Kubernetes DRA and Device Taints. And for the drift-PR pattern applied to config instead of models, see GitOps AI Drift Detection.