Skip to content

Argo CD 3.4: Your Teams Notifications Just Broke

Banner image Banner image

You know that moment when you set something up, it works perfectly, and you stop thinking about it? That's Teams notifications in Argo CD. You wired up the webhook, tested it, watched an alert land in the channel, and moved on to the next thing. Job done. Except it's not done anymore. As of 31 March 2026, Microsoft retired Office 365 Connectors — the underlying mechanism that Argo CD's Teams notifier was built on. No fanfare, no loud breakage. Your notification config still looks correct. The webhook URL is still there. Argo CD still thinks it's sending alerts. They're just not arriving.

Think about what that actually means in practice. Your sync failures are still happening. Your health degradations are still triggering. The alerts are being dispatched — they're just disappearing into the void before they ever reach your Teams channel. If your team's incident detection relies on those notifications, you've been operating with a silent gap since the end of March. Nobody panicked, because nothing obviously broke. That's the worst kind of failure.

Argo CD 3.4 ships a proper fix: a new Teams Workflows notification service that replaces the retired connector. It went GA on 4 May 2026. The migration takes about ten minutes. Let's walk through what happened, how to fix it, and what else is worth your attention in this release.


Quick takeaways

  • If you use Teams notifications and you're not on Argo CD 3.4, your alerts are silently broken right now. Microsoft killed Office 365 Connectors on 31 March 2026.
  • Argo CD 3.4 ships a new teams-workflows notifier that replaces the dead connector — but you'll need to update your notification config, not just upgrade Argo CD.
  • A significant ApplicationSet bug is fixed — cluster Secret changes were triggering reconciliation across all namespaces, bypassing allowedNamespaces. Multi-tenant setups were affected.
  • 3.4 is GA as of 4 May 2026 — upgrade through your normal process.

Silent breakage since 31 March 2026

When Microsoft retired Office 365 Connectors, Argo CD didn't start logging errors — it continued dispatching alerts that silently disappeared. If your team relies on Teams notifications for sync failures or health degradations, you've had a blind spot since the end of March. Check your notification history before assuming things are fine.


The Teams connector retirement — what actually happened

Office 365 Connectors were Microsoft's original webhook mechanism for posting messages into Teams channels. They were simple: you'd generate a webhook URL, POST a JSON payload to it, and a card would appear in the channel. Argo CD's Teams notifier was built on exactly this model.

Microsoft deprecated Office 365 Connectors back in 2024, giving teams until 31 March 2026 to migrate. They've replaced them with Power Automate-based Workflows — a fundamentally different model built on Adaptive Cards and proper flow orchestration. It's more capable, honestly, but it's also a different integration pattern that the old Argo CD notifier simply doesn't support.

Here's why this broke silently rather than loudly: when Microsoft retired the connectors, they didn't immediately start returning error codes that would surface in your logs. The requests just stop being delivered. Argo CD's notification controller dispatches the alert, gets what looks like an acceptable response, and marks it as sent. No errors in your notification controller logs, no obvious signal that anything's wrong. You'd only notice if you were actively watching for alerts that never came — which, honestly, most teams aren't doing on a quiet day.

If you haven't upgraded yet, your team is flying blind right now. Sync failures, OOM kills, degraded health checks — any of those that happened after 31 March didn't send a Teams alert. Worth checking your history.

Apply this: verify notification delivery, not just dispatch

Most teams only check that Argo CD dispatched a notification — they don't verify it was received. Add a monthly canary: deliberately trigger a known alert type and confirm it lands in the channel. Silent delivery failures like this one are nearly undetectable without an active check.


Migrating to the Teams Workflows notification service

The good news is that Argo CD 3.4 has done the work for you on the notifier side. There's a new teams-workflows service type that's built against the Workflows webhook model. The migration involves two things: generating a new webhook URL from your Teams channel (using the Workflows connector, not the old Office 365 one), and updating your argocd-notifications-cm config.

Here's what a realistic config looks like:

# argocd-notifications-cm
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
data:
  service.teams-workflows: |
    webhookUrl: $teams-workflows-webhook-url
  template.app-sync-failed: |
    teams-workflows:
      themeColor: "FF0000"
      title: "{{.app.metadata.name}} sync failed"
      text: "Application {{.app.metadata.name}} failed to sync to {{.app.spec.destination.server}}.\n\n**Error:** {{.app.status.operationState.message}}"
      facts:
        - name: Repository
          value: "{{.app.spec.source.repoURL}}"
        - name: Revision
          value: "{{.app.spec.source.targetRevision}}"
  trigger.on-sync-failed: |
    - when: app.status.operationState.phase in ['Error', 'Failed']
      send: [app-sync-failed]

A few things to note here. The $teams-workflows-webhook-url should be stored as a Secret and referenced via the standard Argo CD notifications secret mechanism — don't hardcode the URL in the ConfigMap. The template structure is similar to what you had before, so if you've got customised templates, they should port over with minimal changes. And you'll want to audit all your subscriptions: any application or AppProject that had a Teams subscription using the old service name will need updating to point to teams-workflows.

Audit subscriptions before you upgrade

Make a list of every Application and AppProject with a Teams subscription before you touch the config. After migration, any subscription still pointing to the old service name will silently do nothing. A 10-minute audit now saves debugging time later.


The ApplicationSet reconciliation fix

This one's quietly significant, particularly if you're running a multi-tenant setup. The bug: when a cluster Secret was updated — credentials rotated, labels changed, whatever — Argo CD's ApplicationSet controller was triggering reconciliation across all namespaces, regardless of the allowedNamespaces setting on your ApplicationSets.

In concrete terms: if you have ApplicationSets scoped to specific namespaces for tenant isolation, this was silently reconciling things it shouldn't have been touching. The allowedNamespaces field is supposed to act as a boundary — only reconcile ApplicationSets in these namespaces. A cluster Secret update was blowing right past that boundary and reconciling everything.

This is fixed in 3.4. But it also means if you've been relying on allowedNamespaces for isolation in a multi-tenant cluster, you've had a gap. After upgrading, it's worth reviewing your ApplicationSet configs and verifying that your namespace scoping is doing what you expect.

Multi-tenant isolation was silently broken

If you rely on allowedNamespaces for tenant isolation in ApplicationSets, cluster Secret updates were bypassing that boundary and reconciling out-of-scope namespaces. This is fixed in 3.4, but review your ApplicationSet configs after upgrading to confirm isolation is working as expected.


Application filtering improvements

Less urgent than the above, but genuinely useful for day-to-day operations. You can now filter the application list by metadata.annotations, not just labels. There's also a new Operation Status filter.

The annotations filter matters more than it might sound. Plenty of teams use annotations for things that don't belong in labels — team ownership, cost centre codes, environment tags, compliance flags. Labels are for selectors; annotations are for everything else. Previously, if you wanted to find all applications owned by a particular team and that ownership was expressed as an annotation, you were doing it by eyeball or with kubectl. Now you can filter directly in the UI.

The Operation Status filter is a straightforward quality-of-life improvement. When you've got dozens or hundreds of applications, being able to filter to "show me everything currently syncing" or "show me everything in an errored operation state" is the kind of thing that sounds small until you've had to hunt for it manually during an incident.


Should you upgrade now?

Yes. 3.4 is GA and the Teams notification fix and ApplicationSet reconciliation fix both have real operational impact.

The Teams notification issue is a now problem — if you're using Teams alerts and you're not on 3.4, those alerts aren't working. The ApplicationSet reconciliation bug is a security and isolation concern: if you're multi-tenant and relying on allowedNamespaces for namespace scoping, it was silently broken. Both of those are good reasons to upgrade through your normal process.

Here's a short checklist:

  • Audit your current notification subscriptions before migrating. Make a list of every application and AppProject with a Teams subscription — you'll need to update all of them.
  • Generate a new Workflows webhook URL from Teams before you update the config. The new service needs a Workflows-type webhook, not the old Office 365 connector URL.
  • Check your ApplicationSets' allowedNamespaces configs after upgrade to verify namespace scoping is behaving correctly.

Frequently asked questions

Will my existing Teams alerts just start working again after I upgrade?

Not automatically, no. Upgrading to 3.4 adds the new teams-workflows notification service, but you still need to update your argocd-notifications-cm to use it, generate a new Workflows webhook URL from Teams, and update any subscriptions that referenced the old Teams service. The upgrade gives you the capability; the config migration makes it work.

We use Slack and email, not Teams — do we need to upgrade urgently?

The Teams retirement doesn't affect you. The ApplicationSet reconciliation fix might, if you're running multi-tenant and relying on allowedNamespaces. If you're single-tenant and not using Teams notifications, upgrade on your normal cadence — nothing here is on fire for you.

What's the difference between the old connector and the new Workflows notifier?

The old Office 365 Connector was a simple POST-to-webhook model. The new Teams Workflows mechanism is built on Power Automate and uses Adaptive Cards — it's more capable and more composable, but it's a different integration pattern and needs a different webhook URL. Functionally for most notification use cases, you won't notice much difference in the output. The main change is on the configuration side.

How do I know if the ApplicationSet reconciliation bug affected my cluster?

If you're running multi-tenant Argo CD with allowedNamespaces on your ApplicationSets, and you've rotated cluster credentials or updated cluster Secret labels since your last Argo CD upgrade, you've likely been affected. The practical impact is unexpected reconciliation events in out-of-scope namespaces. After upgrading to 3.4, verify that your ApplicationSet reconciliation is scoped correctly by watching controller logs when a cluster Secret is next updated.


What you get

  • A working Teams notification integration that'll actually deliver alerts, built on the new Workflows model
  • Fixed ApplicationSet reconciliation — cluster Secret changes no longer bypass allowedNamespaces
  • Annotation-based filtering in the application list, alongside the new Operation Status filter
  • Continued Source Hydrator improvements as the feature matures

Further reading