){kind=small}
)
Anyone running a service mesh knows the stress of Istio upgrades. At Aircall, where every conversation matters and reliability is non-negotiable, we've turned that high-risk operation into a routine Git change, delivering zero-downtime upgrades across 25+ clusters and 300+ mesh workloads.
Where Istio sits at Aircall
Aircall runs its services on Amazon EKS. Clusters are split by product domain across several AWS accounts, with separate sandbox, staging and production environments. That's 25+ mesh clusters today.
Git is the only interface to that fleet. One central Argo CD reads a single GitOps repository and reconciles every cluster add-on through ApplicationSets. Istio is one of those add-ons.
Inside each cluster, Istio runs in sidecar mode, a proven model our teams already knew well. The cert-manager and istio-csr, backed by a private CA, issue the mesh identities, so every hop between sidecars is mTLS. External traffic comes in through a load balancer with WAF, then an Istio ingress gateway, then the workload's Envoy sidecar.
)
What Istio does for us
Every mesh workload gets an Envoy sidecar, and istiod, the control plane, configures all of them. That gives every service the same set of capabilities without any code changes:
Zero-trust networking. Every service-to-service call is mutual TLS, with a workload identity issued by our own CA. No application handles certificates.
Safer releases. Weighted traffic splitting lets Argo Rollouts send a small share of requests to a new version, then promote it or roll it back based on metrics.
Resilience. Retries, timeouts and circuit breaking (outlier detection and connection limits) live in the mesh instead of being reimplemented in each service.
Consistent observability. Every sidecar emits the same request metrics and traces, so every service gets golden signals for free.
One front door. Istio ingress gateways route external traffic into the mesh behind the load balancer and WAF.
All of this depends on istiod. It issues the configuration every sidecar runs on and injects the sidecar into every new pod. Upgrading it means touching the heart of the platform.
Why seamless upgrades matter here
Aircall is a voice platform. Real-time call traffic runs through the mesh, so if the control plane blips, a customer hears it. In sidecar mode every upgrade reaches every pod, across 300+ workloads. And upgrades aren't optional: each Istio release is supported only for a limited window.
Istio's canary upgrades and revision tags give us the right building blocks. But the documented pattern names each control plane after its version (istiod-1-x-y), re-points tags with istioctl on the live cluster, and uninstalls the old revision.
Every cycle brings new service names, certificate SANs and Argo CD config. Across 25+ clusters, that’s a large set of changes to prepare, review and roll out every time, which is time-consuming. It’s also error-prone: one missed SAN or stale webhook can break sidecar injection for new pods.
What we aimed for
Zero user impact. Real-time voice traffic keeps flowing through every step.
Labels set once. Workloads never get relabelled for an upgrade.
No per-cycle plumbing. No new service names, certificate SANs or Argo CD config each time.
Git only. No commands against live clusters. Every change is a reviewed merge request.
Rollback as a config change. Going back is a git change that moves the tag, not a reinstall.
Identical cycles. The second upgrade looks exactly like the tenth.
The solution: two slots, one tag
The design has three pieces. Only one of them moves during an upgrade.
)
Every workload namespace carries label istio.io/rev: stable. It's set once and never changes.
Behind stable sit two permanent control-plane slots: istiod-blue and istiod-green. The slot names never change, so the service names, certificate SANs and Argo CD templates never change either. Only the Istio version inside a slot changes.
At any time, one slot carries the stable revision tag and serves the whole mesh. The other slot is the rollback slot, and it's where the next version goes.
)
A revision tag is a MutatingWebhookConfiguration. When a pod is created in a stable namespace, the istio-revision-tag-stable webhook sends the injection request to whichever slot owns the tag. That slot injects the sidecar and stamps its own revision (blue or green) on the pod. Move the tag, and every pod created afterwards is injected by the other slot.
Both slots run a single mesh, share the same CA and use Kubernetes for service discovery, so blue and green sidecars find each other and communicate over mTLS as usual.
Each upgrade puts the new version in the free slot and moves the tag to it. The roles flip every cycle.
)
The first cycle was a one-time migration. It introduced the slots, moved every namespace onto the stable tag and retired the legacy default istiod. Every cycle since has been an ordinary slot swap.
How an upgrade runs
Each cluster goes through three steps. None of them forces a pod restart or needs a maintenance window.
Deploy. Set the free slot to the new version. Argo CD installs it next to the running control plane. Nothing restarts. For a smoke test, a throwaway namespace can point at the new slot directly.
Promote. Move
stableto the new slot. The webhook now routes new pods to it. Nothing restarts.Converge. Pods pick up the new sidecar as they're replaced by deploys, autoscaling and node rotation. Old and new sidecars interoperate, so mTLS keeps working while the mesh converges. A rolling restart can speed this up, but it isn't required.
The same change moves through the environments in order: sandbox first, then staging, then production one cluster at a time, lowest risk first. Once the fleet is on the new version, the change moves up to the default layer and the per-cluster overrides are deleted.
Rollback is a config change, not a reinstall. Move stable back to the previous slot in git, let Argo CD sync, then restart the affected workloads so their pods are re-injected by the previous version. The previous slot is still installed, so there's nothing to redeploy.
Results: what upgrading a cluster looks like now
Numbers from our latest upgrade, measured on production clusters.
Control plane: about two minutes
Promoting the new version is one merge request. Argo CD does the rest:
Production cluster | Sync start → new control plane healthy | Merge → new control plane healthy |
Largest cluster (100+ workloads) | 1 m 35 s | 2 m 23 s |
Data plane: no forced restarts
No workload was restarted for the upgrade. On our largest cluster, sidecars moved to the new version through normal deploys, autoscaling and node rotation:
Time after promotion | Sidecars on the new version |
First working day | ~50% |
3 days | ~95% |
Within a week | 100% |
Error rates didn't move. Mesh 5xx responses stayed below 0.002% of requests throughout, the same as before the upgrade.
Implementation: one ApplicationSet, three layers
A single Argo CD ApplicationSet generates an Application for every mesh cluster. It merges three config layers, and the most specific one wins:
Default layer: the fleet-wide resting state, inherited by every new cluster
Environment layer: a sandbox-wide or staging-wide change
Cluster layer: a single cluster, which is how production is promoted
)
Three versions per cluster
Each layer can set three versions (simplified):
A slot's Helm release is only rendered when its version is set. istio-base has its own version, so the CRDs never move backwards when a slot is rolled back. Istio doesn't support CRD downgrades.
Which slot is stable
Each slot has its own Helm values, using the standard istiod chart keys:
Promotion flips revisionTags between the two slots and points defaultRevision at the same slot. The slot values load after all the shared values, so no other layer can override a slot's revision or tags by accident. At the default layer, the idle slot runs at 0 replicas until a cycle needs it.
Certificates and ordering
The istiod certificate issued by istio-csr covers both slots permanently, so activating a slot never needs a certificate change:
An Argo CD presync hook waits for the istiod-tls secret before the slots sync, so webhooks never come up without a valid certificate.
Checking the state of a cluster
Further reading
The takeaway
Istio upgrades at scale, without the toil.
With two permanent control-plane slots, one stable revision tag, and GitOps, every upgrade becomes a small, reviewed merge request. Argo CD applies the change, normal pod churn completes the transition, and rollback is moving the tag back.
The fleet grows. The upgrade process stays simple.
If you’d like to hear more tips and stories from our team of innovators and developers, check out our full range of Tech Team Stories.
Published on .
Ready to build better conversations?
Simple to set up. Easy to use. Powerful integrations.
Get started)
