Part 1 of the Multi-Cloud Banking series.

Most banks have a disaster recovery site. Very few have ever sent real customer traffic to it. The DR data centre sits there, kept roughly in sync, tested once a year in a carefully staged exercise, and everyone quietly hopes the day it’s really needed never comes. When it does, the runbook is out of date, a firewall rule is missing, and the 30-minute recovery target turns into a long night.

This series is about how we replaced that model at a GCC bank. I joined the programme as a consultant on the platform side. The goal was to move the bank’s entire infrastructure, more than a hundred business applications, off legacy hosting and onto Kubernetes within two years, across two clouds from two different vendors, both serving traffic. Oracle Cloud is the primary site, Huawei Cloud the secondary, and the bank’s data never leaves the country.

This first part covers the why and the overall architecture. The rest of the series goes layer by layer.

Why leave the data centre at all

Before this programme, the bank ran its infrastructure the way most banks do. It bought servers and storage arrays every few years, racked them, patched their firmware, renewed support contracts with a long list of hardware vendors, and planned capacity a year ahead because a new storage array took months to arrive. A large share of the infrastructure team’s time went into keeping that estate alive, not into the applications the bank actually earns money from.

Moving to the cloud changes what the bank operates.

• The provider runs the infrastructure. Hardware, storage, the network fabric, the virtualisation layer and the cloud’s own control plane are the provider’s job, under a contract with agreed service levels. Failed disks, firmware upgrades and hardware refreshes stop being the bank’s problem.
• Capacity is a request, not a purchase. A dedicated region still has a finite size, and growing it is planned with the provider. But a new environment, cluster or database comes out of that capacity in hours, through Terraform, instead of after a procurement cycle.
• A running cost instead of capital purchases. The old model was capex: a large hardware purchase every few years, depreciated over its life, and sized for growth that might never come. The new one is opex: a service fee for infrastructure the provider refreshes. A dedicated region isn’t pay-as-you-go, though. It’s sold as a multi-year commitment, so what the bank gets is a predictable running cost in place of irregular capital spikes, not unlimited elasticity.
• Everything is an API. In the old estate, a new application meant separate tickets for virtual machines, storage, firewall rules and DNS, each handled by a different team. In the cloud, all of that is code, reviewed in a merge request and repeatable for the next environment (Part 3). It’s better evidence for auditors too: what’s in Git is what’s running.
• A second site without a second build. A second data centre built by the bank would have doubled the hardware, the facilities and the people to run them. A second dedicated region from another vendor gives the bank a second site that someone else operates.
• The team moves up the stack. Engineers who spent their days on hardware and hypervisors now work on the platform, automation and reliability, the work that decides whether mobile banking is up. Part 8 covers how the team made that move.

The bank’s own data centres don’t disappear. They stay the network hub (Part 4), home to the internet edge and to the links to branches, ATMs and partners. What they stop being is the place where the bank runs its servers.

Leaving VMware and the licence bill behind

Running its own hardware wasn’t the only cost the bank wanted to leave behind. The legacy estate ran on VMware, and like many banks, this one had watched its renewals change shape after Broadcom bought VMware: perpetual licences gone, products sold only as subscription bundles, and a bigger bill for the same servers. Licences the bank had bought once as capital spending turned into a yearly operating cost anyway, just a larger one. On top of the hypervisor sat the usual layers of commercial middleware and databases, each priced per core or per instance, each with its own audit clause.

The new platform drops most of that. The bank no longer licenses a hypervisor: both clouds run on the providers’ own compute, and workloads run on Kubernetes above it. Above the infrastructure, everything is open source, with an enterprise support contract where the bank wants one. That changes more than the invoice.

• The bank pays for support, not for the right to run. Adding a node, a replica or a whole environment doesn’t trigger a licence true-up, and there’s no core count to argue about in an audit. Scaling up for month-end is a capacity question, not a procurement one.
• Test environments cost hardware, not licences. The shared non-production cluster and the staging pair in Part 3 run exactly the software production runs. Under per-core licensing, every one of those copies would have been a line in the budget, and staging would have been cut down to save money.
• The support vendor can be replaced. If a support partner underperforms or raises its prices, the bank can move to another one, or support a component itself, without changing the platform.
• The same software runs on both clouds. An Oracle managed service doesn’t exist on Huawei, and the other way round. PostgreSQL, Kafka, Istio and Vault run the same way on both, which is what makes a two-vendor design possible at all.

A few components don’t meet the strict definition, and those calls were made knowingly. MongoDB is published under the SSPL and Elastic Security under the Elastic License: both are source-available rather than open source. Their terms restrict offering the software as a service to others, which the bank doesn’t do, and paid subscriptions cover the support and features it needs. Portworx, used for a handful of legacy volumes (Part 6), is a commercial product. All three sit on a short list of exceptions that’s reviewed whenever a vendor changes its licence terms, which in recent years several have.

Open source isn’t free. Support subscriptions cost money, and the bank needs engineers who can run Kafka and PostgreSQL when the support line is slow to answer. Oracle Database stays licensed for core banking, and the two clouds are commercial contracts in their own right. The difference is where the money goes: into infrastructure and support the bank chose, instead of licences that grow every time it scales.

The constraints we started from

A bank doesn’t get to pick its architecture from a blank page. Four constraints shaped every decision that followed.

• Data residency. The central bank requires customer data to stay inside the country. That rules out the obvious answer, a public hyperscaler region next door, for anything that touches customer data. Whatever we built had to run on infrastructure physically located in-country.
• No single vendor to depend on. Regulators in the region increasingly look at concentration risk: what happens to the bank if its one cloud provider has a bad day, a contract dispute, or a sanctions problem. The bank wanted a second site that fails differently from the first, which means a different vendor, not just a different building.
• No lock-in above the infrastructure. Every component had to be open source with enterprise support available, so the bank could switch support vendors or support itself. Proprietary managed services were acceptable for the infrastructure layer only. This one turned out to pay for a lot of the programme, as the section on leaving VMware above explains.
• Recovery targets set by the business. The bank’s service-tiering framework gives the most critical services (core banking, mobile and internet banking, payments) a 99.9% availability target, a recovery time of 30 minutes or less, and at most 5 minutes of data loss. A cold DR site can’t meet that reliably. A warm one, exercised once a year, barely can.

Put together, those constraints point at one shape: two in-country sites, from two vendors, both running production all the time.

Two dedicated cloud regions, not two data centres

Neither of the bank’s cloud providers runs a public region in the country, so both sites are dedicated regions: the provider’s cloud, installed in a data centre in the country and run for the customer.

• The primary site is an Oracle Cloud dedicated region (OCI Dedicated Region Cloud@Customer). It’s a full OCI region with the same APIs, services and console as the public cloud, but the hardware sits in-country. Kubernetes runs on OKE, Oracle’s managed Kubernetes service.
• The secondary site is a dedicated Huawei Cloud region (Huawei Cloud Stack) in a second city, roughly 500 km away. Kubernetes runs on CCE, Huawei’s managed Kubernetes service.

Why Oracle as the primary? The bank’s core banking system and much of its estate already ran on Oracle databases. Keeping the system of record next to the Oracle-native tooling it depends on, and the licences that come with it, made OCI the natural home for the heaviest, most stateful workloads. Huawei brought a second, fully independent cloud stack: different hardware, different control plane, different operations team. A bug in one provider’s load balancer or IAM can’t take both sites down at once.

The distance matters too. About 500 km is far enough that a regional power or network event is unlikely to hit both cities. It’s also close enough that the round trip between sites is a handful of milliseconds. That’s what makes synchronous replication of the core banking database affordable, as Part 6 explains.

Kubernetes is the common layer

Running two clouds from two vendors only works if you stop caring which cloud you’re on as early as possible. For us, that line is Kubernetes.

Below Kubernetes, everything is cloud-specific, and we accepted that. Each site has its own Terraform, written in layers against its own provider: the environment, the compartment or enterprise project, DNS, the network, the managed cluster, cluster add-ons, object storage. OCI compartments and Huawei enterprise projects aren’t the same thing, and pretending otherwise with a leaky abstraction would have cost more than it saved.

Above Kubernetes, everything is identical. Both clusters run the same platform, delivered by the same GitOps chart, with only a values overlay per cluster:

• Argo CD in a hub-and-spoke setup, deploying both clusters from one place, outside both of them (more on that below).
• Istio as the service mesh, joining both clusters into one mesh.
• Cilium for pod networking policy and flow visibility.
• Vault and External Secrets for secrets, cert-manager for certificates, and Kyverno for admission policy.
• Kubernetes operators for every stateful service: CloudNativePG for PostgreSQL, Strimzi for Kafka, and operators for MongoDB, RabbitMQ, Redis and Elasticsearch.

An application team ships one Helm chart and one Argo CD application, and it lands on both clusters the same way. They don’t write anything that knows whether it’s running on OKE or on CCE. Part 3 covers that platform in detail.

Cloud-specific below Kubernetes, identical above it.

What “active-active” means here

“Active-active” gets used loosely, so here is what it covers in this design and what it doesn’t.

Stateless services run active-active. Every API, every channel back-end and every integration service runs in both clusters at all times, with real traffic. Istio joins the two clusters into one mesh using the multi-primary, multi-network model: each cloud has its own Istio control plane, and traffic between the clouds crosses through dedicated east-west gateways with mutual TLS end to end. A service calls its dependencies locally first. When a local dependency fails or its endpoints are ejected for errors, Istio sends the call to the healthy copy in the other cloud, without the caller doing anything. Part 5 covers how that works and how to stop a failover from turning into a retry storm.

Oracle is the preferred site, not the only one. Customer traffic enters through a global load balancer that sends most of it to the Oracle site and some of it to Huawei. Both sites take real traffic every day, so the secondary is always proven to work. That’s the opposite of a DR site that sits idle until the worst day of the year.

Data has one writer at a time. Writing the same account balance in two places at once and reconciling it later is not something a bank should do. Each database has a single primary, normally at the Oracle site, with a standby at the Huawei site. The core banking database, on Oracle Database outside Kubernetes, replicates synchronously. Everything else replicates asynchronously, with its lag measured against its tier’s data-loss limit. Part 6 is about state, service by service.

So “active-active” in this series means: both sites serve traffic all the time, any stateless service can fail over to the other cloud on its own, and data fails over in a controlled way within the recovery targets of its tier.

The architecture at a glance

Two dedicated cloud regions joined by the bank’s network and one Istio mesh, run and observed from a third cloud.

Some of that picture needs explaining.

The bank’s network is the hub. Neither cloud connects to the other directly. Each one connects to the bank’s own network over redundant private circuits, and traffic between the clouds transits the bank’s network. The bank already owns that network, along with its firewalls, monitoring and the branches, ATMs and partner links that live on it. Part 4 covers the circuits, the encryption on top of them, and DNS across both sites.

The address plan has no overlaps. Each cloud has its own address range for nodes and separate ranges for pods and Kubernetes services, and none of them overlap with each other or with the bank’s existing network. That sounds obvious, but it’s the one decision you can’t change later. It’s also what lets a database in Huawei replicate from a database in Oracle without network address translation in the middle.

Nothing is exposed directly. Both clusters use internal load balancers only. Customer traffic comes in through the bank’s existing edge, which handles the internet-facing side and global load balancing, and is forwarded to the Istio ingress gateway in each cluster.

Control and observability live in a third place. If the tooling for a two-site system runs in one of those two sites, losing that site also takes away your ability to see what happened and to act on the site that’s left. So both live on AWS, in an in-country Local Zone: outside both clouds and inside the country. It’s the bank’s management site.

• The platform control plane: GitLab, its CI runners, Harbor as the source of every image, and the Argo CD hub that deploys both clusters. If Oracle goes down, we can still change, scale and fix Huawei.
• A Grafana LGTM stack: Mimir for metrics, Loki for logs, Tempo for traces, Grafana on top. Part 7 covers how that pipeline works and what we had to strip from the telemetry before it could leave the clusters.

What we kept off the management site matters just as much. Nothing a running pod needs at runtime lives there. Each site runs its own Vault and its own registry mirror, so a pod can restart, scale or move between nodes with AWS unreachable. Losing the management site freezes changes; it doesn’t stop traffic.

Tiering decides how much of this each service gets

Not every service needs everything in this picture, and paying for a synchronous standby on a marketing website would be wasteful. The bank’s tiering framework decides who gets what.

Every business service is scored on three separate axes, confidentiality, integrity and availability, each from tier 1 (critical) to tier 4 (basic). Applications inherit the highest tier of every business service they support, and the infrastructure under them inherits from the applications. A rating like C1-I1-A1 (core banking, mobile banking) means tier 1 on all three. C4-I3-A2 (a public website) means very different controls.

Those tiers translate directly into the architecture:

• Availability tier decides the replica counts in each cluster, whether a service must run in both clouds, and how aggressively the mesh fails it over.
• Integrity tier decides how much data loss a dataset can tolerate, and from that, how it replicates, how often backups run and how long they’re kept.
• Confidentiality tier decides network segmentation, encryption and who can access the service at all.

The framework is simple, and few things saved us more time. Arguments like “my service is important” became a lookup table. Part 2 goes through it, alongside the applications a bank actually runs and how each kind lands on this platform.

What we gave up

This architecture has real costs.

• Two of everything to operate. Two Terraform codebases, two cloud consoles, two support contracts, two sets of quotas and two upgrade calendars. OKE and CCE don’t release Kubernetes versions on the same schedule, so keeping both clusters on compatible versions is a continuous job, not a one-off.
• The lowest common denominator above Kubernetes. We couldn’t use OCI’s or Huawei’s managed databases, queues or caches for anything that had to fail over across clouds, because the other cloud doesn’t have the same service. Everything stateful runs on Kubernetes operators, which means the bank runs its own Kafka, PostgreSQL and MongoDB, with enterprise support contracts behind them.
• Cross-cloud networking is the hardest part. Every cross-site call depends on the circuits, the firewalls in the bank’s network, and the east-west gateways. A misconfigured route there looks exactly like an application outage.
• The bill arrives every month. Opex is visible and grows with use, so cost needs managing all the time, not approving once every five years. Two dedicated regions, with tier 1 services sized to carry the whole load at either site (Part 5), are a standing cost the bank pays whether or not a failover ever happens.
• Skills. The team has to know two clouds well enough to troubleshoot both under pressure, and it started with no cloud experience at all. Consultants built the platform alongside the bank’s engineers and handed it over wave by wave; Part 8 covers how.

We accepted those costs because the alternative, a single cloud with a DR site we’d rarely use, would have failed the regulator’s concentration test and the bank’s own recovery targets.

Next

The next part steps back from the infrastructure to what runs on it: the applications a bank actually operates, from core banking to payments and cards, how the tiering framework classifies them, and which ones moved first.

Multi-Cloud Banking

1. Multi-Cloud Banking: Moving a GCC Bank to Kubernetes on Two Clouds (you are here)
2. Multi-Cloud Banking: What a Bank Runs, and How Tiering Decides Where It Lands (coming soon)
3. Multi-Cloud Banking: One Platform on Two Clouds with Terraform and GitOps (coming soon)
4. Multi-Cloud Banking: The Network Between Two Clouds (coming soon)
5. Multi-Cloud Banking: Istio Active-Active Across Two Clouds (coming soon)
6. Multi-Cloud Banking: State Is the Hard Part (coming soon)
7. Multi-Cloud Banking: Observability From a Third Cloud (coming soon)
8. Multi-Cloud Banking: Moving 100+ Applications Onto a Greenfield Platform (coming soon)

During an incident, the first thing most teams do is add replicas. Latency is up, pods look busy, so you scale the deployment from 40 pods to 120 and wait for things to calm down. Sometimes they do. Sometimes the database starts refusing logins, and an incident in one service becomes an outage of everything that shares that database.

At Anghami / OSN+ we had that second kind of incident more than once. The database connection count followed the number of pods, not the traffic, and every scale-out pushed it closer to max_connections. Our backend was PHP, Java and Python, and later Go, and each one handled connections in its own way. PHP didn't pool at all.

The fix was a pooling tier between the applications and the databases: ProxySQL in front of MySQL, PgBouncer in front of Postgres, both running as central deployments in the cluster. This post covers:

• why connections multiply with pods, and how each runtime makes it worse in its own way
• why PHP needed a proxy, and why the languages that do pool needed one too
• how we ran ProxySQL for the MySQL side and what broke its multiplexing
• how we ran PgBouncer in transaction mode for the Postgres side
• how to size a proxy tier so it doesn’t become the next thing that multiplies
• what changed during incidents once the proxies were in place

I wrote about sizing pods, autoscaling and circuit breaking in reliability at scale with Istio. That post ends at the database. This one picks up there.

Connections follow pods, not traffic

A database connection isn’t free on the server side. MySQL runs a thread for each connection, and Postgres runs a whole process for each one, with its own memory. Both have a hard max_connections, and it's usually set in the hundreds or low thousands, because every connection costs memory whether it's doing work or not.

The applications don’t see that cost. Each runtime opens connections by its own rules, and every rule ends with “× the number of pods”:

• PHP-FPM runs one request per worker process and shares nothing between workers. Each worker opens its own connection, either for each request or, with persistent connections, for the life of the worker. A pod with 20 workers holds up to 20 connections, and 60 pods hold 1,200.
• Java with HikariCP keeps a pool per JVM. The default pool size is 10, and by default HikariCP keeps all of them open even when idle. 40 pods hold 400 connections, all the time.
• Python with SQLAlchemy keeps a pool per process, not per pod. A Gunicorn pod with 4 workers and the default pool of 5 plus 10 overflow can open 60 connections. 30 pods can open 1,800.
• Go’s database/sql keeps one pool per process, which is the right shape. But MaxOpenConns defaults to zero, which means no limit. Under load, a Go service opens as many connections as it has concurrent queries.

On a normal day the totals fit. During a spike or an incident, the autoscaler or an engineer triples the replica count, and every row in that picture triples with it. The database limit stays where it was.

It also fails in the worst possible way. When MySQL or Postgres runs out of connections, it refuses new logins for everyone, not just the service that scaled. A service that had nothing to do with the incident can’t connect either. Its callers retry, the retries open more connections, and the outage spreads sideways through the shared database.

Why PHP needed a proxy, and why the others did too

PHP was the obvious case. A PHP-FPM worker can’t share a connection with the worker next to it, so there’s no pool to tune inside the application. Whatever sits between PHP and the database has to do the pooling for it. Persistent connections don’t help here: they save the handshake, but they keep one connection per worker open forever, which is the opposite of what we needed.

The runtimes that do pool still had the multiplication problem. An in-process pool solves the cost of opening a connection for each request. It doesn’t solve the number of pods. Five connections per pod is a good pool size, but five connections × 300 pods is still 1,500 connections on one Postgres, and most of them are idle at any given moment.

So the rule became: no application talks to a shared database directly. Every service connects to a proxy, and the proxy holds a fixed, small number of connections to the database. How many pods are behind the proxy stops mattering to the database.

We used a different proxy for each engine, because each engine has a proxy that speaks its protocol properly:

• ProxySQL for MySQL. It understands the MySQL protocol and every query that goes through it, so it can multiplex connections, route reads and writes, and tell you which queries are expensive.
• PgBouncer for Postgres. It’s a single small process that pools connections and nothing else. Postgres connections are the more expensive of the two, so it mattered even more there.

ProxySQL in front of MySQL

ProxySQL ran as a Deployment in the cluster with a Service in front of it. Applications pointed their MySQL host at that Service instead of the database. For PHP that was a one-line config change, and nothing in the code knew the difference.

Multiplexing

The feature that made ProxySQL worth it is multiplexing. ProxySQL accepts thousands of client connections, and it answers the MySQL handshake and authentication itself, without touching the database. When a client sends a query, ProxySQL borrows an idle backend connection, runs the query on it and gives the connection back to the pool as soon as the result is returned. A PHP worker that holds a connection for the whole request, but only spends a few milliseconds of it running queries, uses a backend connection only for those milliseconds.

That changed what scaling PHP did to the database. PHP could keep doing what PHP does, opening a connection per request, because opening a connection to ProxySQL inside the cluster is cheap. The database only saw ProxySQL’s pool.

The configuration that mattered

Most of ProxySQL’s configuration lives in a few tables. These were the ones we cared about:

mysql_variables=
{
    threads=4
    max_connections=8000          # client connections ProxySQL accepts
    connect_timeout_server=1000   # ms to open a backend connection
    connect_timeout_server_max=3000  # ms a query may wait for one before it fails
    default_query_timeout=30000   # ms before ProxySQL kills a query
}

mysql_servers=
(
    { address="catalog-db-primary.internal", port=3306, hostgroup=10, max_connections=100 },
    { address="catalog-db-replica-1.internal", port=3306, hostgroup=20, max_connections=100, max_replication_lag=5 },
    { address="catalog-db-replica-2.internal", port=3306, hostgroup=20, max_connections=100, max_replication_lag=5 }
)

mysql_query_rules=
(
    { rule_id=1, active=1, match_digest="^SELECT .* FOR UPDATE", destination_hostgroup=10, apply=1 },
    { rule_id=2, active=1, match_digest="^SELECT", destination_hostgroup=20, apply=1 }
)

max_connections in mysql_servers is the number that protects the database. It caps how many connections one ProxySQL instance opens to that backend. When the pool is full, new queries wait in ProxySQL instead of opening another connection. After connect_timeout_server_max they fail with error 9001, which is what you want: the default is 10 seconds, and we cut it to three.

The query rules split reads from writes: SELECT ... FOR UPDATE goes to the primary, other SELECTs go to the replicas, and everything else falls through to the user's default hostgroup, the primary. max_replication_lag takes a replica out of rotation when it falls more than five seconds behind, so a lagging replica doesn't serve stale data.

Read/write splitting has one trap: a user who writes something and reads it back right away can hit a replica that hasn’t caught up. For the few flows where that mattered, we sent the read to the primary with a query rule matched on the digest, rather than changing the application.

What breaks multiplexing

Multiplexing only works if a backend connection carries no state that belongs to one client. When a client does something that ties state to its connection, ProxySQL stops multiplexing that session and pins it to one backend connection until the state goes away. The usual causes:

• Transactions. The backend connection stays with the client until COMMIT or ROLLBACK. That's expected, but a transaction left open by a bug pins a connection for as long as it stays open.
• User variables. SET @foo = ... lives on the connection, so ProxySQL keeps the client on it. Some ORMs and frameworks set variables on every connection without you asking.
• Temporary tables, GET_LOCK() and LOCK TABLES. All of them are connection state.
• SQL_CALC_FOUND_ROWS. The FOUND_ROWS() that follows has to run on the same connection.

Old PHP code had plenty of these. A pinned session behaves like a direct connection, so the backend pool drains much faster than the traffic suggests. ProxySQL shows it plainly in stats_mysql_connection_pool:

SELECT hostgroup, srv_host, status, ConnUsed, ConnFree, ConnERR, Queries, Latency_us
FROM stats_mysql_connection_pool;

If ConnUsed sits near max_connections while the query rate is low, sessions are being pinned. stats_mysql_query_digest tells you which queries were running, and the fix is usually in the application: drop the user variable, replace SQL_CALC_FOUND_ROWS with a separate COUNT(*), make the transaction shorter.

Users live in two places

ProxySQL authenticates clients itself, so it needs every application user and password in its own mysql_users table, in addition to MySQL. Rotating a password means changing it in both places, in the right order. Keep ProxySQL's users generated from the same source as the database users, or the first rotation will lock a service out.

PgBouncer in front of Postgres

The Postgres side was newer: Python services, and later the Go services from the move from PHP and Java to Go. Both runtimes pool in process, so the problem there was only the pod count, not the per-request connections. PgBouncer handled it.

Transaction mode

PgBouncer has three pool modes, and the one you pick decides what the applications can still do:

• Session mode gives a client a server connection for as long as the client stays connected. It’s the default, and it does almost nothing for our problem, because application pools stay connected all the time.
• Transaction mode gives a client a server connection for one transaction, then returns it to the pool. A thousand idle application connections need no server connections at all.
• Statement mode returns the connection after every statement and doesn’t allow multi-statement transactions. Too strict for most applications.

We ran transaction mode:

[databases]
orders = host=orders-db.internal port=5432 dbname=orders

[pgbouncer]
listen_port = 6432
pool_mode = transaction
max_client_conn = 5000        ; client connections PgBouncer accepts
default_pool_size = 40        ; server connections per database/user pair
max_db_connections = 50       ; hard cap per database, whatever the users
reserve_pool_size = 5
reserve_pool_timeout = 3
query_wait_timeout = 15       ; seconds a client may wait for a server connection
server_idle_timeout = 60
max_prepared_statements = 200
auth_type = scram-sha-256

max_db_connections is PgBouncer's version of ProxySQL's per-server max_connections: the cap that protects the database. default_pool_size applies per database and user pair, so with several users on one database it's easy to add up to more than you meant. The database-level cap is the one to trust.

query_wait_timeout matters more during an incident than any other setting. Its default is 120 seconds. A client that waits two minutes for a connection has long since timed out upstream, and its caller has retried. We lowered it so that when the pool is exhausted, clients get an error quickly and the load sheds, instead of a queue of requests that nobody is waiting for anymore.

What transaction mode breaks

Transaction mode has the same weakness as ProxySQL’s multiplexing: anything that lives on the connection outside a transaction is lost or, worse, leaks to the next client.

• Session SET. SET search_path or SET statement_timeout outside a transaction applies to whichever client gets that server connection next. Use SET LOCAL inside the transaction, or set the value on the role.
• Session advisory locks and LISTEN. Both belong to a session. Use transaction-scoped advisory locks, and give anything that needs LISTEN/NOTIFY a direct connection.
• Prepared statements. PgBouncer tracks protocol-level prepared statements in transaction mode when max_prepared_statements is set, so Go's pgx, which prepares and caches statements by default, works as is. SQL-level PREPARE and DEALLOCATE still aren't tracked, and queries that use them fail randomly with "prepared statement does not exist".

The application pools stay, but small

Python and Go kept their in-process pools. They now hold connections to PgBouncer, which are cheap, instead of connections to Postgres, which aren’t. What still matters is that every pool has a limit:

db.SetMaxOpenConns(10)
db.SetMaxIdleConns(10)
db.SetConnMaxLifetime(5 * time.Minute)

engine = create_engine(url, pool_size=5, max_overflow=0, pool_pre_ping=True)

The Go limit is the one people forget. Without SetMaxOpenConns, a Go service under load can open thousands of connections to PgBouncer and use up max_client_conn for every other service behind it. SetConnMaxLifetime makes connections recycle, so they spread across PgBouncer replicas again after one of them restarts.

Running the proxy tier

The proxy tier sits in the path of every query, so it has to be at least as reliable as the database behind it. And it’s easy to rebuild the original problem inside it.

Proxies multiply too

Every ProxySQL instance has its own pool to each backend, and so does every PgBouncer. With three ProxySQL replicas and max_connections=100 per backend, the primary can see 300 connections. If someone puts an HPA on the proxy Deployment and it scales to ten replicas during an incident, the primary can see 1,000. The proxy tier has just become the thing that scales through the database limit.

So the proxies were fixed-size Deployments, sized from the database, not the traffic:

proxy replicas × pool per replica  ≤  max_connections − reserved

reserved covers what doesn't go through the proxy: admin access, migrations, replication and monitoring. We picked the replica count for availability and CPU headroom first, then divided the database budget between them. Changing either number was a reviewed change to the proxy config, never something an autoscaler did on its own.

Availability

A few things kept the tier from becoming a single point of failure:

• At least three replicas, spread across zones, with a PodDisruptionBudget so node drains and cluster upgrades never took more than one away at a time.
• Requests and limits set for CPU, and room to spare. PgBouncer is single-threaded, so one replica tops out at one core. When it’s busy, add replicas and split the database budget, rather than giving a replica more CPU it can’t use.
• Graceful shutdown. Kubernetes stops a pod with SIGTERM, and PgBouncer answers it by waiting for every client to disconnect, which application pools never do on their own. So a preStop hook sends SIGINT instead: PgBouncer stops taking new connections, lets running transactions finish and exits. ProxySQL got a preStop sleep so the Service stopped sending it traffic first, and both had a termination grace period longer than our longest query timeout.
• Configuration from git. Both proxies read their config from a ConfigMap and their user credentials from a Secret, deployed like everything else. ProxySQL’s admin interface can change the running config live, which is useful during an incident, but a change made there and not written back to git is lost on the next restart.

The traffic from the applications to the proxies went through the mesh like any other call, so the connection limits from the Istio post still applied: a DestinationRule on the proxy Service capped how many connections each calling pod could open.

Watching it

Both proxies expose the numbers that matter, and we scraped them with exporters into the same dashboards described in observability at scale. The three that told us the most:

• Backend connections in use vs. the cap. ConnUsed in ProxySQL, sv_active in PgBouncer's SHOW POOLS. Sitting at the cap means queries are waiting.
• Clients waiting. cl_waiting and maxwait in PgBouncer, queued connections in ProxySQL. This is the earliest sign of trouble, and it shows up before latency does.
• Connection errors to the backend. A spike here means the database is refusing connections, and the budget math is wrong somewhere.

What changed during incidents

The difference showed up the first time a PHP service scaled out under load with ProxySQL in front of it. Before, 40 pods going to 120 pods meant the database was asked for 2,400 connections it didn’t have. After, the database saw the same 300 connections it saw on a quiet afternoon.

That doesn’t make the overload go away. If the database can’t keep up with the queries, they queue in the proxy instead. But a queue is a much better failure than a refused login:

• The blast radius shrank. Overload on one service turned into latency and timeouts for that service. Other services sharing the database kept their connections and kept working.
• The database stayed reachable. Engineers could still connect, look at the process list and kill the bad query, because the reserved connections were actually free.
• Scaling out became safe again. Adding replicas to the PHP or Python tier changed the load on the proxies, not the connection count on the database.

It also changed what we looked at first during an incident. “How many connections does the database have” stopped being an interesting question, because the answer was always the budget. The interesting questions became “how long are clients waiting in the proxy” and “which queries are holding the backend connections”, and the proxies answered both.

What I’d tell a team starting this

• Count connections per pod, then multiply by maxReplicas. Do it for every service on the database. If the total is bigger than max_connections, the next incident will find out for you.
• Put a proxy in front of shared databases, even for languages that pool. In-process pools fix the cost per request, not the number of pods.
• Pick the proxy for the engine. ProxySQL for MySQL, PgBouncer in transaction mode for Postgres. Once they’re set up, neither needs much attention.
• Hunt down connection state. User variables, session SET, temporary tables and session locks quietly turn a pooled connection back into a dedicated one.
• Don’t autoscale the proxies. Size them from the database budget. Proxy replicas × pool per replica is your real connection count.
• Fail fast in the queue. Lower PgBouncer’s query_wait_timeout and ProxySQL's connect_timeout_server_max so a full pool sheds load instead of holding requests nobody is waiting for.

The proxies didn’t make our databases faster. They made the connection count something we set in a config file, instead of something the autoscaler decided during the worst ten minutes of an incident.

Rewrites usually fail at the cutover, not in the code. A team spends months building the new system next to the old one, then switches all the traffic in one night. Something nobody tested breaks, and the old system comes back the next morning.

At Anghami / OSN+ we moved our backend from legacy PHP and Java applications to Go services that talk gRPC. We never did a cutover night. Every new service handled a copy of real production traffic before it handled any real request. After that, it took over its URL paths one prefix at a time, and each legacy component was removed only once nothing was routed to it anymore.

I was the Cloud & DevOps Architect there. This post walks through the migration in the order we did it: why Go and gRPC, how the monorepo and its CI were set up, how web and mobile clients kept their REST API, how Istio moved the traffic route by route, and how the data followed.

Why we left PHP and Java

The legacy backend was a mix of PHP applications and Java services, built over many years by different teams. It worked, but it got harder to scale every year.

The PHP side scaled by adding processes. PHP-FPM handles one request per worker process. Our endpoints mostly waited on databases, caches and other services, so a worker spent most of its time idle while still holding its memory. To serve more concurrent requests we had to run more workers, and more workers needed more memory. Our nodes filled up on memory long before they ran out of CPU.

The Java side was slow to scale out. The JVM services needed large heaps, and a new pod took a while to start and warm up before it reached full speed. When traffic jumped, the autoscaler added pods, but they weren’t useful until a few minutes later, and traffic spikes are exactly when those minutes matter. I wrote about that gap in sizing, autoscaling and circuit breaking with Istio.

Nothing had a clear contract. Services called each other over ad hoc HTTP/JSON. Each team described its payloads differently, and a renamed field could break a consumer nobody knew about.

Why Go and gRPC

We wanted services that use little memory, start fast and handle many concurrent requests in one process. Go fits all three:

• Small footprint. A Go service is one static binary in a minimal container image, and it uses a fraction of the memory of the PHP-FPM pool or JVM it replaces.
• Fast startup. A new pod is ready in seconds, so autoscaling and node churn hurt less. That matters even more when every node is Spot, like in our full-Spot EKS clusters.
• Concurrency built in. Goroutines let one process wait on thousands of slow calls at once, which suits a backend that mostly waits on other systems.
• Easy to read. Engineers coming from PHP and Java picked Go up quickly, and Go code looks much the same whichever team wrote it.

Between services we used gRPC. The .proto files became the contract: the request and response types are written down once, and the Go clients and servers are generated from them. A breaking change fails the build instead of a request in production. The traffic is binary and goes over HTTP/2, which cut payload sizes and connection overhead compared to the JSON calls it replaced.

gRPC has one well-known problem on Kubernetes. It keeps a long-lived HTTP/2 connection and sends many requests over it, and Kubernetes Services balance connections, not requests. One busy client pod can pin all its traffic to one server pod. We already ran Istio, and its sidecars balance gRPC per request, so the problem never reached us. If you adopt gRPC without a mesh, plan for client-side load balancing.

One monorepo, split by domain

All the Go code went into one repository. Inside it, the code was split into domains, and each domain had its own services:

backend/
├── api/                    # .proto files and the generated Go code
│   ├── catalog/v1/
│   └── accounts/v1/
├── pkg/                    # shared packages, each one a Go module
│   ├── db/                 # connection pools, health checks, query metrics
│   ├── logging/            # structured logs with trace IDs
│   ├── events/             # Kafka producer and consumer, event envelope
│   └── grpcx/              # server and client setup, interceptors
└── services/
    ├── catalog/
    │   ├── content/        # one service, one Go module
    │   └── search/
    └── accounts/
        ├── profile/
        └── auth/

A monorepo gave us things that are hard to get across many repositories:

• One place to search. Anyone could find how another team solved a problem, or every caller of an RPC, with a single search.
• One set of conventions. Linting, code generation, Dockerfiles and the service layout were the same everywhere, so moving between services or teams didn’t mean learning a new setup.
• Contracts next to the code. The .proto files sat beside the services that implement them, and a contract change was reviewed in the same place as the code that used it.

The shared packages under pkg/ held everything a service needs but nobody wants to write twice: database access, logging, events, and the gRPC server and client setup with tracing, metrics and auth interceptors. A new service got all of that by importing a few packages.

We had one strict rule for pkg/: no business logic. Shared packages are plumbing. Once a shared package knows what a "subscription" or a "playlist" is, every service that imports it is coupled to every other one, and you've built a distributed monolith inside your monorepo.

CI builds only what changed

A monorepo with 60+ services can’t rebuild everything on every commit. Each service and each shared package had its own GitHub Actions workflow, triggered only by changes under its own path:

# .github/workflows/catalog-search.yml
name: catalog-search
on:
  pull_request:
    paths:
      - "services/catalog/search/**"
  push:
    branches: [main]
    paths:
      - "services/catalog/search/**"

jobs:
  build:
    uses: ./.github/workflows/go-service.yml
    with:
      service: catalog/search
    secrets: inherit

The service workflow is a thin wrapper around a reusable one, so every service runs the same test, lint, security scan and image build. That central workflow is the one I described in rebuilding CI/CD with GitHub Actions and Argo CD.

The two kinds of target ended in different places:

• A shared package was tested and published with a new version to our artifact registry. A change to pkg/events built and released pkg/events, and nothing else.
• A service was built into a container image, tagged with its own version and pushed to ECR. A change to services/catalog/search built that one image.

What matters more is what didn’t get rebuilt. A change to a shared package didn’t rebuild the services that use it. Each service pinned the versions of the packages it imports in its own go.mod, and moved to a new version through a pull request in that service:

// services/catalog/search/go.mod
 require (
-	backend/pkg/events v1.6.2
+	backend/pkg/events v1.7.0
     backend/pkg/logging v1.3.0
 )

The service’s own CI then tested it against the new version before it shipped. The .proto contracts worked the same way: the generated code under api/ was published as a versioned package, so a service picked up a new contract only when its team chose to.

This is a trade-off. Always building every service against the latest pkg/ is simpler, and nobody falls behind. But then one change to the database package rebuilds and redeploys 60 services at once, and one bad release breaks all of them together. Pinned versions let the team that owns pkg/db release on Tuesday, and each service team adopt it when they're ready. The cost is drift: some services fall a few versions behind, and somebody has to notice. We watched which version each service was on and chased the ones that fell too far behind.

Each service deploys on its own

Deployment was separate from the repository layout. Each service had its own Argo CD Application, its own image and its own release version. Shipping catalog/search meant changing its image version in the GitOps repository through a pull request. Argo CD then rolled out that one service and left the other 60 alone.

One repository for the code doesn’t mean one release for the code. Teams released on their own schedule and rolled back their own service. A bad deploy stayed inside one service.

A gateway for web and mobile clients

Web and mobile clients don’t speak gRPC as easily as services do. Browsers can’t make raw gRPC calls, and our mobile apps already used a REST/JSON API. Old app versions stay installed for years, and they would keep calling the old URLs whatever we changed on the server.

So the services spoke gRPC inside the mesh, and an edge service in Go translated REST/JSON from clients into gRPC calls. We built it with grpc-gateway, which generates that translation layer from annotations in the .proto files:

service CatalogService {
  rpc GetContent(GetContentRequest) returns (Content) {
    option (google.api.http) = {
      get: "/v1/catalog/content/{content_id}"
    };
  }
}

The edge service registers the generated handlers and points each one at its gRPC service:

mux := runtime.NewServeMux(
    runtime.WithMarshalerOption(runtime.MIMEWildcard, &runtime.JSONPb{
        MarshalOptions: protojson.MarshalOptions{
            UseProtoNames:   true, // keep snake_case field names in JSON
            EmitUnpopulated: true, // send zero values, like the legacy API did
        },
    }),
)

opts := []grpc.DialOption{grpc.WithTransportCredentials(insecure.NewCredentials())}
err := catalogv1.RegisterCatalogServiceHandlerFromEndpoint(
    ctx, mux, "catalog-content.catalog.svc.cluster.local:9090", opts)

router := chi.NewRouter()
router.Use(middleware.RealIP, middleware.Recoverer)
router.Handle("/v1/*", mux)

The connection is plaintext as far as the code is concerned. The Istio sidecars wrap it in mTLS.

The only web framework we used was chi, and we picked it because it barely is one. It’s a router built on the standard net/http interfaces, so middleware for auth, logging and metrics is a plain func(http.Handler) http.Handler, and grpc-gateway's mux mounts on it like any other handler. Endpoints that weren't generated from a .proto file, like health checks and webhooks, lived on the same router.

This gave us a few things we needed for the migration:

• The URLs and JSON stayed the same. The google.api.http paths and the marshaler options reproduced the legacy API, so existing app versions didn't notice the backend had changed. This is also what made path-based routing possible later.
• One source of truth. The REST API and the gRPC contract came from the same .proto file, so they couldn't drift apart.
• Room for client-specific logic. A mobile screen that needs data from three services can get it in one call. The edge service fans out over gRPC inside the cluster, where calls are cheap, instead of the phone making three round trips over a mobile network.

We also looked at the gRPC-JSON transcoding filter in Envoy, configured on the Istio ingress gateway. It saves a network hop, but it needs an EnvoyFilter and the compiled proto descriptors loaded into the gateway. EnvoyFilter patches Envoy's internal configuration directly, so it's the part of Istio most likely to break on an upgrade. It also leaves no place for client-specific logic. A small Go service built from the same monorepo was easier to own.

Switching over without a cutover night

The migration followed the strangler fig pattern: build the new system around the edges of the old one, move traffic over a piece at a time, and remove the old parts once nothing uses them. Istio did the traffic moving. Every legacy route went through three stages.

Stage 1: mirror production traffic

Before a new service handled a real request, it handled copies of them. Istio’s mirroring sends each request to its normal destination and a copy to a second one. The client only ever gets the legacy response; the mirrored response is thrown away.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: api
spec:
  hosts: ["api.example.com"]
  gateways: ["istio-system/public"]
  http:
    - match:
        - uri:
            prefix: /v1/catalog/
      route:
        - destination:
            host: legacy-php.legacy.svc.cluster.local
      mirror:
        host: edge-shadow.edge.svc.cluster.local
      mirrorPercentage:
        value: 10.0

We started with a small percentage and raised it as the new service held up. The shadow deployment ran the image we were about to ship, against real requests from real clients, with the odd payloads and traffic spikes that no load test we wrote ever had.

Mirroring copies writes too. If the shadow had written to the production database, every mirrored POST would have happened twice. So the shadow deployment was configured with staging data: the same code, pointed at staging databases and topics. Mirrored writes landed somewhere harmless. The trade-off is noise: a request about a record that exists only in production fails in the shadow, and you have to learn to tell those failures apart from real bugs.

Mirrored requests are fire-and-forget, so you can’t compare responses in the client. We compared what the mesh measures instead. Istio produces metrics for the mirror destination like for any other, so error rates and latency for the legacy route and its shadow sat next to each other on the same dashboards, from the stack I described in observability at scale. The shadow’s logs and traces showed which requests failed and why. Istio also appends -shadow to the Host header of mirrored requests, which made them easy to filter.

Stage 2: route by path

Once the shadow’s error rate and latency matched or beat the legacy route, we moved real traffic, one path prefix at a time:

http:
    - match:
        - uri:
            prefix: /v1/catalog/
      route:
        - destination:
            host: edge.edge.svc.cluster.local
    - route:
        - destination:
            host: legacy-php.legacy.svc.cluster.local

Prefixes we had already moved went to the Go edge service, and everything else still went to the legacy app. Rolling back was a one-line change to the VirtualService, applied through Argo CD like everything else, and it took effect in seconds without redeploying any service.

Moving by prefix kept every step small. Each step moved one area of the API that one team owned, and that team watched it. If a step went wrong, only that prefix was affected.

Stage 3: retire the legacy app

When no prefix routed to a legacy component anymore, we watched its traffic go to zero, kept it running a little longer as a fallback, and then removed it. We treated deleting the old code as part of the migration, and didn’t leave it as a cleanup task for later. Every legacy component that stays running “just in case” still needs patching, monitoring and someone who remembers how it works.

Moving the data

New services got their own databases. Sharing the legacy schema would have tied every new service to the old data model, and to the old applications that also wrote to it.

During the migration, though, the legacy applications were still writing to the legacy databases. The new services needed that data before they took any traffic, and needed to keep up with it until the switch. We used change data capture for this: Debezium read the legacy database’s change log and published every insert, update and delete to Kafka. Each new service consumed the topics it needed and wrote the data into its own database, in its own shape.

{
  "name": "legacy-catalog",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "database.hostname": "legacy-db.internal",
    "topic.prefix": "legacy",
    "table.include.list": "catalog.content,catalog.artists",
    "snapshot.mode": "initial"
  }
}

The initial snapshot loaded the existing rows, and the change stream kept the new database current after that. Debezium reads the MySQL binlog, so the legacy apps didn’t need any code changes.

Change data capture meant the legacy database stayed the source of truth for an area until its routes moved. After the switch, the new service owned the writes, and it published its own events to Kafka through pkg/events for anything that still needed the data. Once nothing depended on the legacy table, the connector was removed.

Watch the lag. If the new database runs behind the legacy one, a user can change something through a legacy route and not see it through a migrated one. Alert on consumer lag before you move traffic, and keep it near zero while both systems are live.

What I’d tell a team starting this

• Mirror before you route. Real traffic finds the bugs no test suite finds, and mirroring lets it find them while the legacy system still answers the user.
• Keep the client contract stable. The old URLs and JSON shapes are what make gradual routing possible. Change the API later, as its own project.
• Keep shared packages thin. Plumbing, not business logic, or the monorepo turns into a monolith.
• Pin shared package versions, then watch the drift. Independent releases are worth it, but only if someone notices the services that fall behind.
• Retire as you go. Turn each legacy piece off as soon as nothing routes to it. Otherwise the “temporary” fallback becomes the second system you run forever.

None of this needed a cutover night. Each step was small enough to roll back in seconds, so by the time the last legacy component was removed, switching over had become routine.

Part 5 of the EKS at Scale series.

Every service at Anghami deployed with a rolling update. It’s the Kubernetes default, and for a stateless HTTP API it’s mostly fine: new pods come up, pass their readiness probe, take traffic, and old pods go away a few at a time.

Two kinds of service didn’t fit that model.

The first was our Kafka consumers. Every pod that joins or leaves a consumer group triggers a rebalance, and during a rebalance the group’s partitions stop being consumed. A rolling update of a 20-pod consumer is 20 pods leaving and 20 joining, so it’s a long series of rebalances with lag building up between them. For the whole time, old and new versions take turns on the same partitions. If the new version changed how it reads or writes a message, the two versions were now processing the same stream differently, interleaved, with no clean point where one stopped and the other started.

The second was services that keep state in memory: caches warmed from the database, data loaded at start-up, work in progress. A half-updated fleet means two versions sharing that state and disagreeing about it. These services wanted the new version to be complete and warm before it saw a single request, and the old one to stay intact until we were sure.

And for everything else there was a quieter problem. A rolling update has no idea whether the new version is any good. It only knows whether the pods are Ready. By the time error rates show up on a dashboard, half the pods may already be on the new version, and rolling back is another rolling update, in the other direction, while the errors continue.

We fixed all three with Argo Rollouts, using the Istio VirtualServices we already had, and the Mimir metrics backend we’d built for the cluster. Services that needed a clean switch got blue-green: a complete second copy of the service, tested in place, then promoted in one step. Services that could take mixed traffic got a canary: a slice of real traffic, increased step by step, with Argo Rollouts querying Mimir between steps and rolling back on its own when the numbers went wrong.

One chart, two strategies

All our services shipped through the same Helm chart, which I covered in Rebuilding CI/CD with GitHub Actions and Argo CD. Progressive delivery became another option in that chart. A service opts in with two values:

rollout:
  enabled: true
  type: canary # or blueGreen

With rollout.enabled, the chart renders an Argo Rollouts Rollout instead of a Deployment. The pod template is identical: same containers, probes, secrets, topology spread. Only the strategy block changes. Trimmed to the part that matters:

{{- if eq (.Values.rollout).type "canary" }}
strategy:
  canary:
    dynamicStableScale: true
    abortScaleDownDelaySeconds: 300
    stableService: {{ include "chart.fullname" . }}
    canaryService: {{ include "chart.fullname" . }}-rollout
    trafficRouting:
      istio:
        virtualService:
          name: {{ include "chart.fullname" . }}
    steps:
      {{- toYaml .Values.rollout.steps | nindent 6 }}
    {{- toYaml .Values.rollout.canary | nindent 4 }}
{{- end }}
{{- if eq (.Values.rollout).type "blueGreen" }}
strategy:
  blueGreen:
    activeService: {{ include "chart.fullname" . }}
    previewService: {{ include "chart.fullname" . }}-rollout
    {{- toYaml .Values.rollout.blueGreen | nindent 4 }}
{{- end }}

Both strategies use the same two Services. <name> is the one everything already called: the stable service for a canary, the active service for blue-green. <name>-rollout is the new version: the canary service, or the preview service. Argo Rollouts owns their selectors. It adds a rollouts-pod-template-hash to each one, so each Service points at exactly one ReplicaSet, and changes it as the rollout moves.

Because nothing else changed name, callers, dashboards and alerts didn’t have to know a service had moved to Rollouts. The chart also handles autoscaling: when an HPA or KEDA ScaledObject is enabled, it leaves replicas out of the Rollout and points the autoscaler's scaleTargetRef at kind: Rollout instead of Deployment. Argo Rollouts supports the scale subresource, so the HPA works unchanged.

Canary: a slice of real traffic, then more

Istio does the splitting

Without a traffic router, Argo Rollouts can only approximate a canary by changing pod counts: one new pod out of ten is “10%”, and on a three-pod service the smallest step is 33%. With Istio, the split is done by weight in the VirtualService, and 5% means 5% of requests however many pods each side happens to have.

For a canary service, the chart renders the VirtualService with two destinations on the same route, the stable Service at 100 and the canary Service at 0:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: feed
  namespace: apps
spec:
  gateways:
    - istio-system/alb-internal
  hosts:
    - feed.internal.example.com
  http:
    - route:
        - destination:
            host: feed.apps.svc.cluster.local
            port:
              number: 80
          weight: 100
        - destination:
            host: feed-rollout.apps.svc.cluster.local
            port:
              number: 80
          weight: 0

Argo Rollouts edits those two weights at each step and does nothing else to the VirtualService. The hosts, the gateways, any header-based routes in front, retries and timeouts all stay as the chart rendered them.

There’s one thing to know about where those weights apply. A VirtualService only routes the traffic that passes through it. Ours were bound to the ALB ingress gateways, so the canary saw traffic coming in from outside the cluster. A sidecar calling feed.apps.svc.cluster.local directly goes to the feed Service, which only selects stable pods, so internal callers stayed on the old version for the whole canary. For most of our APIs that was what we wanted: the edge traffic was the real user traffic. If you want the canary to see internal calls too, add the mesh gateway and the Service's own hostname to the VirtualService.

Steps and analysis

A service’s values file describes the steps:

rollout:
  enabled: true
  type: canary
  steps:
    - setWeight: 5
    - pause: { duration: 5m }
    - setWeight: 25
    - pause: { duration: 10m }
    - setWeight: 50
    - pause: { duration: 10m }
  canary:
    analysis:
      templates:
        - templateName: istio-http-health
          clusterScope: true
      startingStep: 1
      args:
        - name: service
          value: feed-rollout.apps.svc.cluster.local

That’s 5% of real traffic for five minutes, then 25%, then 50%, and then 100% once the last step finishes. The analysis runs in the background for the whole canary, from the first pause onwards, and checks the canary every minute. If it fails at any point, the rollout aborts and the traffic goes back to the stable pods. How fast that happens depends on the next setting.

Two settings in the chart’s strategy block make canaries cheaper and easier to debug.

dynamicStableScale: true shrinks the stable ReplicaSet as the canary grows. Without it, Argo Rollouts keeps the stable version at full size for the whole canary, so a 50% canary runs at 150% of normal capacity. With it, at 25% the stable side is scaled down to roughly 75% of its pods. For services with dozens of pods, that's the difference between a canary that fits on the existing nodes and one that has Karpenter launching new ones for every release.

The trade-off shows up on abort. Without dynamicStableScale, the weights go back to 100 and 0 in one update, because the stable side was never scaled down. With it, the stable side has to scale back up first, and Argo Rollouts only moves traffic off the canary as fast as stable pods become available. That's usually seconds, since the nodes are still there, but it's not instant.

abortScaleDownDelaySeconds: 300 keeps the canary pods running for five minutes after an abort, with no traffic, so whoever is looking into the failure can still read their logs, exec into them, or pull a heap dump. Without it, dynamicStableScale scales the canary down as the traffic leaves it.

The analysis queries Mimir

The checks live in a ClusterAnalysisTemplate, so every service in every namespace shares the same definition, and a service only passes in its own canary Service name:

apiVersion: argoproj.io/v1alpha1
kind: ClusterAnalysisTemplate
metadata:
  name: istio-http-health
spec:
  args:
    - name: service
  metrics:
    - name: success-rate
      initialDelay: 2m
      interval: 1m
      failureLimit: 2
      successCondition: len(result) == 0 || result[0] >= 0.99
      provider:
        prometheus:
          address: https://mimir.ops.example.com/prometheus
          timeout: 30
          headers:
            - key: X-Scope-OrgID
              value: prod
          query: |
            sum(rate(istio_requests_total{
              reporter="source",
              destination_service="{{args.service}}",
              response_code!~"5.."
            }[2m]))
            /
            sum(rate(istio_requests_total{
              reporter="source",
              destination_service="{{args.service}}"
            }[2m]))
    - name: p99-latency
      initialDelay: 2m
      interval: 1m
      failureLimit: 2
      successCondition: len(result) == 0 || result[0] < 500
      provider:
        prometheus:
          address: https://mimir.ops.example.com/prometheus
          timeout: 30
          headers:
            - key: X-Scope-OrgID
              value: prod
          query: |
            histogram_quantile(0.99, sum by (le) (rate(
              istio_request_duration_milliseconds_bucket{
                reporter="source",
                destination_service="{{args.service}}"
              }[2m]
            )))

There’s no exporter or SDK in the application for this. Every request through the mesh already produces istio_requests_total and istio_request_duration_milliseconds, and since the canary has its own Service, destination_service separates canary traffic from stable traffic with no extra labels. The metrics were already in Mimir, collected by the Alloy agents on every node, as described in Building Observability at Scale.

Argo Rollouts’ Prometheus provider works against Mimir unchanged, because Mimir serves the Prometheus query API under /prometheus. The only Mimir-specific part is the X-Scope-OrgID header that picks the tenant. The prod and non-prod tenants are separate in our setup, so the chart fills in the header from the environment's values when it renders the template; the example above is the prod one.

A few details in that template matter more than they look:

• initialDelay: 2m. Metrics reach Mimir after a scrape interval and a remote write, and a rate() over two minutes needs two minutes of samples to say anything useful. Measuring straight away judges the canary on almost no data.
• failureLimit: 2. One bad minute shouldn't abort a release, because a single slow request at 5% of a quiet service can move p99 a long way. Three failed measurements abort it.
• Empty results. At 5% of a low-traffic service, the canary may get no requests in a two-minute window, and the query returns nothing. Accessing result[0] on an empty result is an error, and after enough consecutive errors the analysis fails. len(result) == 0 treats "no traffic yet" as a pass. Whether that's right for you is worth deciding explicitly. For a service that sees steady traffic, an empty result probably means something is broken, and you want it to fail.
• Absolute thresholds. 99% and 500 ms are fine for most APIs, but some services run at 97% on a good day because of expected 5xx from a downstream. For those, compare the canary with the stable version in the query instead, so the canary only fails if it’s worse than what’s already running.

When we added a new metric to the template, we first ran it as a dry run. A metric listed under dryRun in the Rollout's analysis is measured and recorded, but its failures don't abort anything. After a few weeks of releases we could see what it would have done, and only then let it make the call.

Blue-green: a complete second copy

For the Kafka consumers and the stateful services, a canary was the wrong tool. A canary is mixed versions by design, which is exactly what these services couldn’t handle. They needed the new version to run as a complete, separate copy, be checked as a whole, and then take over in one step.

rollout:
  enabled: true
  type: blueGreen
  blueGreen:
    autoPromotionEnabled: false
    scaleDownDelaySeconds: 600

When the image changes, Argo Rollouts creates the new ReplicaSet, green, at full size next to the running one, blue. The <name>-rollout preview Service points at green, while <name>, the active Service, keeps pointing at blue. No user traffic reaches green yet.

With autoPromotionEnabled: false, the rollout then pauses and waits. Promotion is a single command, kubectl argo rollouts promote <name>, or a button in the Argo Rollouts dashboard. It switches the active Service's selector from blue to green. That's one API update, not a rolling change, so every caller moves at the same moment.

Blue isn’t deleted at promotion. scaleDownDelaySeconds: 600 keeps it running for ten minutes after the switch, so if green misbehaves under real traffic, kubectl argo rollouts undo moves the selector back to pods that are already running and warm. No image pull, no start-up, no cold caches: the rollback takes as long as the selector change.

Testing green before it’s live

A preview Service is only useful if something can reach it. For HTTP services, the chart renders a second VirtualService whenever the rollout type is blueGreen, with its own hostname, routed to the preview Service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: feed-blue-green
  namespace: apps
spec:
  gateways:
    - istio-system/alb-internal
  hosts:
    - feed-blue-green.internal.example.com
  http:
    - route:
        - destination:
            host: feed-rollout.apps.svc.cluster.local
            port:
              number: 80

In staging and preview environments the hostname follows the usual pattern, <name>-blue-green.staging.<domain>. In production each service lists its own hosts in its values. QA and the owning team could run their smoke tests against green, through the same gateway as real traffic and against the same production dependencies, while every user was still on blue. When they were happy, they promoted.

For services that should check themselves before anyone looks at them, the same analysis templates can go in prePromotionAnalysis, which runs once green is fully available. If it fails, the rollout never reaches the pause.

Kafka consumers: a consumer group per version

Kafka consumers don’t take traffic through a Service, so switching a selector doesn’t change what they do. Green pods start consuming the moment they’re running. If green joined the same consumer group as blue, Kafka would split the partitions across both versions, and we’d have the same mix of versions we were trying to avoid.

So green gets its own consumer group. Blue keeps every partition in its group, green gets every partition in its own, and neither group rebalances because of the other. When green is promoted and blue is scaled down, blue’s group simply goes away, without one rebalance per pod along the way.

This has a cost. While both versions are running, both are consuming the same messages. A new consumer group that starts from the latest offsets doesn’t miss anything, because blue is still consuming until the switch, but every message that arrives during the overlap is processed twice, and the overlap lasts until blue is scaled down, which includes the ten-minute scaleDownDelaySeconds after promotion. That only works if the handlers are idempotent, or if whatever they write can take a duplicate. If processing a message twice charges a customer twice, blue-green with two groups isn't the right tool, and you need a different cut-over: pausing blue before green starts, for example, and accepting the gap.

It also means the consumer group ID has to change between versions, and old groups have to be cleaned up. A group that no longer has members still holds committed offsets, and its lag keeps growing in the exporter’s metrics, which is a good way to page someone at 3 a.m. for a consumer that was retired on purpose.

What blue-green costs

For the duration of a rollout, a blue-green service runs twice. For a service with 40 pods, that’s 80 pods until blue is scaled down, and on a cluster that runs at the edge of its capacity, Karpenter launching nodes for the green copy. previewReplicaCount lets green start smaller and scale to full size only at promotion, but then what you tested isn't quite what takes the traffic. For the services that used blue-green, the extra capacity for an hour or two was worth it.

Rollbacks and GitOps

Argo CD deploys everything from Git, and it deploys a Rollout the same way it deploys a Deployment: the new image tag lands in Git, Argo CD applies the Rollout, and Argo Rollouts takes it from there.

An aborted rollout is the place where those two meet. When the analysis fails, Argo Rollouts puts all the traffic back on the stable version, and marks the Rollout as Degraded. Git still says the new version is the one that should be running. Nothing breaks: the Rollout stays aborted on the stable version, and it won't try the same revision again by itself. But the cluster no longer matches Git, and Argo CD shows it. The fix is the normal GitOps one: revert the version in Git, or ship a fixed version. The automatic rollback buys time, and the revert makes it permanent.

The VirtualService is the other place they meet. During a canary, Argo Rollouts changes the route weights, and Argo CD sees a VirtualService that doesn’t match Git. With self-heal on, Argo CD would put the weights back to 100 and 0 in the middle of a rollout. Tell Argo CD to ignore those fields with ignoreDifferences on the VirtualService's route weights, so the two controllers don't fight over the same field.

What I’d tell you before you start

Use the strategy that matches the service, not one for everything. A canary needs a service that can run two versions side by side and enough traffic to judge the new one. Blue-green needs a service that can run twice and a person, or a check, to decide when to switch. Plain rolling updates are still fine for most things.

Let Istio do the traffic split. Pod-count canaries are coarse, and they tie the share of traffic to how many pods you run. VirtualService weights make 5% mean 5%.

Know which traffic goes through the VirtualService. Weights only apply to requests routed by it. If it’s only bound to the ingress gateways, internal callers never see the canary.

Use metrics you already have. Istio’s request metrics are enough to judge most HTTP canaries, and pointing Argo Rollouts at Mimir is one URL and a tenant header.

Wait before measuring, and tolerate a bad minute. Metrics arrive late and low traffic is noisy. An analysis that aborts on the first data point aborts good releases, and teams then turn it off.

Decide what “no data” means. An empty query result is either “no traffic yet” or “something is badly broken”, and the template should say which.

For Kafka consumers, a new version means a new consumer group, and that means duplicates during the overlap. Make the handlers idempotent first, then switch them to blue-green.

And keep the failed version around for a few minutes. An automatic rollback that deletes the evidence leaves you knowing that something broke, but not what.

EKS at Scale

1. Building a Zone-Aware EKS Cluster with Karpenter and Istio
2. Going 100% Spot on EKS with Karpenter
3. Running ARM and x86 Side by Side on EKS with Karpenter
4. Reliability at Scale: Sizing, Autoscaling and Circuit Breaking with Istio
5. Canary and Blue-Green Deployments with Argo Rollouts, Istio and Mimir (you are here)

Part 5 of the Enterprise Azure Landing Zones series.

We spent most of this series taking doors away. Policy denies public IPs, every PaaS service sits behind a private endpoint, egress goes through a firewall, and the data platform from Part 4 has no public surface at all. That’s the goal. It also means the people who build and use the platform can’t reach it from their laptops. This part covers the few doors we put back on purpose, who each one is for, and how we control what people can do once they’re inside.

The problem with a fully private estate

In a conventional estate, access is restricted after the fact: a public endpoint plus a firewall rule for the office range. In a private estate the default flips. Nothing is reachable until you build a path to it, and that forces three questions early.

• Who needs in? A platform engineer debugging a private cluster needs a network path. A data analyst needs a working environment with the right tools. An engineer who has to look at a production database for ten minutes needs exactly that, and nothing more.
• Where does the data end up? A network path to a private service also lets data flow down it to a laptop. For people working with business data, that undoes the reason the platform is private.
• How do we keep it auditable? Every way in should use Entra ID sign-in, the same groups as everything else (their roles granted as code, their membership managed in Entra ID, as in Part 2), and logs that land with the rest of the platform’s logs.

One mechanism can’t answer all three well, so we didn’t try. Our answer is three doors, each designed for a different group.

A conventional estate is open until a rule closes it; ours is closed until we build a door.

VPN for platform teams

The first door is the one we needed on day one, because the platform team itself had to get in. Platform engineers work on the network, the clusters, the vaults and the pipelines. They need packets, not a remote screen: their own tools, their own terminal, talking directly to private APIs.

One gateway, on the control-plane hub. The control-plane ALZ is the transit point of the estate’s star topology, so that’s where the point-to-site VPN lives. One gateway and one connection reach every regional zone, and an engineer doesn’t juggle a tunnel per region. A regional hub only gets its own gateway when there’s a local reason for one. The trade-off is a shared dependency: if the control-plane gateway is down, remote access is down everywhere. We accepted that: one well-run gateway beats several to patch and monitor.

Identity instead of certificates. Sign-in uses Entra ID. There are no client certificates to issue, rotate or revoke: an engineer signs in with their normal identity, conditional access applies as it would to any other app, and removing someone from the group in Entra ID removes their access. Offboarding happens in the identity system, where it already happens for everything else, not in a certificate revocation list.

One inspected path. VPN traffic doesn’t take a shortcut into a region. It goes through the control-plane firewall and then the regional firewall, and the rule for VPN clients is narrow on purpose. Anything beyond it needs its own rule and its own review. That costs a little latency and makes every remote session visible in the same logs as everything else.

DNS that follows the network. DNS is per ALZ, so a private endpoint in APAC registers its name in APAC’s private DNS. VPN clients resolve through the control-plane hub, so the records platform engineers need are copied into the control-plane hub’s private DNS. Part 3 covers the address plan, the routing through the transit and why the records are copied, including the cost of keeping a second copy in sync.

One gateway on the transit hub reaches every zone, through two firewalls, with access decided in Entra ID.

A platform engineer connects once and reaches a private resource in any zone, through one inspected, logged path. That’s the right tool for the people who run the platform, and the wrong one for everyone else.

Azure Virtual Desktop for data and business teams

A VPN gives a laptop a route into the network. For data and business teams that’s the wrong shape, for two reasons.

The first is where data ends up. A query result, an export or a report cache lands on the device as soon as it’s downloaded, and that device is one we don’t control. The second is setup. Every analyst would need drivers, the data platform’s tooling, Git, an IDE and a BI client, all at versions that work together, on whatever laptop they happen to have.

So we built a different door: Azure Virtual Desktop, with ready-built desktops that sit inside the private network next to the data platform. An analyst signs in with Entra ID from the desktop app or a browser and gets a desktop that already has their tools and already resolves every private endpoint. The data stays in the estate; what reaches the user’s device is pixels, not files. Clipboard and drive redirection become choices we make per group of users, not whatever a laptop allows.

AVD also fits the private model without bending it. Session hosts take no inbound connections from the internet; they connect out to the service, so there’s nothing to expose and no jump box to harden. Their traffic goes through the regional firewall like any other workload’s.

How the design hangs together

AVD is a workload on the foundation like any other. Its subscription is vended through the same process as everything else from Part 4, with its network, peering, DNS and routing already in place. The AVD layer only adds what belongs to it, and a few decisions shape it.

• Pools per persona. Desktops are grouped by the kind of person using them: data engineers and analysts, developers, general business users. Each persona gets its own pool with its own sizing, session limits and settings, because an analyst running notebooks and someone reviewing a dashboard need different machines. Access is one Entra ID group per pool: the group’s assignment to the pool is code, its membership is managed in Entra ID, and that group is the whole access model.
• Per region. Desktops live in the same zone as the data they work on. That keeps data inside its region, which matters for residency, and keeps the desktop close to the platform it talks to. Each zone gets its own set of pools and its own feed.
• Scale to zero. Pools follow a working-day schedule in the zone’s own time zone. Outside it, capacity drops to zero, and the first person to connect in the morning starts a host. Idle desktops cost nothing overnight, at the price of a short wait for the first login.
• Profiles kept separately. Pooled desktops mean a user lands on a different machine each time, so their profile lives on private storage and follows them. Hosts become disposable: we can replace or reimage them without anyone losing their work.
• A golden image. Desktops are built from one image with the tooling preinstalled, produced by its own pipeline and published centrally from the control plane. Each zone pins an exact version, so a new image rolls out as a deliberate change in review, never as a side effect of a rebuild.

Desktops sit in the same zone as the data; only pixels leave.

AVD is still in progress. It’s live in the zone that hosts the data platform, with a pool for the data engineers and analysts who work with it. The other regions are scaffolded with their persona pools, waiting for their subscriptions to be vended. Adding a zone means reserving its address space, vending a subscription and applying a stack that’s already written.

Teleport for privileged access

VPN and AVD answer “how do I reach the estate?” Neither fully answers the harder question: how much should I be able to do once I’m there, and for how long?

Entra ID and Azure RBAC govern what engineers can do to Azure resources. They say less about what happens inside them: a shell on a virtual machine, a session inside a production container, a query against a database. Before Teleport, those relied on standing permissions, and the audit trail was spread across several logs.

So we added Teleport as a privileged access management layer across every resource type.

• Just-in-time access. Nobody holds standing access to production servers, clusters or databases. An engineer requests access to a specific resource, for a reason and for a limited time, and a reviewer approves it.
• Short-lived credentials. Access is granted as a certificate that expires in hours, issued after Entra ID sign-in. There are no long-lived keys or passwords on laptops to leak or forget to revoke.
• Per-resource requests. Requests target one server, cluster or database rather than a whole zone or network range, which narrows the blast radius of any one approval.
• Recorded sessions and one audit trail. Sessions are recorded, and every request and approval is logged in one place, so “who ran what against production, and who approved it” has one answer.

The Teleport flow: one resource, one reason, a few hours, and a recording.

It also changed what the VPN is for. Before, it was both the network path and the gate. With Teleport in front of servers, clusters and databases, a privileged action needs an approved request however someone reached the network, and the VPN went back to being plumbing.

Choosing the right door

Private by default only works if the ways back in are as deliberate as everything else. Each door serves a different group, and none of them puts a public endpoint back on the estate.

Three doors, each for a different group; none puts a public endpoint back.

What they share is identity from Entra ID, access through Entra ID groups whose permissions are granted as code, and every path built through the same reviewed pipeline as the rest of the platform.

What comes next

Every door in this part leads into the cloud. The last part deals with the case where that isn’t enough: data that isn’t allowed to go to the cloud at all, and how we extended the same landing zone design to a sovereign site instead of building a second platform.

Enterprise Azure Landing Zones

1. Azure Landing Zones: Why We Needed Them, and the Architecture We Built
2. Azure Landing Zones: Governance, and How the Platform Is Built
3. Azure Landing Zones: Designing the Network for a Private, Multi-Region Estate
4. Azure Landing Zones: Subscription Vending as a Platform Service
5. Azure Landing Zones: Reaching a Private Estate with VPN, Virtual Desktops and Teleport (you are here)
6. Azure Landing Zones: Going Hybrid When Data Can’t Leave (coming soon)

Part 4 of the Enterprise Azure Landing Zones series.

Everything so far has been foundation. Part 1 laid out regional landing zones provisioned from a control plane, Part 2 the governance tree, policy and bootstrap, and Part 3 the address plan that keeps every network in the estate from colliding. Foundation is necessary, but on its own it doesn’t help the engineer who just needs somewhere governed to build.

Subscription vending is what turns that architecture into something a team can consume: a way to ask for a governed, networked, policy-compliant subscription and get one, without anyone hand-crafting it in the portal.

What vending produces

Part 1 split every landing zone into two halves: the platform landing zones (hub, firewall, DNS, identity, management) that the platform team runs, and the application landing zones where workloads live. A vended subscription is an application landing zone. Vending is the machine that makes them.

The unit is one workload, one environment. A workload that runs in dev, test and production gets three subscriptions, each vended on its own, all in the same regional ALZ. The environment boundary sits at the subscription, which is what lets production carry its own budget, its own access and its own network while living under exactly the same rules as dev.

And a vended subscription isn’t an empty one. By the time it reaches a team it has already been:

• Placed in the correct regional ALZ management group, so it inherits that zone’s policy the moment it exists: allowed regions, required tags, no public IPs, all of it.
• Networked: a spoke network is created, addressed from the regional pool in Part 3, peered into the region’s hub and routed through its firewall, so nothing overlaps and nothing bypasses inspection.
• Governed and accounted for, with access, budgets and tags in place from the first minute.

Because the whole platform is built from Azure Verified Modules wrapped in Terragrunt, vending is too. It’s the AVM subscription-vending pattern, wrapped exactly like the landing zones themselves, so a request is ultimately a small set of inputs describing the subscription a team wants.

Two front doors, one governed path

Here the landing zone starts to work as a product. Its users are the workload teams, and they consume it through self-service rather than a ticket queue. Vending is the first service on that platform, Backstage is its portal, and the blueprints later in this part are its golden paths.

A team can ask for a subscription in one of two ways, and both end at the same reviewed pipeline:

• Through Backstage, the self-service portal from Part 2. A team fills in a form describing what it needs, and the request becomes a proposed change in the platform repository.
• By opening a pull request against that repository directly, with the request written as configuration.

Either door is open to the platform team and to workload teams. Self-service isn’t gated behind the platform team as a bottleneck. But it isn’t a free-for-all either: the platform team reviews every request and applies it. The form or the pull request is a proposal; a platform engineer checks it and merges it. That review is what keeps self-service and governance from pulling against each other. Teams move fast because they can ask directly, and the estate stays coherent because nothing lands without a check.

One decision from Part 3 sits inside this: the network belongs to vending, not to the workload team. A team never creates its own VNet; vending creates it, addresses it from IPAM and peers it into the hub, and the team only adds subnets. Subscriptions are handed out widely; networking stays a platform responsibility.

Two front doors, one governed path: request via Backstage or PR, reviewed and applied, out comes a fully governed subscription.

Who owns what

Handing teams their own subscriptions only works with a clear RACI between the platform team and the workload teams:

• The platform team owns the foundations: the landing zones, the management-group hierarchy, the policy baseline, the network topology and the vending framework itself.
• Workload teams own their subscriptions. Inside the slice they’re vended, they build and run what they need, under management-group policies they inherit and can’t switch off.

The platform team isn’t in the critical path of every deployment, and because the guardrails come from the structure, a team can move freely without being able to move out of compliance.

Beyond the subscription: golden-path blueprints

Vending hands a team a governed, networked subscription with nothing running in it yet. The next question is “now how do I build the thing I came for?”

We maintain a set of hardened, ready-to-use blueprints for the platforms teams keep asking for. A team that wants a data platform or a Kubernetes cluster doesn’t start from a blank file and rediscover every security and well-architected decision. It reuses a blueprint that is already hardened and already aligned with the same policy posture as the rest of the estate, and stands it up with about one click from the same self-service surface. It’s the infrastructure counterpart of the service blueprints application teams use.

Golden paths: hardened, WAF-aligned blueprints (Databricks, AKS) provisioned into a vended subscription in ~one click.

The data platform shows what a blueprint looks like when it’s designed for this foundation rather than dropped onto it.

A private data platform

The data teams wanted a governed data and AI platform per environment with nothing reachable from the internet. The blueprint combines Databricks, a data lake and a Key Vault, and none of them has a public endpoint: the workspace, the storage and the vault are reachable only through private endpoints, and cluster nodes get no public IPs.

The key design choice is what the blueprint doesn’t own. It never creates a network. It builds inside the network vending already made, and only adds the subnets it needs, carved from that subscription’s address range. Routing, peering and DNS are inherited, so egress goes through the hub firewall from the very first deployment and only the traffic the platform explicitly allows ever leaves.

Dev, test and production are three environments from one definition: three vended subscriptions over the same blueprint, with production differing only where it should, such as storage redundancy.

That leaves the obvious question: with no public endpoints, how does a data engineer open a notebook? Not from a laptop. Data teams work from virtual desktops inside the estate, which is the subject of Part 5.

Two tools, two layers: Terraform for the platform, Crossplane for projects

Blueprints cover the big products. Most requests are smaller: an application needs a database, a storage account, a Key Vault, an AI service or an app registration. Routing every one of those through the platform team’s Terraform would make us the bottleneck vending was meant to remove, so we drew a line between two tools:

• Terraform and Terragrunt build the platform: the landing zones, vending, the address plan and the big blueprints such as the data platform and AKS clusters. These are shared, change slowly, and every change is a reviewed plan applied by the platform team.
• Crossplane provisions what belongs to a project: the infrastructure one application needs. Teams declare it in Git next to the application itself, Argo CD hands it to Crossplane, and Crossplane creates it in Azure and keeps reconciling it.

The rule we gave teams fits in one line: if it’s shared or outlives a project, it’s Terraform; if it belongs to one app, it’s Crossplane.

A few design choices make it safe to hand out:

• One control plane. Crossplane runs once, on the hub cluster in the control plane’s DevX subscription (Part 2), not on every cluster. Provisioning stays where the rest of the provisioning lives.
• Scoped to the team’s landing zone. A team’s resources are bound to its own vended subscription, so a project can only create inside its own application landing zone, and the region’s policies still apply: naming, Key Vault rules, no public endpoints. Crossplane doesn’t get a way around the guardrails; it goes through them.
• No secrets in Git. Connection details land in Key Vault and reach the application through External Secrets.
• Onboarding is a boundary, not a ticket. A team gets a tenant in Argo CD that defines where it can deploy from and to, and owns its folder in the repository.

What it buys: teams provision in the same pull request as their application, without Terraform skills or state, drift is corrected continuously instead of at the next plan, and the platform team reviews a team’s boundary once instead of every database it asks for. What it costs: two infrastructure-as-code tools to run and teach, a hub cluster carrying many provider types (we install only the ones we use), and debugging that moves from plan output to Kubernetes events.

Terraform builds the platform and the big blueprints; Crossplane gives each app its own infrastructure, inside its own landing zone.

Today teams still declare individual resources. The next step is compositions: an intent-level API where a team asks for “a database” and the platform decides what that means. That same abstraction is what lets one request be fulfilled in the cloud or on-premises, which Part 6 picks up.

Closing the loop

Vending is what the earlier parts were for. A team asks for what it needs and gets an application landing zone in the right region, already governed, addressed, peered and routed. The blueprints and Crossplane then cover what it builds on top, inside the same guardrails.

There’s one catch to an estate this private: if nothing has a public endpoint, people need a way in. Part 5 covers how we designed that, with VPN for platform engineers, virtual desktops for data teams, and Teleport next.

Enterprise Azure Landing Zones

1. Azure Landing Zones: Why We Needed Them, and the Architecture We Built
2. Azure Landing Zones: Governance, and How the Platform Is Built
3. Azure Landing Zones: Designing the Network for a Private, Multi-Region Estate
4. Azure Landing Zones: Subscription Vending as a Platform Service (you are here)
5. Azure Landing Zones: Reaching a Private Estate with VPN, Virtual Desktops and Teleport
6. Azure Landing Zones: Going Hybrid When Data Can’t Leave (coming soon)

Part 3 of the Enterprise Azure Landing Zones series.

Part 1 laid out the estate: four self-contained regional landing zones (Africa, Europe, LATAM and APAC) and a control-plane landing zone that provisions them. Part 2 covered how that estate is governed and built. This part is about the network underneath it: how addresses are planned, who owns which piece, how traffic moves between regions, and how names resolve when nothing has a public address.

The goals

Three goals shaped every network decision that followed:

• Private by default. Workloads have no public IPs. Databases, storage, Key Vaults and cluster APIs are reached through private endpoints. If something has to face the internet, it does so through a deliberate, reviewed entry point, never by accident.
• Nothing overlaps, ever. Azure refuses to peer two VNets with overlapping ranges, and you usually discover the overlap the day you need the peering. By then the fix is a rebuild.
• Every path is chosen. Traffic between workloads, between regions and out to the internet goes through a firewall we control, along a route we wrote down. No path exists just because the platform happened to allow it.

Three address ranges that never mix

The first decision was to split the address space by purpose, with one range per job and its own rules:

• A range for VNets. Every hub and every spoke takes its addresses from one large private block, and it is the only range routed across the whole estate. Each zone gets its own contiguous slice, with the hub at the start and the rest left for spokes that haven’t been asked for yet. Generous on paper, but regions grow in ways nobody predicts.
• A range for remote access. VPN client pools live in a separate block, with a slice per zone. It is routed into hubs only, so a person on the VPN can never collide with a workload, and distinct pools per hub mean two open tunnels never produce two identical routes.
• A range for the Kubernetes overlay. Pod and service addresses come from the shared address space reserved for this kind of use. They are never routed outside a cluster and every cluster reuses the same ranges, because overlay addresses only need to be unique inside one cluster. Pods leave through their node’s address, which comes from the VNet range.

Splitting by purpose also makes logs readable. In a firewall log, an address from the remote-access range is a person, an address from the VNet range is a workload, and an overlay address means something has gone wrong. Every route and rule becomes easier to reason about.

Keeping the overlay out of the routed space saves a large block per cluster. The trade-off is that it’s fixed for the cluster’s life, since a cluster can’t change its pod ranges in place. We accept that and replace clusters rather than re-address them.

Three ranges, one purpose each: VNets, remote access, and the cluster overlay.

IPAM hands out every range

A spreadsheet can’t stop two people picking the same range on the same afternoon. We wanted allocation done by something that refuses to hand out a range twice, so we used the IPAM built into Azure Virtual Network Manager: a pool per range, a pool per zone beneath it, and a reservation for every network. Hubs get fixed reservations that match the plan; spokes ask for “a block of this size from Europe” and IPAM picks a free one. Nobody types an address. Every consumer looks its range up by name.

IPAM runs as its own stack, applied after the platform foundations and before vending. That ordering matters: by the time vending creates a network, its range is already a known value rather than something that will only exist after creation.

One rule we learned the hard way: treat the top-level pools as permanent. Changing one forces it to be recreated along with everything under it. New top-level ranges are added beside the old ones, never into them.

Vending owns the network, workloads own subnets

An address plan only holds if nobody can get around it, so we took the network away from workload teams. When the platform vends a subscription (the subject of the next part), it also creates and owns everything that connects it to the estate:

• The VNet, with its range taken from IPAM.
• The peering to the zone’s hub.
• The DNS settings, pointing at the hub’s resolver. Peering doesn’t carry DNS settings across, and a spoke that forgets them quietly resolves private endpoints to public addresses.
• A default route sending all traffic to the hub firewall, plus a policy that attaches it to any subnet created without one.

The workload gets the VNet and creates only its subnets inside it, each with its network security group from the start. A workload that declares its own VNet, peering or route table is treated as drift and moved into vending.

The cost is friction: a team that needs more space has to ask for another reservation. In return, every network in the estate is addressed from IPAM, attached to the right hub and forced through a firewall from its first subnet, and nobody has to remember to make it so. Because subnets can’t grow once something is deployed in them, our sizing rule is simple: size for the next two or three subnets, not just today’s, and round up when in doubt. Spare space costs nothing; a full subnet costs a rebuild.

Vending owns the network; a workload only adds subnets inside it.

Hub and spoke in every region

Each regional landing zone is a classic hub and spoke, and it is complete on its own. The hub holds the zone’s firewall, its private DNS zones and resolver, and whatever gateways that region needs. Spokes are the vended workload VNets, peered to their hub and nothing else.

Inside a zone, routing is deliberately boring. Every spoke sends everything to its hub firewall, including traffic to other spokes in the same zone. Two workloads that share a region still talk through an inspection point, so spoke-to-spoke access is a firewall rule someone approved rather than a side effect of peering.

Self-contained regions are also a resilience choice: a region keeps working when another one doesn’t.

A star through the control-plane hub

The regions still need to be connected. As Part 1 described, the control plane’s own hub is the transit point: each regional hub peers with it and with nothing else, so Africa never peers directly with APAC.

We chose that star over a mesh for three reasons:

• One way in for people. Everything is private, so engineers need a network path into it. A point-to-site VPN on the control-plane hub gives them one tunnel that reaches every zone. A region only gets a gateway of its own when there’s a local reason for one, not by default. How people get in is the subject of Part 5.
• The platform reaches every region. The runners and tooling that provision the regional zones live in the control plane. The star gives them one path into each region rather than a web of peerings to maintain.
• One checkpoint for cross-zone traffic. Peering isn’t transitive, so traffic can’t simply hop from one regional hub through another. We route it through the control-plane firewall on purpose, so every cross-zone flow is inspected and logged in one place.

The cost is that cross-zone routing has to be engineered rather than assumed. Routes on the control-plane hub send each zone’s block to that zone’s firewall. The control-plane firewall translates cross-zone traffic to its own address, because the remote zone only knows the way back to the control-plane hub; an untranslated packet from a VPN client would have no return path. And because each zone is one contiguous block, all of this is defined once per zone. A new spoke in APAC lands inside APAC’s block, and every route and rule already covers it.

A cross-zone path: through the control-plane firewall, across one peering, into the regional hub, with names resolved in the control plane.

DNS stays in each zone, with copies where they’re needed

Private endpoints only work if their names resolve to private addresses. The obvious answer is a single central set of private DNS zones for the whole estate. We didn’t do that. Each landing zone has its own private DNS zones and its own resolver, and a private endpoint registers its record in its own zone. That keeps each region self-contained, and a DNS mistake in one region stays there.

The exception is the control plane. VPN clients and control-plane tooling resolve through the control-plane hub’s resolver, which knows nothing about Europe’s or APAC’s private endpoints. So for the regional private endpoints that engineers and runners need, we copy the records into the control-plane hub’s private DNS. The name resolves to the regional private address, and because the control-plane hub is also the transit point, the traffic then follows the routes above through the firewall into the right region.

That choice has costs:

• Copies can go stale. A copy is a second source of truth. If a private endpoint is recreated with a new address, the copy has to follow. That’s why the copies are generated from the source zone automatically and never edited by hand.
• Other regions need their own copies. A workload in Europe that has to reach a private endpoint in APAC can’t see APAC’s records either. Each cross-region dependency is a record copied on purpose, which also makes those dependencies visible.
• Zone-wide forwarding doesn’t help. Forwarding a whole private-link zone to a region’s resolver sounds simpler, but every landing zone uses the same zone names, so one forwarding rule can only ever point at one region.

Each zone owns its names; the control plane and other zones get generated copies of only the records they need.

Egress and inspection

The last piece is the way out. Policy denies public IPs on workload networks, so a workload can’t open its own door to the internet. All outbound traffic follows the default route to its zone’s hub firewall, which decides what is allowed and logs what happened. Traffic from the control plane into a region is inspected twice, once on each side.

No team configures its own egress, and reaching an external service is an explicit rule with an owner. We pay for that with a firewall in every region and the rule reviews that come with it, and in return we know what leaves the estate.

What we’d do again

The decisions that held up best were the ones that took a choice away from someone:

• Separate ranges by purpose. VNets, remote access and the overlay never borrow from each other, and a log line tells you who sent it.
• Make IPAM a service, not a spreadsheet. Its own stack, applied before anything needs an address, and the only place ranges come from.
• Let vending own the network. Workloads create subnets and nothing else.
• Route by zone block. New spokes are reachable without anyone touching routing.
• Keep DNS in each zone. Copy only what crosses a boundary, and automate the copies.

Next in the series is subscription vending itself: how a team asks for a subscription and gets one that is already governed, addressed, peered and routed.

Enterprise Azure Landing Zones

1. Azure Landing Zones: Why We Needed Them, and the Architecture We Built
2. Azure Landing Zones: Governance, and How the Platform Is Built
3. Azure Landing Zones: Designing the Network for a Private, Multi-Region Estate (you are here)
4. Azure Landing Zones: Subscription Vending as a Platform Service
5. Azure Landing Zones: Reaching a Private Estate with VPN, Virtual Desktops and Teleport
6. Azure Landing Zones: Going Hybrid When Data Can’t Leave (coming soon)

Part 2 of the Enterprise Azure Landing Zones series.

In Part 1 I sketched the architecture: self-contained regional ALZs for Africa, Europe, LATAM and APAC, plus a control-plane ALZ that provisions them. This part covers two things. First, governance: how the management-group tree is shaped, what has to exist before anything else can, and which guardrails we relied on. Second, how the platform is actually built: the modules, the delivery model and the access model that keep five zones identical without anyone building them by hand.

The management-group hierarchy

We followed the standard Azure Landing Zone management-group tree rather than inventing our own. That’s the point of adopting CAF, and AVM’s ALZ pattern module builds that tree for you. The interesting part is the top level, where the multi-region design changes how the standard tree is arranged.

Directly under the tenant root management group sit a handful of top-level groups:

• A bootstrap management group, home to the subscription that holds the platform’s Terraform state and the identities the deployment pipelines use.
• One management group per regional ALZ (Africa, Europe, LATAM, APAC), each the root of a full landing-zone subtree following the standard ALZ pattern: Platform (Management, Connectivity, Identity), Landing Zones, Sandbox, Decommissioned.
• A management group for the control-plane ALZ, which provisions the regional zones and hosts the platform tooling. It sits beside the regions, not above them. No region inherits anything from it; each regional subtree has its own policy, security and monitoring.

So rather than a single intermediate root with one landing-zone tree beneath it, we have several peer ALZ subtrees, one per region plus the control plane, each internally identical to the canonical ALZ hierarchy, with bootstrap set off to the side. Each region gets its own policy scope and blast radius, and every subtree stays recognisably the same shape. That sameness doesn’t come from inheritance. It comes from deploying every zone from the same code, which is the second half of this post.

The governance tree: bootstrap and control-plane beside per-region ALZ MGs; each region repeats the standard ALZ subtree.

Bootstrap: state and identities, not deployment

Everything here is created by pipelines, and those pipelines need two things before any landing zone exists: somewhere to keep Terraform state, and an identity permitted to deploy. That’s the chicken-and-egg problem of every landing-zone build. The foundation the automation depends on can’t be produced by that automation.

Bootstrap is our answer, and its name is misleading. Bootstrap is not a deployer. It doesn’t build the regions or the control plane. It’s a single subscription, isolated under its own management group, that holds exactly two things:

• The Terraform state for the platform, the backend every deployment reads from and writes to.
• The identities the pipelines assume to deploy each ALZ and each workload. They use federated credentials, so nothing secret is stored, and each is scoped to what its pipeline may touch.

The deploying itself is done by the pipelines, authenticating as those identities. Bootstrap holds the keys and the state; the pipelines do the work. Keeping it in its own subscription and management group contains its blast radius and separates the sensitive core from the zones it underpins.

That leaves a smaller chicken-and-egg: where do the pipelines run before the platform’s own runners exist? The first pass ran on GitHub-hosted (public) runners, using the same federated identities, because at that point there was nothing private to reach. Once the control plane was up, we moved to self-hosted runners inside the platform. That closes the loop, makes private targets with no public exposure reachable, and lets us enforce that all platform automation runs on private, controlled compute.

A deployment authenticates through a federated identity, uses state held in bootstrap, and applies to the target zone. Bootstrap deploys nothing.

Policy: the guardrails that come for free

On governance, we deliberately leaned on the default ALZ policy set rather than writing a large body of custom policy. The CAF/enterprise-scale archetypes that ship with the ALZ module encode years of Microsoft’s own hardening, and re-deriving that in-house would have been effort spent rebuilding something that already works.

On top of the defaults, the guardrails that mattered most for this estate were the obvious, high-leverage ones:

• Allowed regions: subscriptions can only deploy into the regions we sanction, which keeps workloads inside the right residency and connectivity boundaries for their zone.
• Required tagging: resources carry the tags the group needs for ownership, cost allocation and lifecycle, enforced at creation rather than chased down later.
• Deny public IP: no workload gets a public IP by default. Egress and ingress go through the hub and its Azure Firewall, so “accidentally internet-facing” stops being a failure mode.

Because these are inherited through each region’s management-group subtree, a newly vended subscription is compliant the instant it exists. Nobody applies the guardrails to it; it gets them from where it sits in the tree.

Built from verified modules

The whole platform is Terraform. For an estate under this much governance, code isn’t a preference: it’s repeatable, reviewable and auditable. When compliance asks whether every subscription enforces a control, the answer is in version control.

The design has three layers, each solving a different problem:

• Azure Verified Modules are the base. They’re Microsoft’s maintained library of infrastructure modules, secure by default (private endpoints, RBAC, diagnostics and zone redundancy switched on rather than optional) and aligned with CAF. Security fixes and new baselines arrive as module upgrades we consume, not code we write.
• Our own wrapper module sits on top. It composes the AVM landing-zone modules and exposes only what differs between regions: the region, its address ranges, its policy parameters, its connectivity settings. A region is described entirely by its inputs. Nobody edits module internals to stand up APAC; they supply APAC’s values. Our architectural decisions live in this layer.
• Terragrunt instantiates that wrapper once per zone. Plain Terraform would still leave us copy-pasting near-identical root configurations for every zone, each with its own backend, state and provider wiring. That duplication is where drift is born: someone updates three copies and forgets the fourth. With Terragrunt the wrapper is written once, and each zone is just a set of values.

That gives the estate the property it depends on: Africa, Europe, LATAM and APAC are the same code with different values. Environments don’t multiply the zones. Every ALZ is production infrastructure, and dev, test and production are application landing zones vended inside it, one subscription each (Part 4 covers how). When the baseline changes, it changes in one place and rolls out everywhere.

How the pieces stack: secure AVM modules, our regionalising wrapper, Terragrunt, and every zone.

Every change is a reviewed plan

Shared code raises the stakes of every change, so the delivery model is built around one idea: a human approves exactly what will happen, and then exactly that happens.

The repository isn’t one deployment. Each zone, each cluster and each platform service is its own unit, with its own plan, its own state and its own blast radius. A change to Europe plans Europe and nothing else, and an apply that goes wrong there can’t touch a cluster in APAC.

On a pull request, every affected unit is validated and planned against real state, and the plan lands in the pull request for review. Two checks read that same plan: one prices it, so the reviewer sees the monthly cost delta before approving, and one checks every resource name against the convention. The cost check is advisory on purpose; we wanted the approver to see the number, not a pipeline rejecting a change for costing money. Most changes cost nothing, and the ones that do are rarely obvious from the diff.

On merge, the unit is planned again, the naming gate runs before any approval is requested, an approver signs off, and the pipeline applies exactly the saved plan without re-planning. If nothing changed, nobody is asked to approve anything. Destroying something is only ever a deliberate, manual request through the same gate.

We also split the estate into two delivery modes. Foundation stacks (the zones, vending, DNS, identity, platform tooling) plan on pull request and apply on merge automatically, because they’re few and change slowly. Workload stacks such as clusters and data platforms are deliberately manual: an operator chooses when they roll. They have ordering constraints across stacks and are expensive to get wrong, so we wanted a human to decide the timing.

Plan on the pull request; apply exactly the reviewed plan on merge.

Naming as code

Most Azure resources can’t be renamed, and in an estate this size a resource name is often the only context an on-call engineer has: which zone, which workload, which environment, which region. So naming is one of the few decisions where mistakes have to be caught before the first apply, and a wiki page wasn’t going to do it.

The convention is written once, as a single rule set, and enforced at three points. When code is written, a naming module generates names so nobody builds them by hand. Before apply, the plan gate checks every planned resource, including the ones AVM modules name themselves. After the fact, a weekly audit scans the whole estate, catching anything created in the portal or outside the pipelines, and keeps a single backlog of what doesn’t conform. None of it depends on a reviewer noticing a misplaced token.

Access as code

Access is how the boundary between platform and workload teams becomes real, so it’s managed like everything else. People never get roles; groups do. A grant to an individual is invisible debt that outlives the reason it was made. Terraform owns what each group may do, so “who can change the Europe data subscription?” has an answer in a file, and granting access is a reviewed diff. Entra owns who is in each group, through its own access reviews. We split it that way on purpose: permissions change rarely and deserve review, membership changes constantly and belongs to the identity system.

Pipelines follow the same rule. Every deployable unit has its own identity, scoped to its own state and the roles its stack needs, and federated with the pipeline so there are no secrets to store or rotate. Git becomes the audit trail: when a group got a role, who asked and who approved is one file’s history.

What runs in the control plane

The control plane is a landing zone like the others. Most of what it runs sits in a developer-experience (DevX) subscription that hosts the platform tooling the rest of the group builds on:

• GitHub self-hosted runners: the private compute that runs all platform and workload automation.
• Argo CD: GitOps continuous delivery, reconciling desired state from Git into the clusters and environments it manages.
• Backstage: the developer portal, where teams discover services, request environments and see what the platform offers.
• Crossplane: a Kubernetes-style control plane for provisioning, so infrastructure can be requested and reconciled through the same API surface teams already use.
• Assorted other platform tools that support the above.

Argo CD and Crossplane run on one AKS cluster in the DevX subscription, the hub cluster. The name comes from Argo CD’s hub-and-spoke model: the hub cluster holds the delivery and provisioning controllers, and the workload clusters in the regional zones are the ones it manages. It has nothing to do with the hub networks in each zone. Later parts come back to it: Crossplane on the hub cluster is how app teams provision their own infrastructure (Part 4), and the reason on-prem clusters can’t depend on it is in Part 6.

The DevX subscription: the platform tooling engineers use every day.

The regional ALZs are where workloads live, and each one governs itself. The control plane is where the platform team provisions, deploys to and services them all from one place, and it’s a natural place to centralise security findings.

What comes next

We now have the governance skeleton and the machinery that builds it: the management-group tree, the bootstrap underneath it, the policy that guards it, the code and pipelines that deploy every zone from one definition, and the control plane that provisions them. The next part turns to the network: how one address plan holds together a fully private estate across four regions.

Enterprise Azure Landing Zones

1. Azure Landing Zones: Why We Needed Them, and the Architecture We Built
2. Azure Landing Zones: Governance, and How the Platform Is Built (you are here)
3. Azure Landing Zones: Designing the Network for a Private, Multi-Region Estate
4. Azure Landing Zones: Subscription Vending as a Platform Service
5. Azure Landing Zones: Reaching a Private Estate with VPN, Virtual Desktops and Teleport
6. Azure Landing Zones: Going Hybrid When Data Can’t Leave (coming soon)

Part 1 of the Enterprise Azure Landing Zones series.

Most cloud estates don’t fail because a team can’t spin up a VM. They fail because nobody agreed on where the VM is allowed to land: under which policies, in which network, billed to which subscription, governed by whom. Leave that open across four continents and a few hundred subscriptions, and the tenant stops being a platform. It becomes a large, expensive collection of resources that nobody can fully account for.

This series is about how we built an enterprise Azure platform for one of the world’s largest producers of phosphates and phosphate-based fertilizers, a Morocco-headquartered group with a multi-region estate spanning Africa, Europe, Latin America and APAC. This first part covers the why: why Azure Landing Zones answer that question, and the architecture we chose.

What an Azure Landing Zone actually is

The term gets used loosely, so here is what it means in this series. An Azure Landing Zone (ALZ) is Microsoft’s reference architecture, part of the Cloud Adoption Framework (CAF), for a governed, scalable Azure environment. It is not a product you install. It’s a set of design decisions and guardrails that answer a single question: when a workload arrives, what does it land into?

An ALZ has two halves, and the split is the most useful idea in the whole framework:

• Platform landing zones hold the shared services every workload depends on. CAF defines three, each its own subscription: Connectivity (the hub network, firewall, DNS and gateways), Identity, and Management (logging, monitoring and security tooling such as Defender for Cloud). The platform team owns them.
• Application landing zones are where workloads actually run: one subscription per workload or team and environment, placed under the Landing Zones management group so it inherits the right policy from day one. Workload teams own what runs inside them.

An ALZ is both halves together, held in place by a management-group tree and policy. Concretely, that gives you:

• A management-group hierarchy: the governance skeleton. Subscriptions inherit policy and access from where they sit in the tree, not from whoever created them.
• Policy as the enforcement layer: Azure Policy definitions and initiatives that make non-compliant configurations either impossible or automatically remediated. Encryption, allowed regions, required tags, denied public IPs, all enforced by default rather than by code review.
• A network topology: a hub-and-spoke design that every workload plugs into, with connectivity, DNS and firewalling already solved.
• Identity and management baselines: logging, monitoring and RBAC that exist before the first workload, not bolted on after an audit.

Most of the value is in what a landing zone takes off the plate of the teams shipping workloads. They don’t choose whether to encrypt, which regions are allowed, or how to peer their network. The platform has already decided, and it enforces the decision. That’s how teams can move quickly without each one turning into its own compliance project.

A landing zone on its own is still only infrastructure, though. What makes teams want to use it is platform engineering: a platform team runs the landing zone as an internal product, with self-service instead of tickets, a developer portal as the front door, and golden paths, the supported, ready-made ways to build the things teams ask for most. The guardrails set what nobody is allowed to do. The golden paths make the right way the easiest one. That side of the platform comes through most clearly in Part 4, and I wrote about the team behind it in The Platform Team Is the Backbone, Not the Help Desk.

Why it was required here

For a group of this size and geography, skipping landing zones doesn’t mean a bit more manual work. It means governance gaps that an auditor eventually finds for you.

Operating across Africa, Europe, LATAM and APAC means data-residency and regulatory boundaries that differ by continent. It means regional autonomy, for local teams that need to move without filing a ticket to a central group in another time zone, balanced against consistent governance that has to hold everywhere at once. And it means scale: dozens of business units, each needing their own isolated, billed, governed slice of Azure, provisioned in days rather than quarters.

A flat tenant and good intentions don’t meet those constraints. The guardrails have to be part of the structure.

The architecture: regional landing zones and a control plane

Rather than one monolithic landing zone for the whole group, we deployed an ALZ per region (Africa, Europe, LATAM and APAC), each a full landing-zone implementation in its own right, with its own management-group subtree, its own policy, and its own platform landing zones: its own hub network, its own private DNS, and its own security and monitoring in its Management subscription, tuned to that region’s residency and connectivity needs.

The network in each region follows a hub-and-spoke model: a central hub VNet carrying shared services and an Azure Firewall for egress control and inspection, with each workload’s VNet joining as a spoke. Traffic between spokes and out to the internet flows through the hub, so segmentation and inspection are properties of the topology rather than something each team has to configure.

Inside each region: a hub VNet with Azure Firewall, workload VNets as spokes, all egress inspected.

Next to them sits a control-plane ALZ. It’s a full landing zone too, with platform and application landing zones of its own, but it hosts no business workloads. Its job is to provision the other ALZs: the pipelines, runners and platform tooling that deploy and change the regional zones run here. It’s also where we can centralise security findings, so the platform and security teams get one view across every zone without each region giving up its own.

What the control plane is not is a parent. The regional ALZs don’t inherit from it. Each one has its own management-group tree and policy assignments, and runs its own security and monitoring in its own Management platform landing zone. A problem in one region’s platform stays in that region, and each region can meet its own residency rules without routing its logs elsewhere.

What keeps the zones consistent is the code, not a hierarchy. None of this is clicked together in the portal: the whole estate is built as code on Microsoft’s Azure Verified Modules, and every regional ALZ is deployed from the same modules with different inputs. The security baseline, naming, tagging and allowed regions match everywhere because they’re the same definition. Each region keeps its autonomy, the baseline stays uniform, and no zone depends on another to stay governed.

One ALZ per region, not per environment

One decision shapes everything else: an ALZ is per region, not per environment. An ALZ is production by default. Its hub, firewall, DNS and monitoring are run as production, whatever the workloads inside it are. We didn’t build a production ALZ and a non-production ALZ. Each regional ALZ holds dev, test and production side by side, and the environment boundary sits one level down, at the subscription: a workload gets one application landing zone per environment, each its own subscription with its own budget, RBAC and network.

What that buys us:

• One platform to run per region, not two or three. One hub, one firewall, one DNS setup and one monitoring stack per region, instead of a copy per environment to pay for, patch and keep in sync.
• The same rules in every environment. Dev and production sit under the same policy, so something that works in dev doesn’t fail in production because the guardrails were different. Promoting a workload changes its subscription, not its governance.
• A hard boundary that’s cheap to create. A subscription is a real isolation boundary for access, billing and quotas, and vending creates one in minutes. Adding an environment is a new subscription, not a new landing zone.

What it costs:

• Platform changes reach every environment at once. A policy change, a firewall rule or a hub upgrade applies to dev and production together. There’s no non-production platform where the platform itself gets tested first.
• A shared blast radius in the platform layer. A mistake in a region’s hub or firewall affects every environment in that region, not just one.
• Isolation by subscription, RBAC and network rules, not by separate platforms. Production and non-production share a hub, so keeping them apart is a matter of firewall rules and access control rather than of physical separation. Some regulated workloads need more than that.

We accepted those costs because the code takes most of the sting out of them. Every change is planned and reviewed before it applies, and because an ALZ is entirely defined by its inputs, we can spin up a playground ALZ from the same modules whenever a major change is coming: a new version of the ALZ library, a policy overhaul, a change to the hub design. We rehearse the upgrade there, end to end, then tear the playground down and roll the change out to the real zones. A second permanent landing zone would cost every day; a temporary one costs only the days we need it.

An ALZ spans environments; subscriptions separate them. A throwaway playground ALZ takes the risk out of big upgrades.

The control-plane hub is the transit point

The control plane also has a job in the network. Its hub VNet is the transit point for the whole estate: each regional hub peers with the control-plane hub and with nothing else. It’s a star, not a mesh, so Africa never peers directly with APAC. One hub gives people and platform tooling a single way into every region, and puts every cross-zone flow through one firewall where it’s inspected and logged. Part 3 covers the network design behind it: the address plan, the routing, and how name resolution works when DNS isn’t centralised.

The estate at a glance: four self-contained regional ALZs, provisioned from a control-plane ALZ whose hub is the transit point.

Within each regional ALZ, we then vend subscriptions. That’s how individual workloads and business units get their own governed environment. Subscription vending is the engine that turns “a team needs an Azure environment” into a repeatable, policy-compliant, fully networked subscription without anyone touching the portal. It’s important enough that it gets its own article (Part 4).

Where this is going

So that’s the foundation: self-contained regional landing zones, each with its own governance, hub and DNS, provisioned from a control-plane ALZ whose hub is the transit point, and kept consistent by being deployed from the same code.

The next part goes inside one of those zones: how the management-group tree and policy baseline are designed, what the control plane runs, and how the platform is delivered as code.

Enterprise Azure Landing Zones

1. Azure Landing Zones: Why We Needed Them, and the Architecture We Built (you are here)
2. Azure Landing Zones: Governance, and How the Platform Is Built
3. Azure Landing Zones: Designing the Network for a Private, Multi-Region Estate
4. Azure Landing Zones: Subscription Vending as a Platform Service
5. Azure Landing Zones: Reaching a Private Estate with VPN, Virtual Desktops and Teleport

At Yassir, we asked every squad how often they deployed to production. The answers ranged from “several times a day” to “I’d have to check”. A few squads had a spreadsheet. One had a Grafana panel built on CI logs that only its author understood. Nobody was lying. Nobody actually knew.

That’s a problem when more than 350 people build the product. Leadership wanted to know whether we were getting faster. Squads wanted to know where their time went. DevX, the team that builds our internal developer platform, wanted to know whether the platform work was paying off. All three questions came down to the same four numbers, and we had none of them.

I manage several squads at Yassir, including DevX. This post covers how we got DORA metrics for every squad with Apache DevLake: what the four metrics need from your tools, how we wired GitHub, Argo CD, Jira and PagerDuty into one model, the definitions we had to argue about, and how we kept the numbers from turning into a leaderboard.

Looking back, the tooling was the easy part. The hard part was agreeing what counts as a deployment, what counts as an incident, and which squad owns which repository. Once those were written down, DevLake did the rest.

The four numbers

DORA (DevOps Research and Assessment) found that four metrics predict how well a team delivers software:

• Deployment frequency: how often a squad ships to production.
• Lead time for changes: how long a change takes from first commit to running in production.
• Change failure rate: the share of deployments that cause an incident.
• Failed deployment recovery time: how long it takes to recover when a deployment does break production. (Older write-ups call this MTTR.)

The 2024 DORA report added a fifth, rework rate: the share of deployments that weren’t planned and only happened to fix a problem in production. We stuck to the original four, because they’re the ones DevLake computes and the ones people already recognise.

The first two measure speed, the last two measure stability. The research’s point is that they move together: teams that ship small changes often also break production less, and recover faster when they do. A squad that improves speed by giving up stability isn’t improving.

What makes them hard to get is that no single tool knows all four. GitHub knows about commits and pull requests. The CD system knows what reached production and when. The incident tool knows what broke. Lead time needs a commit in one system joined to a deployment in another. Change failure rate needs a deployment joined to an incident in a third. Each tool has its own IDs, its own idea of a team, and its own clock.

Why DevLake

We looked at three options.

Build it ourselves. Pull from each API into BigQuery, write the joins, build dashboards. We could have done it, but every one of those joins is a decision someone has already made more carefully than we would in a sprint, and we’d own the collectors forever.

Google’s open-source Four Keys project used to be the starting point for this. It’s no longer maintained, and it only covered deployments and incidents, not the squad model we needed.

Buy a SaaS product. LinearB, Swarmia, Sleuth and others do this well. They also want read access to all our source code and every ticket, and they price per seat. At 350+ people that’s a large bill for four numbers, and a hard conversation with security.

Apache DevLake. Open source, self-hosted, with collectors for the tools we already used and the DORA logic already written. It runs as a handful of containers next to a MySQL database, and ships with Grafana dashboards for DORA out of the box.

We picked DevLake. The data never leaves our cloud, there’s no per-seat cost, and when a number looks wrong we can read the SQL behind it.

How DevLake models it

DevLake has three ideas you need before any of the rest makes sense.

• Connections and scopes. A connection is a tool account (our GitHub org, our Jira site). A scope is one thing inside it: a repository, a Jira board, a PagerDuty service.
• The domain layer. Each plugin collects raw data from its tool, then converts it into shared tables: pull_requests, commits, cicd_deployment_commits, issues. Once it's there, a deployment from Argo CD and a deployment from GitHub Actions look the same.
• Projects. A project groups scopes together, and the DORA metrics are calculated per project. An incident only counts against deployments in the same project.

The last one matters most. A DevLake project is a squad. It holds the squad’s repositories, its Jira board, and its PagerDuty services. Get the projects wrong and every number is wrong.

Wiring in the tools

Code: GitHub

The GitHub plugin collects pull requests, commits and reviews for every repository. That gives DevLake the start of the lead time clock: the first commit on a PR.

The first collection was the slowest part. A full first sync of several hundred repositories hits GitHub’s API rate limits fast. We used a GitHub App instead of a personal token for the higher limit, collected in batches, and switched to incremental syncs once the history was in.

Deployments: Argo CD, through a webhook

We spent more time on deployments than on everything else combined.

DevLake can find deployments in CI runs. You give the scope config two regexes: one to match the jobs that deploy, one to match production:

Deployment pattern:  (?i)deploy
Production pattern:  (?i)prod

That works if CI deploys. Ours didn’t. GitHub Actions built and pushed an image, then bumped its tag in our central Argo CD repository, which holds every squad’s manifests. Argo CD did the actual deployment, minutes or hours later, and only Argo CD knew whether the sync to production succeeded. Counting the CI job would have counted builds, not deployments.

So production deployments come from Argo CD. Only production applications subscribe to this trigger, and Argo CD Notifications calls DevLake’s webhook each time one of them syncs a new version:

# argocd-notifications-cm
service.webhook.devlake: |
  url: https://devlake.internal/api/rest/plugins/webhook/connections
  headers:
    - name: Authorization
      value: Bearer $devlake-api-key
template.devlake-deployment: |
  webhook:
    devlake:
      method: POST
      path: /{{index .app.metadata.annotations "devlake/connection-id"}}/deployments
      body: |
        {
          "id": "{{.app.metadata.name}}-{{.app.status.operationState.finishedAt}}",
          "result": "SUCCESS",
          "environment": "PRODUCTION",
          "startedDate": "{{.app.status.operationState.startedAt}}",
          "finishedDate": "{{.app.status.operationState.finishedAt}}",
          "deploymentCommits": [{
            "repoUrl": "{{index .app.metadata.annotations "devlake/source-repo"}}",
            "refName": "main",
            "commitSha": "{{index .app.metadata.annotations "devlake/source-sha"}}",
            "startedDate": "{{.app.status.operationState.startedAt}}",
            "finishedDate": "{{.app.status.operationState.finishedAt}}"
          }]
        }
trigger.on-prod-sync: |
  - when: app.status.operationState.phase == 'Succeeded'
    oncePer: app.metadata.annotations["devlake/source-sha"]
    send: [devlake-deployment]

A few details in there took us a while to get right.

The commit has to be the service’s commit. The revision Argo CD syncs is a commit in the central Argo CD repo, not in the service’s repo, and lead time needs the service commit. So the image-bump step in CI writes the source repo and SHA as annotations on the Argo CD application, and the webhook reports those.

Only code changes count. Argo CD also syncs when someone changes a replica count, a config map or a secret reference. Those are real changes, but they aren’t the deployments DORA means, and they’d inflate the frequency of the squads that tune their config most. oncePer on the source SHA sends one deployment per new service commit, and nothing for a sync that didn't change it.

Only successful syncs are sent. Our first version also sent failed syncs, marked as failures. DevLake ignores failed deployments for DORA anyway, and with oncePer a failed attempt used up the one notification for that commit. The retry that reached production was never sent.

Each squad gets its own webhook connection. In DevLake, a webhook connection is a single scope. Every deployment sent to it belongs to every project that includes it. With one shared connection, every squad would have shared every other squad’s deployments. So each squad project has its own webhook connection, and an annotation on each Argo CD application says which one to call.

We also left DevLake’s own deployment detection off on the GitHub repositories (the deployment regexes stay empty), so a CI job with “deploy” in its name can’t count as a second deployment.

DevLake then works out which commits each deployment shipped by diffing it against the previous production deployment of the same repo. Every PR merged in between gets that deployment’s time as the end of its lead time.

DevLake has had a native Argo CD plugin since late 2025, and we looked at switching to it. It reads sync operations straight from the Argo CD API, which is simpler than a webhook. But it records the revision Argo CD synced as the deployed commit, and for us that’s always a commit in the central Argo CD repo. Every deployment would have pointed at that one repo, the diff would have found image-tag bumps instead of service commits, and no service PR would ever have been linked to a deployment. We kept the webhook.

If your manifests live next to the code, in the service’s own repository, the revision Argo CD syncs is the service commit. Then the plugin is the better choice: no annotations, no notification templates, nothing to maintain.

Incidents: Jira and PagerDuty

Change failure rate and recovery time need incidents. DevLake treats an incident as an issue of type INCIDENT, from either Jira or PagerDuty.

PagerDuty incidents map over directly, per service. Jira needed a mapping in the scope config: our Incident issue type becomes INCIDENT, everything else is ignored for DORA. We used both, because not every production problem paged someone. A broken checkout flow found by support was a Jira incident, not a PagerDuty one.

DevLake links each incident to the most recent production deployment in the same project before the incident started. That deployment counts as failed, and the time from the deployment to the incident’s resolution is the recovery time.

The link is based on time alone. DevLake doesn’t know whether the deployment caused the incident. A cloud provider outage, an expired certificate or a partner’s API going down gets blamed on whatever the squad shipped last. We handled it on the Jira side: incidents with an external cause get their own issue type, External outage, which isn't mapped to INCIDENT. They still exist, they just don't count against a deployment. For PagerDuty, only services owned by a squad are in scope. Infrastructure alerts belong to the platform squads' projects.

Two more things decide whether recovery time means anything. Incidents have to be resolved when service is restored, not when the post-mortem is done, or recovery time becomes post-mortem time. And DevLake only links incidents to deployments when its DORA step runs, at the end of the daily collection. Deployments arrive through the webhook all day, so today’s numbers are provisional until tomorrow’s run.

Feature flags complicate this too. Many of our changes reached users through a LaunchDarkly flag weeks after the code was deployed. DORA measures the deployment, not the release. An incident caused by flipping a flag gets linked to whatever deployment happened before it, which is often unrelated. We accepted that, and made sure the post-mortem template asks whether a flag change was involved, so we could see how often it happens.

Squads, not repositories

DevLake needs to know which squad owns which repository, board and service. We already had that written down.

Our access platform describes every squad as a Team file in Git, with its members and the repositories it owns (I wrote about it in Rebuilding Access Management at Yassir). A small job reads those files and creates one DevLake project per squad, with the right scopes and the squad’s own webhook connection, through DevLake’s API. When a repository moves to another squad, the owner: change in Git moves it in DevLake too.

Without that, someone would have kept a mapping of a few hundred repositories to squads by hand, and it would have been wrong within a month. Squads split and merge. Repositories outlive them.

Shared repositories were the awkward case. A library used by ten squads has one owner, and its deployments count for that owner only. We accepted that. Splitting credit across squads made the numbers harder to explain and didn’t change any decision.

Running DevLake itself

DevLake is three things: a backend that runs the collectors, a config UI, and Grafana. We ran it on one of our GKE clusters with the official Helm chart, with the database on Cloud SQL for MySQL instead of the chart’s bundled one, so database backups and upgrades were Google’s problem.

A few things we’d set up the same way again:

• Keep the encryption secret safe. DevLake encrypts connection credentials with a key you pass in as ENCRYPTION_SECRET. Lose it, or let a redeploy generate a new one, and every connection has to be re-entered. It lives in Secret Manager next to the database credentials.
• Put the config UI behind your SSO. The config UI holds tokens for GitHub, Jira and PagerDuty, and its only built-in protection is a shared basic-auth login. Ours is only reachable through Identity-Aware Proxy, for DevX. Grafana uses Google sign-in, read-only for everyone in engineering.
• Build your own dashboards on the same tables. The bundled DORA dashboard is the right tool for checking the data, but we needed things it doesn’t do: excluding bots, rolling quarters, a “not applicable” state. Ours are separate Grafana dashboards that query DevLake’s tables, kept in their own folder so upgrades don’t overwrite them.
• Collect once a day. A daily blueprint run was enough. DORA trends move over weeks, and hourly collection only burned GitHub API quota.
• Set a collection start date. We collected from the start of the previous quarter, not the beginning of time. Older data was slower to collect and described squads that no longer existed.
• Alert on silence. A squad with no deployment for a week is much more likely to be a broken annotation or a revoked token than a squad that stopped shipping. A daily check flags any project whose deployments stop, so we hear about it before the squad does.

The definitions we had to argue about

Every one of these looked obvious until two squads disagreed.

What is a deployment? A successful sync of a production Argo CD application. Not a merge to main, not a build, not a staging sync. A rollback is a deployment too.

What is production? Any Argo CD application in a production cluster, in any country. We argued about this one the longest, and it turned out to matter less than we thought for frequency. DevLake doesn’t count deployments, it counts deployment days: a squad is elite if the median week has five or more days with a production deployment. Deploying one service to four countries on the same day is one deployment day either way. Where it does matter is change failure rate: four deployments, one incident is 25%, while one deployment, one incident is 100%. We kept one deployment per country, because each country was a separate rollout and a separate risk.

What is an incident? A PagerDuty incident on the squad’s services, or a Jira issue of type Incident. A bug found in production that nobody raised as an incident doesn't count. That pushes squads to declare incidents, which we wanted anyway.

What starts lead time? The first commit on the PR. Long-lived branches show up as long lead times, which is the point.

We wrote these down in one page in the DevX handbook and linked it from every dashboard. When someone questioned a number, the answer was a definition, not an opinion.

The edge cases

The definitions cover the normal path. Most of the work after launch was the cases that didn’t fit.

Rollbacks. Rolling back in Argo CD syncs an older commit. DevLake diffs each deployment against the previous one, so after a rollback the next forward deployment picks up the rolled-back commits again, and their lead time includes the time they spent rolled back. We decided that was correct: the change didn’t reach users until it stuck. The rollback itself counts as a deployment. If it ended an incident, the incident’s resolution time is what records the recovery.

Hotfixes. A hotfix that skips the normal path still goes through Argo CD, so it’s counted. Its lead time is short, which flatters the median a little. The PR that fixes the incident is the interesting signal here, and we let the post-mortem cover it rather than the dashboard.

Monorepos. DevLake works out what a deployment shipped by diffing commits in a repository. In a monorepo with several services, a deployment of service A picks up every commit since A’s last deployment, including commits that only touched service B. The few monorepos we had belonged to one squad each, so the error stayed inside one project. If yours are shared across squads, expect lead time to be wrong there.

Mobile apps. The super app ships through the App Store and Google Play. Between a merge and a user’s phone there’s a release branch, a build, store review and a phased rollout, so “deployed” could mean any of four moments. We picked the moment the phased rollout starts, when the first real users can get the new version.

Submission is too early: nobody has the code yet. 100% rollout is too late, because DevLake links an incident to the last deployment before it. A crash spike on day two of a seven-day rollout would have been blamed on the previous release. Starting the clock at rollout start puts incidents on the release that caused them.

Neither store tells a CI pipeline when that happens. On iOS, the release often goes out by hand after approval. So a small scheduled job polls the App Store Connect and Google Play APIs for each app’s release state, and posts a deployment to the squad’s webhook when a rollout starts. The deployment commit is the release tag in the app repository. DevLake diffs it against the previous release tag, which links every PR in the release to it.

Pausing a phased rollout or shipping a hotfix release is the recovery, and the incident is resolved when the crash rate is back to normal. Store review time lands in lead time, which is honest, since users wait for it too, but it’s outside the squad’s control. And mobile squads release on a weekly train by design, which caps their deployment frequency. Their numbers sit on their own dashboard, compared with their own past releases, never next to backend squads.

Bots. Renovate and Dependabot open a lot of PRs. They’re real changes, but their lead time is mostly time spent waiting for someone to merge them, and including them buried the human signal. We excluded bot authors from lead time and kept their deployments in the frequency count.

Repositories that aren’t services. Infrastructure code, the access repo, shared libraries and documentation have no production deployment of their own. Leaving them in a squad’s project adds PRs that never reach a deployment. They live in a separate project per domain, outside DORA.

Squads with nothing in production. Data and research squads, or a squad still building its first service, have no deployments. Their dashboard says “not applicable”, not “low”. A red tile for a squad that has nothing to deploy is the fastest way to lose trust in the whole thing.

Re-orgs. DevLake calculates a project’s metrics from the scopes it has today. When a repository moves from one squad to another, its whole history moves with it. The receiving squad inherits the old squad’s deployments and incidents for that repo. We put a note on the dashboard for the quarter after any move, and compared trends from after the move only.

Validating before showing anyone

The first numbers were wrong, and we expected that. Before showing any dashboard, we checked two squads end to end: picked a week, listed every PR, deployment and incident by hand, and compared it with what DevLake computed.

DevLake ships a DORA validation dashboard for exactly this. It shows the deployments, the commits linked to each, and the incidents linked to each deployment, as raw rows. That’s where we found the problems:

• Missing deployments, from applications that hadn’t been annotated with their source repo yet.
• Huge lead times, from the first deployment of each repo, which DevLake linked to every commit in its history. We set the collection start date so history didn’t count.
• Incidents on the wrong squad, from PagerDuty services that were still mapped to a squad that no longer existed.

We rolled out to everyone only after both squads agreed their numbers matched what they’d seen.

Keeping it from becoming a leaderboard

This was the part I worried about most. As soon as a number is on a dashboard, someone will rank squads by it. And as soon as squads are ranked, they start optimising the number instead of the delivery: splitting deployments, not declaring incidents, closing them early.

So we set a few rules before launch.

Squads see their own trend first. The default view is one squad over time, not every squad side by side. The question is “are we getting better?”, not “who is best?”.

The numbers are for squads, not for performance reviews. We said it out loud, in writing, and leadership said it too. DORA metrics describe a squad and its system, never a person. DevLake has no per-engineer DORA view in our setup, on purpose.

Speed and stability always appear together. No dashboard shows deployment frequency without change failure rate next to it.

Small squads get wide windows. A squad of four that deploys twice a week and has one incident a quarter doesn’t have enough data points for a weekly change failure rate to mean anything. One incident moves it from 0% to 50%. We showed stability metrics over a rolling quarter, and showed the distribution of lead times next to the median, so one stuck PR didn’t look like a trend.

Numbers need a second source. DORA tells you how the delivery system performs, not how it feels to work in it. We ran a short developer experience survey every quarter, and read the two together. When lead time and the survey agreed that review was the bottleneck, nobody argued about it.

A bad number starts a conversation, not a ticket. When a squad’s lead time went up, the first question was what changed: a new approval step, a flaky test suite, a squad that lost two people. Most of the time the cause was something the squad already knew about but couldn’t prove.

What the numbers showed

Three things stood out:

• Lead time was mostly waiting. For most squads, the biggest part of lead time wasn’t coding or CI. It was the time a PR sat waiting for review.
• The slowest squads weren’t the least skilled. They were the ones with the most manual steps between merge and production.
• Platform work showed up. When DevX moved a domain onto the shared deployment workflows, its lead time dropped within weeks, and we could show it rather than claim it.

That last one changed how DevX planned its work. Instead of guessing which platform improvement would help most, we looked for the squads where the numbers pointed at the platform.

What I’d tell you before you set this up

Agree on the definitions before you install anything. What counts as a deployment, a production environment, an incident. The tool will compute anything. It can’t tell you what you meant.

Measure deployments where they happen. If CD is GitOps, CI doesn’t know when production changed. Report deployments from the thing that does the deployment.

Get ownership from a source of truth. A hand-kept mapping of repositories to squads will be wrong. If you already describe teams as code, generate the projects from it.

Validate by hand first. Before anyone sees a dashboard, pick a week and check every row. The first wrong number someone finds will cost you more trust than the whole setup earned.

Decide what counts as an incident caused by a change. DevLake links incidents to deployments by time alone. If outages you didn’t cause count as incidents, your change failure rate measures your cloud provider.

Write down the edge cases as you find them. Rollbacks, monorepos, mobile releases, bots, re-orgs. Each one is a sentence in the definitions page, and each one saves the same argument next quarter.

Decide what the numbers are not for. Say it before launch. After launch, it sounds like an excuse.

Where it ended up

Every squad at Yassir has its four DORA numbers, from the tools it already uses, updated every day, without anyone filling in a spreadsheet. The definitions fit on one page. Squad ownership comes from the same files as access, so it’s never out of date.

The numbers didn’t make anyone faster on their own. What they did was replace “I think we’re slow” with “our PRs wait two days for review”, and that’s a problem a squad can fix.