Managed Service for Apache Airflow versioning overview

This page describes Managed Service for Apache Airflow versioning and lists differences between major Managed Airflow versions.

Major versions of Managed Airflow

Managed Airflow has the following major versions:

  • Managed Airflow (Gen 3). This version has simplified network setup and hides infrastructure components including the environment's cluster and environment's dependencies on other services.

  • Managed Airflow (Gen 2). In this version, the environment's cluster scales automatically to meet the resource demands.

  • Managed Airflow (Legacy Gen 1). Has manual environment scaling, the infrastructure is deployed to your projects and networks.

Comparison of Managed Airflow versions

The following table lists major differences between Managed Airflow (Legacy Gen 1), Managed Airflow (Gen 2), and Managed Airflow (Gen 3).

Feature Managed Airflow (Legacy Gen 1) Managed Airflow (Gen 2) Managed Airflow (Gen 3)
Image version composer-1.b.c-airflow-x.y.z composer-2.b.c-airflow-x.y.z composer-3-airflow-x.y.z-build.t
Airflow versions Airflow 1.10.* and Airflow 2 Airflow 2 Airflow 2, Airflow 3
Python versions 3.8.12 3.11.8 (3.11.5 and 3.8.12 in earlier versions) 3.11.8
Environment's cluster Standard mode VPC-native or Routes-based Google Kubernetes Engine cluster Autopilot mode VPC-native Google Kubernetes Engine cluster Environment's cluster is not deployed into your project.
Horizontal scaling Can adjust the number of nodes in the environment's cluster. This changes the number of Airflow workers.
Can adjust the number of Airflow schedulers.
Automatically scaling number of Airflow workers, based on demand. Can set and change upper and lower limits for the number of workers.
Can adjust the number of Airflow schedulers and triggerers
Automatically scaling number of Airflow workers, based on demand. Can set and change upper and lower limits for the number of workers.
Can adjust the number of Airflow schedulers, triggerers, and DAG processors
Vertical scaling Can set machine types for cluster nodes, Airflow web server and database when creating an environment. Can change machine types for Airflow web server and database. Can set and change the workloads configuration: CPU, memory, and storage parameters for Airflow workers, schedulers, web server, and the environment size. Can set and change the workloads configuration: CPU, memory, and storage parameters for Airflow workers, schedulers, web server, DAG processors, and the environment size.
Pricing model Managed Airflow (Legacy Gen 1) pricing model Managed Airflow (Gen 2) pricing model Managed Airflow (Gen 3) pricing model
Access control Managed Airflow (Legacy Gen 1) roles Managed Airflow roles. Uses Workload Identity Federation for GKE and requires additional permissions for the Managed Airflow Service Agent account. Managed Airflow roles. Uses Workload Identity Federation for GKE. Extra permissions are not required
Access through Workforce Identity Federation Not supported Supported Supported
Highly Resilient environments Not supported Supported Supported
Database retention policy Not supported Not yet available Supported
Private IP networking VPC peerings Private Service Connect Simplified network setup. Can be changed between Public and Private IP in an existing environment.
VPC Service Controls Supported Supported Supported
Airflow executors Celery Executor Celery Executor CeleryKubernetes Executor
Airflow triggerer (deferrable operators) Not supported Supported Supported
Custom service account for KubernetesPodOperator Supported Supported Not supported
CMEK Supported Supported Supported
Web server access controls Supported Supported Supported
Restarting web server Supported Supported Supported
Web server plugins Supported in Airflow 1 environments with disabled DAG serialization. Supported Supported, plugins can be disabled and enabled on demand.
Scheduler plugins Not supported Supported Not supported
Listeners Not supported Supported Only task-level listeners
TimeTables Only Built-In Custom and Built-In Only Built-In
Scheduler custom PyPI packages Not supported Supported Not supported
Installing custom plugins Supported Supported Supported
Maintenance operations All tasks can be impacted. Tasks that take less than 55 minutes to execute are not impacted. Tasks that take less than 24 hours to execute are not impacted.
Storing Snapshots Supported for Airflow 2.x and 1.10.15 versions Supported Supported
Loading Snapshots Not supported Supported Supported
Scheduled Snapshots Not supported Supported Supported
Custom environment's bucket Not supported Supported Supported
Saving Airflow task logs only in Cloud Logging Not supported Supported Supported
Data lineage integration Not supported Supported Supported
Composer Local Development CLI tool Not supported Supported Supported

Comparison of Airflow versions

Managed Airflow (Gen 3) provides environments with Airflow 2 and Airflow 3.

Support for Managed Airflow (Gen 3) features

The following table summarizes Managed Airflow (Gen 3) features that aren't yet supported in Airflow 3.

Feature Status Description
Upgrades to Airflow 3 (snapshots) Not yet supported in Airflow 3 Side-by-side migration of existing Managed Airflow (Gen 3) environments from Airflow 2 to Airflow 3 using snapshots.
Upgrades to Airflow 3 (in-place) Not yet supported in Airflow 3 In-place upgrades of existing Managed Airflow (Gen 3) environments from Airflow 2 to Airflow 3 by specifying a version with Airflow 3 during an upgrade.

Known issues in Airflow 3

The following known issues have been identified in Airflow 3:

  • Warning messages in the logs of Airflow workloads. These messages can be safely ignored.
  • The airflowctl utility can't be easily used from headless environments such as Airflow workers in Managed Airflow or Docker containers. This is a limitation of the utility. For more information about this limitation, see Issue 58230 in the Airflow repository on GitHub.

Support for Airflow 3 features

The following table lists Airflow 3 features and their support:

Feature Status
Airflow 3.0 UI Supported
DAG versioning Supported
Backfills (from Airflow UI) Supported
Event-driven scheduling Supported
Assets Supported
Inference execution and hyper-parameter tuning Supported
airflowctl command-line tool Supported
DAG bundles (other than LocalDagBundle) Not yet supported in Airflow 3
Edge Executor and tasks in other programming languages Not yet supported in Airflow 3

Managed Service for Apache Airflow images

To run Apache Airflow, Managed Airflow builds Docker images that bundle Airflow releases with other common binaries and Python libraries.

Managed Airflow images include Airflow modifications that are specific to Managed Airflow and unsuitable for the upstream Airflow codebase. Each Managed Airflow image contains preinstalled packages along with Managed Airflow-specific Airflow changes.