To more strongly embrace the success and growing customer preference for OSS solutions, Cloud Composer is evolving to become Managed Service for Apache Airflow. This name change provides improved customer understanding of our portfolio while reinforcing our commitment to being the most open cloud ecosystem.

Save and load environment snapshots

Managed Airflow (Gen 3) | Managed Airflow (Gen 2) | Managed Airflow (Legacy Gen 1)

This page explains how to save and load the state of your environment using environment snapshots.

You can configure your environment to automatically save snapshots. For more information, see Configure scheduled snapshots.

About environment snapshots

Environment snapshots store the state of your environment. You can save and load environment snapshots on demand.

You can use snapshots to:

Restore your environment to an earlier state. Create backups of your environment using snapshots, then load one of the snapshots to restore your environment. For example, you can roll back an update operation that installs PyPI packages or overrides Airflow configuration options, even after your environment is successfully updated.
Perform side-by-side upgrades and copy environments. Create a snapshot of an existing environment, create a new environment that uses the same or later version of Managed Airflow, then apply the snapshot to this new environment. This procedure is similar to migrating. Make sure to pause DAGs before you make the snapshot, to avoid duplicate DAG runs.

How snapshots are stored

An environment snapshot is a set of files that describe the state of your environment and store the backup of the environment data.

You can create multiple snapshots of your environment. Environment snapshots are non-incremental. You can use any snapshot independently of other snapshots.

Managed Airflow does not delete snapshots when you delete your environment.

By default, Managed Airflow stores snapshots in the snapshots/ folder in your environment's bucket. You can also specify a custom location when you create a snapshot.

While the Cloud Storage UI displays data as folders for convenience, buckets work in a flat-file structure. This doesn't change how you work with saving and loading snapshots in the Google Cloud console, but keep this in mind when you perform operations directly on the files in your bucket. For more information, see Object namespace.

Security considerations for snapshots

To mitigate this security risk, you can store sensitive information that is used by Airflow DAGs, such as keys or passwords, in Secret Manager. For more information, see Configure Secret Manager for your environment.

Make sure to check security permissions for your environment's bucket. If you store environment snapshots in a custom bucket, make sure that access permissions for it are configured properly in your project. When assigning permissions, make sure that environment's service account has enough permissions to save and load snapshots from the bucket.

What data is saved in snapshots

Managed Airflow saves the following data in snapshots: