This guide describes how to create and deploy an orchestration pipeline using Google Cloud Data Agent Kit.
The example pipeline runs a PySpark script in Managed Service for Apache Spark.
You can deploy orchestration pipelines from your IDE as
local versions or through a GitHub action such as when merging changes
to the main branch. This document demonstrates how to deploy the local
version of an orchestration pipeline.
Before you begin
Before you begin, complete the following:
- Install the Data Agent Kit.
- Configure your settings.
- Add a GitHub repository to your IDE workspace to store orchestration pipelines and assets such as scripts.
Review the required IAM roles
To obtain the permissions to create resources in your project, deploy, and run orchestration pipelines, ask your administrator to grant you the required roles.
To create and manage Managed Service for Apache Airflow environments and manage objects in their associated buckets, you need the following roles. For more information about these user roles, see Grant roles to users in the Managed Service for Apache Airflow documentation.
- Environment and Storage Object Administrator (composer.environmentAndStorageObjectAdmin)
- Service Account User
(
iam.serviceAccountUser)
To work with BigQuery and Cloud Storage resources, you need the following roles.
- BigQuery Data Editor (
roles/bigquery.dataEditor) - Storage Object Admin (
roles/storage.objectAdmin)
Depending on the resources you plan to access, you might need additional roles beyond the roles that let you use the extension and work with orchestration pipelines.
Create a service account and grant it IAM roles
Use a unique service account for the Managed Airflow Gen 3 environment. The service account creates a Managed Airflow Gen 3 environment and runs all orchestration pipelines you deploy.
Ask your administrator to complete the following steps: