This page describes pricing for Dataflow. To see the pricing for other products, read the Pricing documentation.
To learn how you can save 40% with a three-year commitment or 20% with a one-year commitment, review our committed use discounts (CUDs) page.
Dataflow usage is billed for resources that your jobs use. Depending on the pricing model that you use, resources are measured and billed differently.
| Data Compute Units (DCUs) (batch and streaming) |
Data Compute Units (DCUs)
(batch and streaming)
Other Dataflow resources that are billed for all jobs include Persistent Disk, GPUs, and snapshots.
Resources from other services might be used for the Dataflow job. Services that are used with Dataflow might include BigQuery, Pub/Sub, Cloud Storage, and Cloud Logging, among others.
Although the rate for pricing is based on the hour, Dataflow usage is billed in per second increments, on a per job basis. Usage is stated in hours in order to apply hourly pricing to second-by-second use. For example, 30 minutes is 0.5 hours. Workers and jobs might consume resources as described in the following sections.
Future releases of Dataflow might have different service charges or bundling of related services.
Dataflow compute resources
Dataflow billing for compute resources includes the following components:
For more information about the available regions and their zones, see the Compute Engine Regions and Zones page.
Worker vCPU and memory
Each Dataflow job uses at least one Dataflow worker. The Dataflow service provides two worker types: batch and streaming. Batch and streaming workers have separate service charges.
Dataflow workers consume the following resources, each billed on a per second basis:
Batch and streaming workers are specialized resources that use Compute Engine. However, a Dataflow job does not emit Compute Engine billing for Compute Engine resources managed by the Dataflow service. Instead, Dataflow service charges encompass the use of these Compute Engine resources.
You can override the default worker count for a job. If you are using autoscaling, you can specify the maximum number of workers to allocate to a job. Workers and respective resources are added and removed automatically based on autoscaling actuation.
In addition, you can use pipeline options to override the default resource settings, such as machine type, disk type, and disk size, that are allocated to each worker and that use GPUs.
FlexRS
Dataflow provides an option with discounted vCPU and memory pricing for batch processing. Flexible Resource Scheduling (FlexRS) combines regular and preemptible VMs in a single Dataflow worker pool, giving users access to cheaper processing resources. FlexRS also delays the execution of a batch Dataflow job within a 6-hour window to identify the best point in time to start the job based on available resources.
Although Dataflow uses a combination of workers to execute a FlexRS job, you are billed a uniform discounted rate of about 40% on vCPU and memory cost compared to regular Dataflow prices, regardless of the worker type. You instruct Dataflow to use FlexRS for your autoscaled batch pipelines by specifying the FlexRS parameter.
Dataflow Shuffle data processed
For batch pipelines, Dataflow provides a highly scalable feature, Dataflow Shuffle, that shuffles data outside of workers. For more information, see Dataflow Shuffle.
Dataflow Shuffle charges by the volume of data processed during shuffle.
Streaming Engine pricing
For streaming pipelines, the Dataflow Streaming Engine moves streaming shuffle and state processing out of the worker VMs and into the Dataflow service backend. For more information, see Streaming Engine.
Streaming Engine Compute Units
With resource-based billing, Streaming Engine resources are measured in Streaming Engine Compute Units. Dataflow meters the Streaming Engine resources that each job uses and then bills based on the total resources that are used by that job. To enable resource-based billing for your job, see Use resource-based billing. When you use resource-based billing, existing discounts are automatically applied.
When you use Dataflow Prime with resource-based billing, you're billed based on the total resources that each job uses, but the Data Compute Unit (DCU) SKU is used instead of the Streaming Engine Compute Unit SKU.
Streaming Engine data processed (legacy)
Dataflow continues to support the legacy data-processed billing. Unless you enable resource-based billing, jobs are billed by using data-processed billing.
Streaming Engine data-processed billing measures usage by the volume of streaming data processed, which depends on the following factors:
Examples of what counts as a byte processed include the following items:
Dataflow compute resource pricing - Batch
The following table contains pricing details for worker resources and Shuffle data processed for Batch.