Basics of the Beam model

Apache Beam is a unified model for defining both batch and streaming data-parallel processing pipelines. To get started with Beam, you’ll need to understand an important set of core concepts:

The following sections cover these concepts in more detail and provide links to additional documentation.

Pipeline

A Beam pipeline is a graph (specifically, a directed acyclic graph) of all the data and computations in your data processing task. This includes reading input data, transforming that data, and writing output data. A pipeline is constructed by a user in their SDK of choice. Then, the pipeline makes its way to the runner either through the SDK directly or through the Runner API’s RPC interface. For example, this diagram shows a branching pipeline:

The pipeline applies two transforms to a single input collection. Eachtransform produces an output collection.

In this diagram, the boxes represent the parallel computations called PTransforms and the arrows with the circles represent the data (in the form of PCollections) that flows between the transforms. The data might be bounded, stored, data sets, or the data might also be unbounded streams of data. In Beam, most transforms apply equally to bounded and unbounded data.

You can express almost any computation that you can think of as a graph as a Beam pipeline. A Beam driver program typically starts by creating a Pipeline object, and then uses that object as the basis for creating the pipeline’s data sets and its transforms.

For more information about pipelines, see the following pages:

PCollection

A PCollection is an unordered bag of elements. Each PCollection is a potentially distributed, homogeneous data set or data stream, and is owned by the specific Pipeline object for which it is created. Multiple pipelines cannot share a PCollection. Beam pipelines process PCollections, and the runner is responsible for storing these elements.

A PCollection generally contains “big data” (too much data to fit in memory on a single machine). Sometimes a small sample of data or an intermediate result might fit into memory on a single machine, but Beam’s computational patterns and transforms are focused on situations where distributed data-parallel computation is required. Therefore, the elements of a PCollection cannot be processed individually, and are instead processed uniformly in parallel.

The following characteristics of a PCollection are important to know.

Bounded vs. unbounded:

A PCollection can be either bounded or unbounded.

These two categories derive from the intuitions of batch and stream processing, but the two are unified in Beam and bounded and unbounded PCollections can coexist in the same pipeline. If your runner can only support bounded PCollections, you must reject pipelines that contain unbounded PCollections. If your runner is only targeting streams, there are adapters in Beam’s support code to convert everything to APIs that target unbounded data.

Timestamps:

Every element in a PCollection has a timestamp associated with it.

When you execute a primitive connector to a storage system, that connector is responsible for providing initial timestamps. The runner must propagate and aggregate timestamps. If the timestamp is not important, such as with certain batch processing jobs where elements do not denote events, the timestamp will be the minimum representable timestamp, often referred to colloquially as “negative infinity”.

Watermarks:

Every PCollection must have a watermark that estimates how complete the PCollection is.

The watermark is a guess that “we’ll never see an element with an earlier timestamp”. Data sources are responsible for producing a watermark. The runner must implement watermark propagation as PCollections are processed, merged, and partitioned.

The contents of a PCollection are complete when a watermark advances to “infinity”. In this manner, you can discover that an unbounded PCollection is finite.

Windowed elements:

Every element in a PCollection resides in a window. No element resides in multiple windows; two elements can be equal except for their window, but they are not the same.

When elements are written to the outside world, they are effectively placed back into the global window. Transforms that write data and don’t take this perspective risk data loss.

A window has a maximum timestamp. When the watermark exceeds the maximum timestamp plus the user-specified allowed lateness, the window is expired. All data related to an expired window might be discarded at any time.

Coder:

Every PCollection has a coder, which is a specification of the binary format of the elements.

In Beam, the user’s pipeline can be written in a language other than the language of the runner. There is no expectation that the runner can actually deserialize user data. The Beam model operates principally on encoded data, “just bytes”. Each PCollection has a declared encoding for its elements, called a coder. A coder has a URN that identifies the encoding, and might have additional sub-coders. For example, a coder for lists might contain a coder for the elements of the list. Language-specific serialization techniques are frequently used, but there are a few common key formats (such as key-value pairs and timestamps) so the runner can understand them.

Windowing strategy:

Every PCollection has a windowing strategy, which is a specification of essential information for grouping and triggering operations. The Window transform sets up the windowing strategy, and the GroupByKey transform has behavior that is governed by the windowing strategy.


For more information about PCollections, see the following page:

PTransform

A PTransform (or transform) represents a data processing operation, or a step, in your pipeline. A transform is usually applied to one or more input PCollection objects. Transforms that read input are an exception; these transforms might not have an input PCollection.

You provide transform processing logic in the form of a function object (colloquially referred to as “user code”), and your user code is applied to each element of the input PCollection (or more than one PCollection). Depending on the pipeline runner and backend that you choose, many different workers across a cluster might execute instances of your user code in parallel. The user code that runs on each worker generates the output elements that are added to zero or more output PCollection objects.

The Beam SDKs contain a number of different transforms that you can apply to your pipeline’s PCollections. These include general-purpose core transforms, such as ParDo or Combine. There are also pre-written composite transforms included in the SDKs, which combine one or more of the core transforms in a useful processing pattern, such as counting or combining elements in a collection. You can also define your own more complex composite transforms to fit your pipeline’s exact use case.

The following list has some common transform types:

For more information about transforms, see the following pages:

Aggregation

Aggregation is computing a value from multiple (1 or more) input elements. In Beam, the primary computational pattern for aggregation is to group all elements with a common key and window then combine each group of elements using an associative and commutative operation. This is similar to the “Reduce” operation in the MapReduce model, though it is enhanced to work with unbounded input streams as well as bounded data sets.

Aggregation of elements.

Figure 1: Aggregation of elements. Elements with the same color represent those with a common key and window.

Some simple aggregation transforms include Count (computes the count of all elements in the aggregation), Max (computes the maximum element in the aggregation), and Sum (computes the sum of all elements in the aggregation).

When elements are grouped and emitted as a bag, the aggregation is known as GroupByKey (the associative/commutative operation is bag union). In this case, the output is no smaller than the input. Often, you will apply an operation such as summation, called a CombineFn, in which the output is significantly smaller than the input. In this case the aggregation is called CombinePerKey.

In a real application, you might have millions of keys and/or windows; that is why this is still an “embarrassingly parallel” computational pattern. In those cases where you have fewer keys, you can add parallelism by adding a supplementary key, splitting each of your problem’s natural keys into many sub-keys. After these sub-keys are aggregated, the results can be further combined into a result for the original natural key for your problem. The associativity of your aggregation function ensures that this yields the same answer, but with more parallelism.

When your input is unbounded, the computational pattern of grouping elements by key and window is roughly the same, but governing when and how to emit the results of aggregation involves three concepts:

For more information about available aggregation transforms, see the following pages: