Transform received events

Advanced

You can transform your event data by writing transformation expressions using CEL. For example, you can modify event payloads to satisfy a destination's specific API contract.

Note that events are always delivered in a CloudEvents format using an HTTP request in binary content mode unless you specify a message binding.

Set the input and output data formats

In addition to writing a transformation expression in CEL, you can optionally specify the data format of the incoming event data. This lets Eventarc Advanced know how to parse the payload of the event. You can also convert the data from one format to another.

The following formats are supported: Avro, JSON, and Protobuf. For more information, see Format received events.

Transformation expressions

When transforming events, all event attributes can be accessed in a CEL expression as variables through a predefined message object. These variables are populated with values based on the event data at runtime. For example:

message.id returns the id attribute of the event
message.data returns a CEL value representation of the event payload
message.data.some-key returns the content of a field named some-key from the event payload

Fields in message.data are always represented as String types and values are mapped from the original event using the schema specified when setting the input data format.

The transformation expression should express a complete event that includes the event context attributes and the event data payload. Expressions are written in JSON but predefined CEL functions, macros, and operators, as well as regular expressions using RE2 are supported. Eventarc Advanced also supports certain extension functions that can be used to transform the event data.

The following are two examples of using CEL expressions to transform your event data. For more use cases and examples, see Transformation examples.

Example: Format attribute values

The following example formats phone_number attribute values using regular expression functions. (Other attributes have been omitted.)

  // Input:
  // {
  //   "data":
  //   {
  //     "email_address": "charlie@altostrat.com",
  //     "phone_number": "8005550100",
  //   }
  // }
  // Output:
  // {
  //    "data":
  //    {
  //      "email_domain": "altostrat.com",
  //      "phone_number": "(800) 555-0100",
  //      "area_code": "800",
  //      "local_number": "5550100",
  //    }
  // }

  {
    "data":
    {
      "email_domain": re.capture(
                        message.data.email_address,
                        "\\S+@(\\S+)"),

      "phone_number": re.extract(
                        message.data.phone_number,
                        "^(\\d{3})(\\d{3})(\\d{4})", "(\\1) \\2-\\3"
                      ),

    }.merge ( re.captureN(message.data.phone_number,
                        "^(?P\d{3})[\w\-)(]*(?P\d{7})"
                      )
    )
  }

These are the regular expression functions used in the preceding example:

re.capture: captures the first unnamed or named group value. Arguments are the following:
- target: string that should be parsed
- regex: regular expression used to capture values
Returns a string of the first captured group value.
re.captureN: does a full match on the given string and regular expression. Arguments are the following:
- target: string that should be parsed
- regex: regular expression used to capture values
Returns a map with key and value pairs for a named group (group name, captured string) or an unnamed group (group index, captured string).
re.extract: matches group values from the given target string and rewrites the string. Arguments are the following:
- target: string that should be parsed
- regex: regular expression used to extract values
- rewrite: regular expression for how the result should be formatted
Returns a string of the extracted values that is formatted based on the rewrite argument.

Example: Map an array to an array of objects

The following example maps an array of integers into an array of objects. (Other attributes have been omitted.)

  // Input:
  // {
  //   "data":
  //   {
  //        "product_ids": [1, 2, 3]
  //   }
  // }
  // Output:
  // {
  //    "data":
  //    {
  //             "products": [
  //                {
  //                   "name": "apple",
  //                   "price": 70
  //                },
  //                {
  //                    "name": "orange",
  //                    "price":  80
  //                },
  //                {
  //                    "name": "Product(3)",
  //                    "price": 0
  //                },
  //                {
  //                     "name": "apple",
  //                     "price": 70
  //                }
  //            ]
  //    }
  // }

  {
    "data":
    {
      "products":  message.data.product_ids.map(product_id,
              product_id == 1?
              {
                "name": "apple",
                "price": 70
              } :
              product_id == 2?
              {
                "name": "orange",
                "price":  80
              } :
              // Default:
              {
                "name": "Product(" + string(product_id) + ")",
                "price": 0
              }
          )
    }
  }

Configure a pipeline to transform events

You can configure a pipeline to transform event data in the Google Cloud console or by using the gcloud CLI.

Note that only one mediation per pipeline is supported.

Console

In the Google Cloud console, go to the Eventarc > Pipelines page.

Go to Pipelines
You can create a pipeline or, if you are updating a pipeline, click the name of the pipeline.
In the Pipeline details page, click Edit.
In the Event mediation pane, do the following:
1. Select the Apply a transformation checkbox.
2. In the Inbound format list, select the applicable format.
  
  For more information, see Format received events.
3. In the CEL expression field, write a transformation expression in JSON. Predefined CEL functions, macros, and operators, as well as regular expressions are supported. For example:
```
{
"id": message.id,
"datacontenttype": "application/json",
"data": "{ \"scrubbed\": \"true\" }"
}
```
  The preceding example does the following:
  - Removes all attributes from the original event except its id
  - Sets the datacontenttype attribute to application/json
  - Replaces the event payload with a static JSON string
4. Click Continue.
In the Destination pane, do the following:
1. If applicable, in the Outbound format list, select a format.
  
  For more information, see Format received events.
2. Optionally, apply a Message binding. For more information, see the Define a message binding section in this document.
Click Save.

It can take a couple of minutes to update a pipeline.

gcloud

Open a terminal.
You can create a pipeline or you can update a pipeline using the gcloud eventarc pipelines update command:
```
gcloud eventarc pipelines update PIPELINE_NAME \
    --location=REGION \
    --mediations=transformation_template=\
'
 {
    TRANSFORMATION_EXPRESSION
 }
'
```
Note: The --mediations flag uses a hash symbol (#) as the default delimiter. If your expression contains a hash, specify a different delimiter to avoid parsing conflicts. For more information, see gcloud topic escaping. For example, to use a semicolon (;) as a delimiter:
--mediations=^;^transformation_template=TRANSFORMATION_EXPRESSION

Replace the following:
- PIPELINE_NAME: the ID of the pipeline or a fully qualified name
- REGION: a supported Eventarc Advanced location
  
  Alternatively, you can set the gcloud CLI location property:
```
gcloud config set eventarc/location REGION
```
- TRANSFORMATION_EXPRESSION: an expression written in JSON. Predefined CEL functions, macros, and operators, as well as regular expressions are supported. A mediations flag is used to apply a transformation_template key.
It can take a couple of minutes to update a pipeline.

Example:
```
gcloud eventarc pipelines update my-pipeline \
    --location=us-central1 \
    --mediations=transformation_template=\
'
 {
```