Reference

Components

otelcol

otelcol.exporter.kafka

General availability (GA) Open source

`otelcol.exporter.kafka`

otelcol.exporter.kafka accepts logs, metrics, and traces telemetry data from other otelcol components and sends it to Kafka.

It’s important to use otelcol.exporter.kafka together with otelcol.processor.batch to make sure otelcol.exporter.kafka doesn’t slow down due to sending Kafka a huge number of small payloads.

Note
otelcol.exporter.kafka is a wrapper over the upstream OpenTelemetry Collector kafka exporter. Bug reports or feature requests will be redirected to the upstream repository, if necessary.

Multiple otelcol.exporter.kafka components can be specified by giving them different labels.

Usage

otelcol.exporter.kafka "LABEL" {
  protocol_version = "PROTOCOL_VERSION"
}

Arguments

You can use the following arguments with otelcol.exporter.kafka:

Name	Type	Description	Default	Required
`protocol_version`	`string`	Kafka protocol version to use.		yes
`brokers`	`list(string)`	Kafka brokers to connect to.	`["localhost:9092"]`	no
`allow_auto_topic_creation`	`bool`	Whether to allow automatic topic creation.	`true`	no
`client_id`	`string`	Consumer client ID to use. The ID will be used for all produce requests.	`"otel-collector"`	no
`conn_idle_timeout`	`duration`	Time after which idle connections are not reused and may be closed.	`"9m"`	no
`encoding`	`string`	(Deprecated) Encoding of payload read from Kafka.	`"otlp_proto"`	no
`include_metadata_keys`	`list(string)`	List of metadata keys to propagate as Kafka message headers.	`[]`	no
`partition_metrics_by_resource_attributes`	`bool`	Whether to include the hash of sorted resource attributes as the message partitioning key in metric messages sent to Kafka.	`false`	no
`partition_logs_by_resource_attributes`	`bool`	Whether to include the hash of sorted resource attributes as the message partitioning key in log messages sent to Kafka.	`false`	no
`partition_logs_by_trace_id`	`bool`	Whether to use the 16-bit hex string of the trace ID as the message partitioning key in log messages sent to Kafka.	`false`	no
`partition_traces_by_id`	`bool`	Whether to include the trace ID as the message key in trace messages sent to Kafka.	`false`	no
`resolve_canonical_bootstrap_servers_only`	`bool`	Whether to resolve then reverse-lookup broker IP addresses during startup. Deprecated: now a no-op after the upstream Kafka client migration from `sarama` to `franz-go`.	`false`	no
`signal_header`	`bool`	Whether to add an `otelcol.signal` header identifying the signal type to every outgoing Kafka record.	`false`	no
`timeout`	`duration`	The timeout for every attempt to send data to the backend.	`"5s"`	no
`topic_from_attribute`	`string`	A resource attribute whose value should be used as the message’s topic.	`""`	no
`topic`	`string`	(Deprecated) Kafka topic to send to.	See below	no

Warning
The topic and encoding arguments are deprecated in favor of the logs, metrics, and traces blocks.

When topic_from_metadata_key is set in a signal-specific block, it will take precedence over topic_from_attribute and topic arguments. When topic_from_attribute is set, it will take precedence over the topic arguments in logs, metrics, and traces blocks.

partition_traces_by_id doesn’t have any effect on Jaeger encoding exporters since Jaeger exporters include trace ID as the message key by default.

partition_logs_by_resource_attributes and partition_logs_by_trace_id are mutually exclusive and can’t both be true.

include_metadata_keys specifies metadata keys to propagate as Kafka message headers. If one or more keys aren’t found in the metadata, they are ignored. The keys also partition the data before export if sending_queue.batch is defined.

When signal_header is true, the otelcol.signal header key is reserved: it’s an error to also include it in include_metadata_keys.

Blocks

You can use the following blocks with otelcol.exporter.kafka:

Block	Description	Required
`authentication`	Configures authentication for connecting to Kafka brokers.	no
`authentication` > `kerberos`	Authenticates against Kafka brokers with Kerberos.	no
`authentication` > `plaintext`	(Deprecated) Authenticates against Kafka brokers with plaintext.	no
`authentication` > `sasl`	Authenticates against Kafka brokers with SASL.	no
`authentication` > `sasl` > `aws_msk`	Additional SASL parameters when using AWS_MSK_IAM_OAUTHBEARER.	no
`authentication` > `tls`	(Deprecated) Configures TLS for connecting to the Kafka brokers.	no
`authentication` > `tls` > `tpm`	(Deprecated) Configures TPM for the TLS `key_file`.	no
`debug_metrics`	Configures the metrics which this component generates to monitor its state.	no
`logs`	Configures how to send logs to Kafka brokers.	no
`metadata`	Configures how to retrieve metadata from Kafka brokers.	no
`metadata` > `retry`	Configures how to retry metadata retrieval.	no
`metrics`	Configures how to send metrics to Kafka brokers.	no
`producer`	Kafka producer configuration,	no
`producer` > `compression_params`	Configures the compression parameters for the Kafka producer.	no
`record_partitioner`	Selects the Kafka record partitioning strategy.	no
`record_partitioner` > `sticky_key`	Configures the sticky-key partitioner.	no
`retry_on_failure`	Configures retry mechanism for failed requests.	no
`sending_queue`	Configures batching of data before sending.	no
`sending_queue` > `batch`	Configures batching requests based on a timeout and a minimum number of items.	no
`tls`	Configures TLS for connecting to the Kafka brokers.	no
`tls` > `tpm`	Configures TPM settings for the TLS `key_file`.	no
`traces`	Configures how to send traces to Kafka brokers.	no

`logs`

The logs block configures how to send logs to Kafka brokers.

Name	Type	Description	Default	Required
`encoding`	`string`	The encoding for logs. Refer to Supported encodings.	`"otlp_proto"`	no
`message_key_from_metadata_key`	`string`	The name of the metadata key whose value should be used as the message’s key. Mutually exclusive with `partition_logs_by_resource_attributes` and `partition_logs_by_trace_id`.	`""`	no
`topic`	`string`	The name of the Kafka topic to which logs will be exported.	`"otlp_logs"`	no
`topic_from_metadata_key`	`string`	The name of the metadata key whose value should be used as the message’s topic. Takes precedence over `topic_from_attribute` and `topic` settings.	`""`	no

`metrics`

The metrics block configures how to send metrics to Kafka brokers.

Name	Type	Description	Default	Required
`encoding`	`string`	The encoding for metrics. Refer to Supported encodings.	`"otlp_proto"`	no
`message_key_from_metadata_key`	`string`	The name of the metadata key whose value should be used as the message’s key. Mutually exclusive with `partition_metrics_by_resource_attributes`.	`""`	no
`topic`	`string`	The name of the Kafka topic to which metrics will be exported.	`"otlp_metrics"`	no
`topic_from_metadata_key`	`string`	The name of the metadata key whose value should be used as the message’s topic. Takes precedence over `topic_from_attribute` and `topic` settings.	`""`	no

`traces`

The traces block configures how to send traces to Kafka brokers.

Name	Type	Description	Default	Required
`encoding`	`string`	The encoding for traces. Refer to Supported encodings.	`"otlp_proto"`	no
`message_key_from_metadata_key`	`string`	The name of the metadata key whose value should be used as the message’s key. Mutually exclusive with `partition_traces_by_id`.	`""`	no
`topic`	`string`	The name of the Kafka topic to which traces will be exported.	`"otlp_spans"`	no
`topic_from_metadata_key`	`string`	The name of the metadata key whose value should be used as the message’s topic. Takes precedence over `topic_from_attribute` and `topic` settings.	`""`	no

`authentication`

The authentication block holds the definition of different authentication mechanisms to use when connecting to Kafka brokers. It doesn’t support any arguments and is configured fully through inner blocks.

The sasl and kerberos blocks are mutually exclusive; configuring both is rejected.

`kerberos`

The kerberos block configures Kerberos authentication against the Kafka broker.

The following arguments are supported:

Name	Type	Description	Default	Required
`config_file`	`string`	Path to Kerberos location, for example, `/etc/krb5.conf`.		no
`disable_fast_negotiation`	`bool`	Disable PA-FX-FAST negotiation.	`false`	no
`keytab_file`	`string`	Path to keytab file, for example, `/etc/security/kafka.keytab`.		no
`password`	`secret`	Kerberos password to authenticate with.		no
`realm`	`string`	Kerberos realm.		no
`service_name`	`string`	Kerberos service name.		no
`use_keytab`	`string`	Enables using keytab instead of password.		no
`username`	`string`	Kerberos username to authenticate as.		yes

When use_keytab is false, the password argument is required. When use_keytab is true, the file pointed to by the keytab_file argument is used for authentication instead. At most one of password or keytab_file must be provided.

disable_fast_negotiation is useful for Kerberos implementations which don’t support PA-FX-FAST (Pre-Authentication Framework - Fast) negotiation.

`plaintext`

Caution
The plaintext block has been deprecated. Use sasl with mechanism set to PLAIN instead.

The plaintext block configures plain text authentication against Kafka brokers.

The following arguments are supported:

Name	Type	Description	Default	Required
`password`	`secret`	Password to use for plain text authentication.		yes
`username`	`string`	Username to use for plain text authentication.		yes

`sasl`

The sasl block configures SASL authentication against Kafka brokers.

The following arguments are supported:

Name	Type	Description	Default	Required
`mechanism`	`string`	SASL mechanism to use when authenticating.		yes
`password`	`secret`	Password to use for SASL authentication.		yes
`username`	`string`	Username to use for SASL authentication.		yes
`version`	`number`	(Deprecated) Version of the SASL Protocol to use when authenticating.	`0`	no

You can set the mechanism argument to one of the following strings:

"PLAIN"
"SCRAM-SHA-256"
"SCRAM-SHA-512"
"AWS_MSK_IAM_OAUTHBEARER"

When mechanism is set to "AWS_MSK_IAM_OAUTHBEARER", the aws_msk child block must also be provided.

version has been deprecated. The client negotiates the SASL handshake version automatically; setting version doesn’t produce an error, but has no effect.

`aws_msk`

The aws_msk block configures extra parameters for SASL authentication when using the AWS_MSK_IAM_OAUTHBEARER mechanisms.

The following arguments are supported:

Name	Type	Description	Default	Required
`region`	`string`