Internal telemetry

You can inspect the health of any OpenTelemetry Collector instance by checking its own internal telemetry. Read on to learn about this telemetry and how to configure it to help you monitor and troubleshoot the Collector.

Activate internal telemetry in the Collector

By default, the Collector exposes its own telemetry in two ways:

• Internal metrics are exposed using a Prometheus interface which defaults to port 8888.
• Logs are emitted to stderr by default.

Configure resource attributes

The Collector’s automatically attaches the service.name, service.version, and service.instance.id (randomly generated) resource attributes to its internal telemetry signals. These can be disabled by setting the attribute value to null (ex. service.name: null).

If you’d like to add additional resource attributes to the Collector’s internal telemetry signals (traces, metrics, and logs) you can set them under service::telemetry::resource:

service:
  telemetry:
    resource:
      attribute_key: 'attribute_value'

Configure internal metrics

OTLP exporter for internal metrics

You can configure how internal metrics are generated and exposed by the Collector. By default, the Collector generates basic metrics about itself and exposes them using the OpenTelemetry Go Prometheus exporter for scraping at http://127.0.0.1:8888/metrics.

The Collector can push its internal metrics to an OTLP backend via the following configuration:

service:
  telemetry:
    metrics:
      readers:
        - periodic:
            exporter:
              otlp:
                protocol: http/protobuf
                endpoint: https://backend:4318

For all available options, see OTLP exporter options.

Prometheus endpoint for internal metrics

Alternatively, you can expose the Prometheus endpoint to one specific or all network interfaces when needed. For containerized environments, you might want to expose this port on a public interface.

Set the Prometheus config under service::telemetry::metrics:

service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: '0.0.0.0'
                port: 8888

If you want to add additional labels to the Prometheus metrics, you can add them with prometheus::with_resource_constant_labels:

prometheus:
  host: '0.0.0.0'
  port: 8888
  with_resource_constant_labels:
    included:
      - label_key

And then reference the labels in service::telemetry::resource:

resource:
  label_key: label_value

Service address

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8888

Metric verbosity

You can adjust the verbosity of the Collector metrics output by setting the level field to one of the following values:

• none: no telemetry is collected.
• basic: essential service telemetry.
• normal: the default level, adds standard indicators on top of basic.
• detailed: the most verbose level, includes dimensions and views.

Each verbosity level represents a threshold at which certain metrics are emitted. For the complete list of metrics, with a breakdown by level, see Lists of internal metrics.

The default level for metrics output is normal. To use another level, set service::telemetry::metrics::level:

service:
  telemetry:
    metrics:
      level: detailed

Metric views

You can further configure how metrics from the Collector are emitted by using views. For example, the following configuration updates the metric named otelcol_process_uptime to emit a new name process_uptime and description:

service:
  telemetry:
    metrics:
      views:
        - selector:
            instrument_name: otelcol_process_uptime
            instrument_type:
          stream:
            name: process_uptime
            description: The amount of time the Collector has been up

You can also use views to update the resulting aggregation, attributes, and cardinality limits. For the full list of options, see the examples in the OpenTelemetry Configuration schema repository.

Configure internal logs

Log output is found in stderr. You can configure logs in the config service::telemetry::logs. The configuration options are:

You can also see logs for the Collector on a Linux systemd system using journalctl:

• All logs
• Errors only