On This Page

Home / Stream/ Monitor Health and Metrics/Internal Logs

Internal Logs ​

Cribl Stream generates internal application logs that monitor its own operations and health. They provide valuable insights into the system’s behavior, performance, and potential issues.

Distributed deployments emit a larger set of logs than single-instance deployments. We’ll describe the distributed set first.

You can display and export all internal logs by selecting Monitoring in the sidebar, then selecting Logs (Stream) or selecting Logs in the sidebar (Edge). Logs’ persistence depends on event volume, not time - for details, see Log Rotation and Retention.

Several logs listed on this page are exposed only in customer-managed (on-prem) deployments. In Cribl.Cloud, Leaders support Cribl Stream Worker Node logs on hybrid Workers.

However, Organization Members who have the Admin Permission on Cribl Search can use that product to search the cribl_internal_logs Dataset for additional details about the Leader and its Cribl-managed Workers.

Leader Node Logs (Distributed) ​

The API/main process emits the following logs into the Leader Node’s $CRIBL_HOME/log/ directory.

Logfile NameDescriptionEquivalent on Logs page
cribl.logPrincipal log in Cribl Stream. Includes telemetry/license-validation logs. Corresponds to top-level cribl.log on Diag page.Leader > API Process
access.logAPI calls, for example, GET /api/v1/version/info.Leader > Access
audit.logActions pertaining to files, for example, create, update, commit, deploy, delete.Leader > Audit
notifications.logMessages that appear in the Notification list in the UI. For each event, the time and _time fields record the start of the monitored interval (matching starttime), not when the Notification was written.Leader > Notifications
ui-access.logInteractions with different UI components described as URLs, for example, /settings/apidocs, /dashboard/logs.Leader > UI Access

The API/main process emits the following service logs into the Leader Node’s $CRIBL_HOME/log/service/ directory. Each service includes a cribl.log file that logs the service’s internal telemetry and an access.log file that logs which API calls the service has handled.

Service NameDescriptionEquivalent on Logs page
Connections ServiceHandles all Worker connections and communication, including heartbeats, bundle deploys, teleporting, restarting, and so on. Workers are assigned to connection processes using a round-robin algorithm.Leader > Connections Service
Lease Renewal ServiceHandles lease renewal for the primary Leader Node.Leader > Lease Renewal Service
Metrics ServiceHandles in-memory metrics, merging of incoming packets, metrics persistence and rehydration, and UI queries for metrics.Leader > Metrics Service
Notifications ServiceTriggers Notifications based on its configuration.Leader > Notifications Service

The Config Helper process for each Worker Group/Fleet emits the following log in $CRIBL_HOME/log/group/GROUPNAME.

Logfile NameDescriptionEquivalent on Logs page
cribl.logMessages about config maintenance, previews, and so on.GROUPNAME > Config helper

Outpost Logs ​

The Outpost service emits the following logs into the $CRIBL_HOME/log/service/outpost/N/ directory, where N is the Outpost Node Process ID:

Logfile NameDescriptionEquivalent on the Logs tab in Outpost info drawer
cribl.logInternal telemetry logs.Outpost Service N
metrics.logOutpost Node metrics.Outpost Service N - Metrics

Worker Node Logs (Distributed) ​

The API Process emits the following log in $CRIBL_HOME/log/.

Logfile NameDescriptionEquivalent on Logs page
cribl.logMessages about the Worker/Edge Node communicating with the Leader Node (that is, with its API Process) and other API requests, for example, sending metrics, reaping job artifacts.GROUPNAME > Worker:HOSTNAME > API Process

Each Worker Process emits the following logs in $CRIBL_HOME/log/worker/N/, where N is the Worker/Edge Node Process ID. (The metrics.log file is written only when HTTP-based Destinations exist.)

Logfile NameDescriptionEquivalent on Logs page
cribl.logMessages about the Worker/Edge Node processing data.GROUPNAME > Worker:HOSTNAME > Worker Process N
metrics.logMessages about the Worker/Edge Node’s outbound HTTP request statistics.GROUPNAME > Worker:HOSTNAME > Worker Process N

If TCP load balancing is enabled in at least one supported Source (such as Syslog, TCP JSON, or Cribl TCP), the Worker Process also emits the following log to $CRIBL_HOME/log/worker/LB:

Logfile NameDescriptionEquivalent on Logs page
cribl.logInternal telemetry of the load balancing process.GROUPNAME > Worker:HOSTNAME > Load Balancer

Load balancing logs use different channel names to indicate the Source they come from than Worker Process logs. To search by channel name, use channel == "<source-id>_lb" instead of channel == "input:<source-id>". For example, for the built-in TCP JSON Source, enter channel == "in_tcp_json_lb" in the search box instead of channel == "input:in_tcp_json".

For convenience, the UI aggregates the Worker/Edge Node Process logs as follows.

Logfile NameDescriptionEquivalent on Logs page
N/AAggregation of all the Worker Process N logs and the API Process log.GROUPNAME > WORKER_NAME

In Cribl Stream, the logs listed above are currently available only on customer-managed hybrid Workers. The single-instance logs listed below are not relevant to Cribl.Cloud.

Single-Instance Logs ​

The API/main process emits the same logs as it does for a Distributed deployment, in$CRIBL_HOME/log/:

  • cribl.log
  • access.log
  • audit.log
  • notifications.log
  • ui-access.log

Each Worker/Edge Node Process emits the following logs in $CRIBL_HOME/log/worker/N/, where N is the Worker/Edge Node Process ID. (The metrics.log file is written only when HTTP-based Destinations exist.)

Logfile NameDescriptionEquivalent on Logs page
cribl.logMessages about the Worker/Edge Node processing data.GROUPNAME > Worker:HOSTNAME > Worker Process N
metrics.logMessages about the Worker/Edge Node’s outbound HTTP request statistics.GROUPNAME > Worker:HOSTNAME > Worker Process N

_raw stats Event Fields ​

Each Worker/Edge Node Process logs this information at a 1-minute frequency. So each event’s scope covers only that Worker/Edge Node Process, over a 1-minute time span defined by the startTime and