On This Page

Home / Stream/ Deploying Cribl Stream Software· Manual Deployment·Distributed Deployment

Distributed Deployment ​

Getting started with a distributed Cribl Stream deployment

To sustain higher incoming data volumes, and/or increased processing, you can scale from a single instance up to a multi-instance, distributed deployment. The Worker instances are centrally managed by a single Leader Node, which is responsible for keeping configurations in sync, and for tracking and monitoring the instances’ activity metrics.

To sustain high availability, Cribl Stream offers failover to an alternate Leader.

See common distributed deployment use cases in Worker Groups - What Are They and Why You Should Care.

As of version 3.0, Cribl Stream’s former “master” application components are renamed “leader.” While some legacy terminology remains within CLI commands/​options, configuration keys/values, and environment variables, this document will reflect that.

Concepts ​

Single Instance - a single Cribl Stream instance, running as a standalone (not distributed) installation on one server.

Leader Node - a Cribl Stream instance running in Leader mode, used to centrally author configurations and monitor Worker Nodes in a distributed deployment.

Worker Node - a Cribl Stream instance running as a managed Worker, whose configuration is fully managed by a Leader Node. (By default, will poll the Leader for configuration changes every 10 seconds.)

Worker Group - a collection of Worker Nodes that share the same configuration. You map Nodes to a Worker Group using a Mapping Ruleset.

Worker Process - a Linux process within a Single Instance, or within Worker Nodes, that handles data inputs, processing, and output. The process count is constrained by the number of physical or virtual CPUs available; for details, see Sizing and Scaling.

Mapping Ruleset - an ordered list of filters, used to map Workers Nodes into Worker Groups.

Options and Constraints ​

A Worker Node’s local running config can be manually overridden/changed, but changes won’t persist on the filesystem. To permanently modify a Worker Node’s config: Save, commit, and deploy it from the Leader. See Deploying Configurations below.

With an Enterprise license, you can configure role-based access control at the Worker Group level. Non-administrator users will then be able to access Workers only within those Worker Groups on which they’re authorized.

Aggregating Workers ​

To clarify how the above concepts add up hierarchically, let’s use a military metaphor involving toy soldiers:

  • Worker Process = soldier.
  • Worker Node = multiple Worker Processes = squad.
  • Worker Group = multiple Worker Nodes = platoon.

Multiple Worker Groups are very useful in making your configuration reflect organizational or geographic constraints. E.g., you might have a U.S. Worker Group with certain TLS certificates and output settings, versus an APAC Worker Group and an EMEA Worker Group, each with their own distinct certs and settings.

Architecture ​

This is an overview of a distributed Cribl Stream deployment’s components.

Distributed deployment architecture
Distributed deployment architecture

Here is the division of labor among components of the Leader Node and Worker Node.

Leader Node ​

  • API Process - Handles all the API interactions.

  • N Config Helpers - One process per Worker Group. Helps with maintaining configs, previews, etc.

Worker Node ​

  • API Process - Handles communication with the Leader Node (i.e., with its API Process) and handles other API requests.

  • N Worker Processes - Handle all the data processing.

Single-Instance Architecture ​

For comparison, here’s the simpler division of labor on a single-instance deployment, where the separate Leader versus Worker Nodes are essentially condensed into one stack:

  • API Process - Handles all the API interactions.

  • N Worker Processes - Handle all data processing,

  • One of the Worker Processes is called the leader Worker Process. (Not to be confused with the Leader Node.) This is responsible for writing configs to disk, in addition to data processing.

So here, the API Process handles the same responsibilities as a Leader Node’s API Process, while the Worker Processes correspond to the Worker Nodes’ Worker Processes. The exception is that one Worker Process does double duty, also filling in for one of the Leader Node’s Config Helpers.

Leader Node Requirements ​

  • OS:

    • Linux: RedHat, CentOS, Ubuntu, AWS Linux (64bit).

  • System:

    • +4 physical cores, +8 GB RAM.
    • 25 GB free disk space.

  • Git: git must be available on the Leader Node. See details below.

  • Browser Support: Firefox 65+, Chrome 70+, Safari 12+, Microsoft Edge

We assume that 1 physical core is equivalent to 2 virtual/hyperthreaded CPUs (vCPUs). All quantities listed above are minimum requirements.

We recommend deploying the Leader on stable, highly available infrastructure, because of its role in coordinating all Worker instances.

Worker Node Requirements ​

Port Requirements ​

During installation of any Leader or Worker instance, firewalls on that instance’s host must enable outbound communication to https://cdn.cribl.io on port 443.

Other ports must remain open continuously for a Distributed deployment to function.

If any of this traffic must go through a proxy - except for cluster communication, which can’t be proxied - see System Proxy Configuration for configuration details.

Installing on Linux ​

See Single-Instance Deployment, as the installation procedures are identical.

Version Control with git ​

Cribl Stream requires git (version 1.8.3.1 or higher) to be available locally on the host where the Leader Node will run. Configuration changes must be committed to git before they’re deployed.

If you don’t have git installed, check here for details on how to get started.

The Leader node uses git to:

  • Manage configuration versions across Worker Groups.
  • Provide users with an audit trail of all configuration changes.
  • Allow users to display diffs between current and previous config versions.

Setting up Leader and Worker Nodes ​

This section covers:

  1. Configuring a Leader Node.
  2. Configuring a Worker Node.

Configuring a Leader Node ​

You can configure a Leader Node through the UI, through the instance.yml config file, or through the command line.

Using the UI ​

At Settings > Distributed Settings > General Settings, select Mode: Leader.

Next, on the Leader Settings left tab, confirm or enter the required Leader settings (Address and Port). Customize the optional settings if desired. Then click Save to restart.

In Cribl Stream 4.0.3 and above, if you later configure Leader High Availability/Failover, further Distributed Settings changes will be locked in each Leader’s UI. You can still update distributed settings by modifying configuration files, as outlined on the page linked above.

UI Access ​

This useful option enables you to click through from the Leader’s Manage Worker Nodes page to an authenticated view of each Worker’s UI. The instructions below correspond to enabling the groups.yml file’s workerRemoteAccess configuration key.

To enable (remote) UI access to Workers from the Leader’s UI:

  1. From Cribl Stream’s top nav, click Manage > Groups.

  2. On the Groups tab: For each desired Worker Group, toggle UI Access to On.

    Worker UI access setting
    Worker UI access setting

  3. Select the Workers tab.

  4. On the Manage Workers page: Click the link for any Worker you want to inspect.

    To confirm that you are remotely viewing a Worker’s UI, Stream will add a magenta border, with a badge labeled Viewing host: <host/GUID>.

    At the upper right, you can click Restart to restart the Worker, or click the X close button to return to the Manage Workers page. Note that any changes you make here will not be propagated to the Leader.

Authenticated view of a Worker
Authenticated view of a Worker

Any changes that you make here will not be propagated to the Leader.

Prior to Cribl Stream 3.4, Worker UI access was a global setting per deployment. If you upgrade from a pre-3.4. version to v.3.4 or higher, regardless of your prior configuration, (Remote) UI access will initially be set to No for all Worker Groups. Re-enable access for each desired Group, as shown above.

Using YAML Config File ​

In $CRIBL_HOME/local/_system/instance.yml, under the distributed section, set mode to master:

$CRIBL_HOME/local/_system/instance.yml
distributed:
  mode: master
  master:
    host: <IP or 0.0.0.0>
    port: 4200
    tls:
      disabled: true
    ipWhitelistRegex: /.&ast;/
    authToken: <auth token>
    enabledWorkerRemoteAccess: false
    compression: none
    connectionTimeout: 5000
    writeTimeout: 10000
Using the Command Line ​

You can configure a Leader Node using a CLI command of this form:

./cribl mode-master [options] [args]

For all options, see the CLI Reference.

Configuring a Worker Node ​

On each Cribl Stream instance you designate as a Worker Node, you can configure the Worker through the UI, the instance.yml config file, environment variables, or the command line.

Using the UI ​

At Settings > Global Settings > Distributed Settings > General Settings, select Mode: Managed Worker (managed by Leader).

Next, on the Leader Settings left tab, confirm or enter the required Address (e.g., criblleader.mycompany.com). Customize the optional settings if desired. Then click Save to restart.

If you need to customize the Leader’s port (from its default 4200), you can do so via the CLI’s mode-worker command.

Using YAML Config File ​

In $CRIBL_HOME/local/_system/instance.yml, under the distributed section, set mode to worker:

$CRIBL_HOME/local/_system/instance.yml