On This Page

Home / Stream/ Sources/ Collector Sources/Script Collector

Script Collector ​

Cribl Stream supports flexible data collection configured by your custom scripts.

In Cribl.Cloud, the Script Collector is only available on hybrid, customer-managed Worker Nodes.

How the Collector Pulls Data ​

When you run a Script Collector in Discovery mode, the first available Worker returns one line of data per discovered item. Each item (line) turns into a Collection task on the Leader Node.

In Full Run mode, the Leader passes the item to collect into the script in the $CRIBL_COLLECT_ARG variable, and spreads collection across all available Workers.

If you run the Script Collector with a given Time Range, the earliest and latest values will be available to the script via the environment variables EARLIEST and LATEST, respectively.

Script Caveats ​

Scripts will allow you to execute almost anything on the system where Cribl Stream is running. Make sure you understand the impact of what you’re executing before you do so! These scripts run as the user running Cribl Stream, so if you are running it as root, these commands will run with root user permissions. ☠️ ☠️

Scripts must finish on their own. The Collector will not stop a script once its task has started. You must ensure scripts do not use infinite loops.

Configuring a Script Collector ​

From the top nav, click Manage, then select a Worker Group to configure. Next, select Data > Sources, then select Collectors > Script from the Manage Sources page’s tiles or left nav. Click Add Collector to open the Script > New Collector modal, which provides the following options and fields.

The sections described below are spread across several tabs. Click the tab links at left to navigate among tabs. Click Save when you’ve configured your Collector.

Collector Sources currently cannot be selected or enabled in the QuickConnect UI.

Collector Settings ​

The Collector Settings determine how data is collected before processing.

Collector ID: Unique ID for this Collector. E.g., sh2GetStuff. If you clone this Collector, Cribl Stream will add -CLONE to the original Collector ID.

Discover script: Script to discover which objects/files to collect. This script should output one task per line in stdout. Discovery is especially important in a distributed deployment, where the Leader must track all tasks, and must guarantee that each is run by a single Worker. See Examples below.