These docs are for Cribl Stream 4.10 and are no longer actively maintained.

See the latest version (4.20).

Learn about issues affecting Cribl Stream and/or Cribl Edge. Find out if they’re already fixed, or discover temporary workarounds.

In the tables below, “Reported in” refers to the product version in which the given issue was first recorded. In some cases, you might encounter that issue in an older version as well.

• Unresolved
• Resolved in 4.10

See also Known Issues resolved in older product versions:

• Resolved in 4.9
• Resolved in 4.8
• Resolved in 4.7
• Resolved in 4.6
• Resolved in 4.5
• Resolved in 4.4
• Resolved in 4.3
• Resolved in 4.2
• Resolved in 4.1
• Resolved in 4.0

Unresolved ​

Error saving Pipeline with Publish Metrics Function set to Counter when Name Expression is blank ​

Problem: When using the Publish Metrics Function with a Counter metric type without specifying a Metric Name Expression, attempting to save the Pipeline results in an error: Cannot read properties of null (reading 'startsWith').

Workaround: Provide a Metric Name Expression when configuring the Publish Metrics Function to avoid this error. For example, set the Metric Name Expression to _metric to use the field name as the metric name.

Some Job Settings for Purging Artifacts Are Not Configurable ​

Problem: Cribl Stream disregards customizations to certain job limits at the Worker Group level. As a result, ingest might appear to complete successfully, even though Cribl Stream has prematurely deleted some retrieved files.

Workaround: None identified.

Cribl Edge Subfleet Member and Team Mapping ​

Problem: Mapping Subfleets to specific Members and Teams is not supported. As a result, when you attempt to access the Members and Teams option within the More options menu for a Subfleet, you will encounter a 404 error.

Workaround: None identified.

Erroneous FIPS mode detection in Cribl Edge on Windows ​

Problem: Cribl Edge on Windows machines might encounter issues booting up when the OPENSSL_MODULES and OPENSSL_CONF environment variables are set.

Workaround: Use the Windows Registry to set the following environment variables to empty strings in the environment where the Cribl Edge service runs. For example:

CRIBL_VOLUME_DIR=C:\ProgramData\Cribl\
OPENSSL_CONF=
OPENSSL_MODULES=

Note: Upgrading Cribl Edge or running MSI repair will revert the workaround implemented in the registry. You will need to reapply the changes after the upgrade or repair.

Encrypted secrets are broken when using encoding other than latin1 ​

Problem: Secrets that contain characters in encodings other than latin1 will return an incorrect value when decoded.

Workaround: Make sure your secrets use latin1 encoding.

When you teleport into a Windows Edge Node and navigate to Health, the CPU Load Average panel displays no results. ​

Problem: In Windows deployments of Cribl Edge, the CPU Load Average panel is blank when you teleport into a Node and view the Health page.

Workaround: None identified.

Dashboard UI charts missing metrics after application update ​

Problem: After upgrading the application, dashboard charts may fail to display metrics recorded prior to the upgrade. This concerns the following charts:

• Events In and Out
• Bytes In and Out
• Free Memory
• CPU Load Average

This issue is intermittent and only affects visualization in the dashboard charts. Underlying data is flowing correctly.

Workaround: None identified.

Empty directories that contained Parquet files are not deleted ​

Problem: When certain Sources and Collectors ingest or collect Parquet files, they create directories for storing those files. After they send and delete the files, they should also delete the directories, but fail to do that, potentially consuming large numbers of inodes on the storage volume. Sources affected: Azure Blob Storage, Amazon S3. Collectors affected: Amazon S3, Filesystem/NFS, Google Cloud Storage.

Workaround: Configure a cron job to delete the empty directories.

After restarting the File Monitor Source, Event Breakers aren’t processing CSV files correctly when the File Monitor Source is configured to collect CSV logs ​

Problem: When the File Monitor Source is configured to collect CSV file logs and use CSV Event Breakers, subsequent event breaking is not working when the Source resets. The Event Breaker expects the first line of column headers, but they resume reading from a later point in the file where those headers aren’t present.

Workaround: Use the default line breaker and a pre-processing Pipeline to drop lines that start with #. Optionally, use an Eval function to split _raw values on the column-separator character into separate fields and rename them as needed.

With Leader HA/Failover, creating diag bundles via CLI produces incorrect results ​

Problem: For deployments configured for Leader high availability/failover, creating diagnostic bundles via the UI works normally but creating them via the CLI does not. In Cribl Stream 4.4.0 and newer, the operation fails with a not a git repository error. In 4.4.0 and older, the diag bundle created contains obsolete data. In both cases the underlying cause is that the process that creates the diag bundle cannot access the CRIBL_CONF_DIR environment variable.

Workaround: Use the UI to create the diag bundle. Or, if you use the CLI, preface the diag command with the needed environment variable, like this: CRIBL_CONF_DIR=/<path_to_failover_volume> cribl diag create. If you have an instance.yml config file, you can find the path to the failover volume in the distributed section.

Fix: Error message improvements were made as part of CRIBL-21258. Fix for CRIBL-21557 is TBD.

Migrated Local Users appear in the Members UI with “No Access” ​

Problem: Local Users are incorrectly shown in the Settings > Members UI with No Access. However, their Roles still function as originally configured, and still display correctly at Settings > Global Settings > Access Management > Local Users.

Workaround: Rely on the Local Users UI.

Fleet Upgrade Errors ​

Problem: A Fleet’s upgrade from a Leader can result in errors when the Edge Nodes use different host/port/tls settings than Worker Nodes.

Workaround: The Edge Nodes and Workers must connect to the Leader using the host/port/tls connection details. If this is not possible, upgrade Edge Nodes separately using the bootstrap script.

Datadog Agent Source fails to receive metrics ​

Problem: When a Datadog Agent using Datadog API v2 sends metrics, Datadog Agent Source fails to receive them, because the Source uses Datadog API v1. Datadog Agent logs will show API Key invalid, dropping transaction errors. Debug logging will likely produce Dropping request because of unallowed path message errors with statusCode: 403, which Datadog Agent interprets as “invalid API key.”

Workaround: In the datadog.yaml file, ensure that use_v2_api.series is set to false. Cribl Stream’s Datadog Agent Source will then receive metrics normally, since both the Source and the Datadog Agent will be using Datadog API v1, which Datadog still supports.

Cannot clear Messages drawer while in GitOps Push mode ​

Problem: When in GitOps Push mode, you cannot clear messages from the Messages drawer. Cribl Stream returns a Forbidden error.

Workaround: None identified.

Amazon Kinesis Data Streams Source resumes reading at earliest event when Leader Node is down ​

Problem: After the Worker on which it is running restarts, the Kinesis Streams Source should resume reading a shard from the point where it left off. That point is a sequence number obtained from the Leader Node - but if the Leader Node is down, sequence numbers (state) will not be available. In this situation, the Worker should wait for the Leader Node to come back up, obtain the state, and then resume reading from the correct point in the shard.

However, due to this issue, the Worker does not wait for the Leader to come back up if the Leader stays down for over one minute. Instead, the Worker starts where Optional Settings > Shard iterator start says it should.

• If Shard iterator start is set to its default of Earliest record, the Worker will resume reading from the beginning of the shard, potentially collecting events it already collected earlier, producing duplicate events.
• If Shard iterator start is set to Latest record, the Worker will resume reading from the end of the shard, potentially missing events if the last event it collected before the restart is not the last event in the shard.

Workaround: None identified.

Recently added Worker Nodes fail to appear on the Monitoring page ​

Problem: When Worker Nodes are added, there may be a delay before the Worker count is updated on the Monitoring page.

Workaround: Refresh your web browser.

Default commit message missing for non-admin users ​

Problem: For users who have Roles as high as owner_all, but not admin, the Commit modal fails to display any Default commit message saved in Git Settings.

Workaround: Enter (or paste) a message per commit.

Diag bundles might fail to download from teleported Worker Nodes ​

Problem: When you’re remotely accessing a Worker Node’s UI, diag bundles might fail to download from Global Settings > Diagnostics.

Workaround: Refresh the page and Export the bundle again. Alternatively, log directly into the Worker Node’s UI before creating the diag.

Kubernetes Logs Source doesn’t collect logs from exited or short-lived containers ​

Problem: The Kubernetes Logs Source collects logs only from running containers, which can lead to missed logs from short-lived containers or those that exit before they start running.

Workaround: None identified.

Pipelines with Clone and other Functions can show inconsistent total processing times ​

Problem: For Pipelines containing both a Clone Function and an async Functions (like Redis), the Pipeline Diagnostics modal’s Pipeline Profile > Process Time graph will sum up the duration of all Functions’ processing times. This sum can exceed the Pipeline’s total duration displayed in the Summary.

Workaround: None identified.

Splunk HEC shows higher outbound data volume than other Splunk Destinations ​

Problem: Events sent to the Splunk HEC Destination will show higher outbound data volume than the same events sent to the Splunk Single Instance or Splunk Load Balanced Destinations, which use the S2S binary protocol.

Workaround: None identified.

Persistent queue with compression requires oversize max queue limit ​

Problem: If you configure persistent queueing to both enable Compression and set a Max queue size limit, set that limit optimistically and monitor the results. Due to an error in computing the compression factor, Cribl Stream will not fully use Max queue size values below or equal to the volume’s available disk space. This can lead to a mostly empty disk, lost data (either blocked or dropped), and/or excessive backpressure.

Workaround: With Compression enabled, set the Max queue size to a value higher than the volume’s total available physical disk space (disregarding compression). Alternatively, leave Max queue size empty, to impose no limit. Monitor overall throughput and the queue size (if PQ engages), to verify that Cribl Stream is maximizing the queue.

GitOps + License expiration = Catch-22 ​

Problem: If your Enterprise license expires while you have enabled the GitOps Push workflow, you will encounter the following block: Cribl Stream is in read-only mode, triggering a Forbidden error when you try to update your license key. But you also cannot reset the workflow from Push to None, because the expired license disables GitOps features.

Workaround: Contact Cribl Support for help updating your license.

Syslog Source’s previewed data doesn’t proceed through Routes ​

Problem: The Syslog Source’s General Settings > Input ID field’s default filter expression is __inputId.startsWith('syslog:in_syslog:'). The same filter appears in the Preview Full tab’s Entry Point drop-down, except without the final colon. This means that data sent using Preview Full’s version of the filter will not go through Routes that use the Input ID version.

Workaround: When sending data from Preview Full to a Route, if both use the syslog:in_syslog filter, edit the Route’s filter to remove the final colon. This will make the filter identical in both places, routing data correctly.

Upgrading a Pack overwrites Lookup and sample-data customizations ​

Problem: Upgrading a Pack overwrites any customizations you’ve made to the Pack’s original/default Lookup tables and sample-data files.

Workaround: Duplicate the Pack’s default Lookup tables and sample-data files, and customize your duplicate copies. Upgrading the Pack will not overwrite your local copies.

Re-enabling a Function group mistakenly re-enables all its Functions ​

Problem: When you change a Function group from disabled to enabled, all of its Functions are enabled, regardless of their individual enabled/disabled states when the group was disabled.

Workaround: Avoid disabling and re-enabling Functions as a group (e.g., for testing or stepwise debugging purposes).

Duplicate Workers/​Worker GUIDs ​

Problem: Multiple Workers have identical GUIDs. This creates problems in Monitoring, upgrading and versioning, etc., because all Workers show up as one.

Cause: This is caused by configuring one Worker and then copying its cribl/ directory to other Workers, to quickly bootstrap a deployment.

Workaround: Don’t do this! Instead, use the Bootstrap Workers from Leader endpoint.

If you already have existing duplicate GUIDs, you can resolve the issue by stopping all Cribl Stream/Edge instances that have a duplicate and then deleting the $CRIBL_HOME/local/cribl/auth/676f6174733432.dat file before restarting the app. The system will generate a new GUID at startup.

Can’t Enable KMS on Worker Group after installing license ​

Problem: Enabling HashiCorp Vault or AWS KMS on a Worker Group, after installing a LogStream license on the same Group, fails with a spurious External KMS is prohibited by the current license error message.

Workaround: On the Leader, navigate to Settings > Worker Processes. Restart the affected Worker Group’s CONFIG_HELPER process. Then return to that Worker Group’s Security > KMS Settings, re-enter the same KMS configuration, and save.

Resolved in 4.10 ​

The following issues were resolved in version 4.10 and its maintenance releases.

Potential EMFILE Errors and Incorrect Signal Emitter Location on Cribl.Cloud ​

Problem: You might encounter the error Error: EMFILE: too many open files, watch <filename> in Cribl.Cloud. This often happens when the system runs out of resources for tracking file changes. It could be related to an underlying issue where our system’s signal emitter points to the wrong location.

Workaround: To resolve the EMFILE error, increase the system’s file watcher limits. Edit the /etc/sysctl.conf file and add or modify these lines:

fs.inotify.max_user_instances=8192
fs.inotify.max_user_watches=524288

Windows Event Logs (WEL) Sources on Windows Server 2016 and 2019 might not consume all ETL log events ​

Problem: On Windows Server 2016 and 2019, Windows Event Logs Sources that are configured to consume ETL logs might not consume all events in the ETL log.

Workaround: In the Windows Event Logs Source, Advanced Settings, toggle Use Windows Tools on.

Windows Event Logs (WEL) Source Performance Issues on Windows Servers (2016 and 2019) ​

Problem: The WEL Source on Windows Servers (2016 and 2019) might receive delayed events, due to the operating system’s inaccurate updates to event log modification times. This affects the Source’s ability to efficiently detect new events, leading to delays.

Workaround: In the Source configuration modal, on the Advanced Settings tab, enable Use Windows Tools to resolve delays.

Configuring Edge Add-on for Amazon EKS fails ​

Problem: The documented procedure for configuring Cribl Edge Add-on (available in Amazon Marketplace) via EKS Settings & AWS CLI is no longer functional and results in errors.

Workaround: None identified.

Scheduled jobs in Single-instance mode do not resume when orphaned ​

Problem: In Single-instance mode of Cribl Stream, orphaned scheduled jobs do not resume, resulting in stalled execution. This issue does not affect deployments in Distributed mode.

Workaround: None identified.

Cribl server fails to start with AWS KMS and AssumeRole enabled ​

Problem: When using AWS KMS with AssumeRole enabled, the Cribl server fails to start due to an error when accessing the getCredentialsAsync function. The error is logged as boot failed, checking for rollback in the cribl.log file.

Workaround: None identified.

Stopping a container does not initiate clean shutdown ​

Problem: Stopping a container does not issue a graceful shutdown signal SIGTERM to the Cribl process. Containers apply their own timeouts for shutdown, and these override any application-level shutdown timeouts.

Workaround: None identified.

Resolved in 4.9 ​

The following issues were resolved in version 4.9 and its maintenance releases.

Windows Event Logs Source (Cribl Edge): imprecise seek position ​

Problem: Within the Windows Event Logs Source, a Windows-native module incorrectly interprets the seek position argument as a 32-bit integer, leading to problems seeking to the correct position in the event log. This can lead to duplicate events, longer-than-expected collection times, and other issues.

Workaround: In the Source configuration modal, on the Advanced Settings tab, enable Use Windows Tools. While this workaround can resolve the seek position problem, it might degrade performance, especially for high-volume logs.

Significant increase in malformed HEC event errors ​

Problem: A change to HEC parsing in 4.9.1 might cause an increase in Dropping malformed HEC event, enable debug to see details error logs, even when the incoming HEC payloads are known to be valid. Specifically, this can happen in JSON with multi-escaped quote characters (for example, "type\\\\\\\":\\\\\\\"someType\\\\\\\"). When this error occurs, Cribl Stream will not ingest the payload and will instead return a 400 error response.

Workaround: None identified. Users are advised to upgrade directly to 4.9.2 or later to avoid this issue.

Worker Nodes or Edge Nodes send false down alerts after turning off load balancing ​

Problem: This issue affects Worker Nodes or Edge Nodes with load balancing enabled in a distributed environment. When you disable load balancing, commit and deploy the configuration changes, and restart the Nodes, Cribl may send a false alert indicating the Nodes are down. Despite the alert, the Nodes remain healthy and operational.

Workaround: None identified.

Unable to upgrade or install Edge to 4.9.0 on Windows machines running .NET version older than 4.7.2 ​

Problem: Users are not able to install or upgrade to Edge 4.9.0 on Windows machines that are running .NET framework versions older than 4.7.2.

Workaround: Upgrade instances of .NET on your Windows machine to 4.7.2 or newer, then rerun the Cribl Edge installation or upgrade.

REST Collectors query unexpected pages with dual pagination ​

Problem: A REST Collector configured to use pagination for both the discovery and collection steps could collect the wrong page. The field used to track the current page for the discover step interferes with the page tracking for the collect step. This causes the collection step to query unexpected pages, leading to inconsistent results.

Workaround: None identified.

Cannot use some words when accessing properties in Pipelines ​

Problem: When Pipelines refer to property names containing certain words, an error message occurs: Failed to build property accessor (...) err=Unallowed ref to '<keyword>'. The words causing issues are: AsyncFunction, AsyncGenerator, AsyncGeneratorFunction, Function, GeneratorFunction, Promise, Proxy, Reflect, Symbol, WebAssembly, console, eval, global, globalThis, process, setInterval, setTimeout, uneval, window.

Workaround: When using any of the above words as property names, surround them by quotation marks, for example: process.part.part -> 'process.part.part'.

Cribl Copilot for Pipelines does not work if accessed from Stream Projects ​

Problem: After upgrading to Cribl 4.8.2, the “Add function with Copilot” feature does not work when accessed from Projects > Data Projects > Pipelines.

Workaround: Access the “Add function with Copilot” feature in Cribl Stream by going to Processing > Pipelines > select a Pipeline > Add Function.

Google Cloud Pub/Sub Source not populating attributes field in events ​

Problem: The attributes from a PubSub message do not translate to the Cribl event because the message.attributes property from the PubSub.Message type is treated as an array instead of an object, resulting in a missing __attributes field.

Workaround: None identified.

REST Collector configuration no longer supports the ability to add placeholder values ​

Problem: When you configure a REST Collector as a JSON file, you can normally define a placeholder value for various fields. Then, when you switch to the user interface, it prompts you to fill in the placeholders with the actual values. This feature is missing in Cribl Stream 4.9.0.

Workaround: None identified.

Can’t upgrade Cribl products from UI when CRIBL_VOLUME_DIR environment variable is set ​

Problem: On Leader, Cribl Stream, or Cribl Edge (Linux) instances that have the CRIBL_VOLUME_DIR environment variable set, you can’t upgrade from 4.8.2 to 4.9.0 using the UI. This affects only upgrades from version 4.8.2.

These scenarios are not impacted by this issue:

• Upgrading from version 4.8.1, or older, to 4.9.0.
• Installations where CRIBL_VOLUME_DIR is not set.
• Windows installations of Cribl Edge.
• Containers or RPM installs.

Workaround: Upgrade these instances from the command line. See Upgrading Cribl Edge on Linux (CLI) for instructions.

The Cloud persistent queue (PQ) health checker sends false alerts for staging environment with only one Worker Node ​

Problem: When a staging environment has only one Worker Node, the Cloud PQ health checker creates a critical alert with the message “cloud worker pq config errors”.

Workaround: None identified.

RangeError: offset is out of bounds and nested field serialization issues in Splunk Load Balanced Destination with S2Sv4 ​

Problem: The Splunk Load Balanced Destination using S2Sv4 encounters issues when processing certain events:

1. When events contain a field with an array value with many items, parsing could fail with the error message RangeError: offset is out of bounds.
2. When events contain a field with a non-stringified object value greater then 1 KB, and Nested field serialization is set to JSON, the value could be truncated after stringification. The event is still sent but may miss some of the stringified object value.

Workaround:

1. Smaller-sized arrays may help.
2. Serialize nested fields as JSON before they reach the Splunk Destination with the Eval function.

Deploying configurations to Edge Subfleets sometimes fail with TLS enabled ​

Problem: In a Cribl-managed instance of Edge, attempting to deploy configuration changes to a Subfleet sometimes fails if the Subfleet, or its parent Fleet, has TLS enabled and contains a certificate with a passphrase.

Workaround: There are two workaround options, in the Fleet or Subfleet settings:

• Use a certificate without a passphrase.
• Disable TLS in API Server Settings.

HMAC authentication fails due to empty signature strings ​

Problem: HMAC authentication could fail when an API requires empty strings in the signature string builders. Cribl currently filters out any falsy values, including empty strings, which can prevent the correct generation of the HMAC value and lead to authentication failures.

Workaround: Varies. See a workaround example in our templates repository.

Persistent queue settings are missing in hybrid Worker Groups ​

Problem: On Cribl.Cloud, persistent queue settings are unavailable in the UI for Sources and Destination running in hybrid (customer-managed) Worker Groups.

Workaround: You can configure persistent queues on hybrid Worker Groups by using the Manage as JSON option in Sources or Destinations.

A Google Cloud Pub/Sub Source authentication tooltip gives incorrect Google authentication information ​

Problem: When you are configuring the Google Cloud Pub/Sub Source, the Authentication screen has a field named Authentication method with an incorrect tooltip. Starting in 4.8.2, you should no longer use the environmental variables PUBSUB_PROJECT and PUBSUB_CREDENTIALS to authenticate with Google, as stated in the tooltip.

Workaround: In 4.8.2 or later, use the GOOGLE_APPLICATION_CREDENTIALS environment variable to provide the location of a credential JSON file or the JSON file contents. When this option is set, Cribl Stream will first check whether the GOOGLE_APPLICATION_CREDENTIALS environment variable exists and properly set with valid credentials. If this environmental variable is missing or incorrect, Cribl Stream checks for credentials in two additional fallback locations, as described in the official Google Cloud documentation: How Application Default Credentials works.

C.Time Functions can crash Worker Nodes with old timestamps ​

Problem: The C.Time.s3TimePartition() and C.Time.timePartition() Functions, commonly used for S3 and Cribl Lake Destinations, are susceptible to crashing Worker Processes when encountering events with timestamps prior to January 1st, 1970 (Unix epoch). These events, represented by negative (or 0) values in the _time field, cause Worker Nodes to consume 100% CPU and become unresponsive.

Workaround: To prevent Worker Nodes crashes, insert a Pipeline before any Function or Destination using these methods. Configure the Pipeline to modify the _time field of affected events to a positive value (>0) (e.g., using the Date.now() function).

Unable to Load CPU Profiles in Cribl.Cloud ​

Problem: Cribl.Cloud is currently experiencing an issue where users cannot view or download CPU profiles directly through the UI. CPU profiling is useful in troubleshooting performance issues.

Workaround:

1. Open DevTools: While on the Worker Process details page, right-click anywhere and select Inspect.
2. Navigate to the Network Tab: Locate the Network tab within DevTools.
3. Filter Network Requests: In the filter bar at the top of the Network tab, type download. This will filter the requests to only show those related to CPU profile downloads.
4. Identify API Call: Look for the latest network request with a path containing download (e.g., /api/v1/workers/<worker_id>/profile/download). This is the backend API call that fetches the CPU profile data.
5. Access Profile Data: Right-click on the identified API call and select Open in Sources panel.
6. Save Profile: Copy this data and save it to a file with a .cpuprofile extension.

For profile visualization, consider using an open-source tool like Speedscope after saving the profile data. A more permanent solution involving UI updates is planned for a future release.

REST Collector state not updated on job cancellation ​

Problem: Canceling a REST Collector job from the Job Inspector before it completes prevents state updates for events collected prior to cancellation. As a result, no state is saved for uncompleted tasks within the collection run.

Workaround: None identified.

When Source PQ is enabled, disk space fills to near capacity and Cribl Stream issues an error ​

Problem: With Source-side persistent queues (PQs) enabled, the system is not releasing files correctly, causing the disk space to fill and potential Worker Node failures. In addition, the system frequently logs the following error: Cannot read properties of undefined (reading 'on').

Workaround: You may be able to mitigate this issue by adjusting your Source-side PQ settings. Significantly decreasing the Max buffer size and increasing the Max file size may resolve the issue.

Chain Functions in Cribl Stream Projects can lead to infinite loops ​

Problem: Currently, users can create infinite loops within a project by referencing a Pipeline within itself using a Chain Function.

Workaround: Until a permanent fix is implemented, users should avoid creating circular dependencies within Pipelines within a single project. Do not reference a Pipeline within itself using a Chain Function.

Upgrading to 4.7.0 with Google Cloud Security Operations Destination (formerly Google Chronicle) ​

Problem: When upgrading to Cribl Stream 4.7.0, if an existing Google Chronicle Destination (now Google Cloud Security Operations) is configured on a particular Worker Group, all Destinations in that Worker Group will fail to load and process data until a commit and deploy is performed on the Worker Group.

If you use the Google Chronicle Destination:

Recommended: Proceed with the upgrade, but be prepared to immediately perform a commit and deploy on affected Worker Groups after upgrading Worker Nodes and Leaders. Worker Nodes will only resume data processing after a successful commit and deploy.

Workaround: None identified.

Licensing page displays a permission access warning when a Cloud Connection is enabled ​

Problem: When you have an on-prem Cloud Connection to Cribl.Cloud and navigate to the Licensing page, a warning displays that the user does not have permission to access the resource. You won’t be able to access the license page with a connected environment (as your Cribl.Cloud account now handles credits for the connected environment) but it is not a permissions issue.

Workaround: To view billing information for your connected environments, log into your Cribl.Cloud account and go to Organization, then Billing & Usage.

OTLP Metrics Function does not process metrics unless all metric names are quoted literals ​

Problem: When the Publish Metrics Function is followed by the OTLP Metrics Function in a Pipeline, the OTLP Metrics Function may fail to produce any results. This failure is possible when the metric name expression for any of the published metrics uses an unquoted string literal and/or evaluates to a numeric value.

Workaround: If you must use a metric name expression in Publish Metrics, wrap the results of the expression in single or double quotes to ensure that it returns a string.

Optionally, leave the metric name expression blank and use the Rename Function to rename the metric names before sending them to the OTLP Metrics Function.

A File Monitor memory leak causes high memory usage only on Windows Edge Nodes ​

Problem: When Cribl Edge is deployed on a Windows Node and the File Monitor Source is configured on that Node, a memory leak will cause high memory usage on the Windows Edge Node.

Workaround: None identified.

Kafka and Confluent mTLS requirement limits Schema Registry authentication ​

Problem: The Kafka and Confluent Cloud Source and Destination only support mutual TLS (mTLS) authentication for the Kafka Schema Registry. This prevents you from configuring separate authentication mechanisms for the Schema Registry and Kafka brokers, which is necessary when the Schema Registry and brokers are managed by different departments with distinct authentication requirements.

Workaround: None identified.

Healthy Destinations spuriously report Blocked Status ​

Problem: Healthy TCP-based, Kafka-based, Splunk, and Metrics Destinations can spuriously report Blocked Status in the Charts tab, sometimes also logging the blocked status inaccurately.

Workaround: None identified.

Resolved in 4.8 ​

The following issues were resolved in version 4.8 and its maintenance releases.

Removing Member from Team removes access after a lag ​

Problem: As the Owner or Admin of a Cribl.Cloud Organization, if you remove a Member from a Team, that user will retain any Team-associated Permissions on products and product resources for up to 30 minutes. (This lag does not affect Team changes on ​on-prem Organizations.)

Workaround: The intended access changes will take effect within 30 minutes.

In Cribl Edge, TLS certificate paths in Sources and Destinations are incorrect ​

Problem: After Cribl Edge upgrades to 4.8.0 or later, TLS certificate paths for Cribl Edge Sources and Destinations are not updated. This may affect data flow in Fleets and Subfleets containing TLS-configured Sources or Destinations.

Workaround: Update the TLS certificate path.

1. In the parent Fleet, navigate to the Source or Destination that uses a TLS certificate.

2. In the Certificate path field, change default to local.

   For example:

   • Old path: groups/default_fleet/default/edge/auth/certs/asd.crt

   • New path: groups/default_fleet/local/edge/auth/certs/asd.crt

3. Save the Source or Destination.

4. Select Commit and Deploy with Subfleets so the Subfleets receive the change.

Splunk Destinations incorrectly detect that a downstream Splunk instance is shutting down ​

Problem: Splunk Single Instance and Splunk Load Balanced Destinations might incorrectly detect that a downstream instance of Splunk is shutting down at frequent intervals. This only affects Destinations with Max S2S version set to v3 and Minimize in-flight data loss toggled on.

Workaround: Change the Max S2S version setting for the Destination to v4. Or, toggle Minimize in-flight data loss off.

Incoming data from Collectors sometimes incorrectly displays UTF-8 characters ​

Problem: Cribl Stream sometimes displays incoming data containing multi-byte UTF-8 characters incorrectly when Collectors such as AWS S3 ingest events.

Workaround: None identified.

Wiz Source does not correctly paginate ​

Problem: The Wiz Source in Cribl Stream does not correctly handle pagination. The default queries for each data type in the Wiz Source incorrectly pass the endCursor parameter, resulting in the same page of data being returned repeatedly.

Workaround: Modify each default query by changing endCursor: endCursor to after: endCursor.

Some elements are missing from JSON-formatted events collected by the Windows Event Logs Source ​

Problem: JSON-formatted events collected by the Windows Event Logs Source don’t contain the elements UserData, DebugData, BinaryEventData, and ProcessingErrorData.

Workarounds:

You can either:

• Toggle Use Windows Tools on in the Windows Event Logs Source Advanced Settings.

Or:

• Set the Event format to XML. Create a Pipeline that contains the C.Text.parseWinEvent(_raw) expression, which will analyze the incoming XML event data and generate JSON-formatted output.

Missing in-product documentation for ServiceNow Cloud Observability Destination ​

Problem: The ServiceNow Cloud Observability Destination documentation is not available in-product. Attempting to access the documentation within the product interface returns the error “Unable to load docs. Please check Cribl’s online documentation instead.”

Workaround: Refer to the online documentation for configuration instructions.

Windows Event Log events are mishandled when they’re missing provider metadata ​

Problem: Events without a valid provider are missing the _raw property. This happens in the Windows Event Logs Source when Use Windows Tools is toggled off in Advanced Settings, and Render event message strings is enabled.

Workaround: Toggle Use Windows Tools on or toggle Render event message strings off.

_raw field expansion is incomplete ​

Problem: When capturing live data, the _raw field unexpectedly expands only to the first level. It should fully expand to reveal its entire nested JSON structure when tapped.

Workaround: None identified.

Splunk TCP Source and Destination: Incorrect Max S2S Version displayed in UI ​

Problem: If a Splunk TCP Source or Destination was created before the Max S2S Version setting existed, the UI incorrectly displays v4 as the Max S2S Version, instead of the actual default v3. This discrepancy can lead to version mismatch errors when forwarders upgraded to Splunk 9.1.4 attempt to send data.

Workaround: Manually change the Max S2S Version dropdown to v3 and then back to v4 in the UI, and save the configuration. This ensures the correct version is set in the config.

Unable to Send Data via S2S to Splunk Cloud and Splunk Enterprise ​

Problem: Attempting to send data to recent versions of Splunk Cloud or Splunk Enterprise via Cribl Stream’s Splunk Single Instance or Splunk Load Balanced Destinations integrations may encounter issues after upgrading Splunk.

Workaround: Use the Splunk HEC Destination to send traffic to Splunk. HEC is the recommended protocol for sending data to Splunk due to its better load balancing, compression, and overall performance.

Jobs stuck in “pending” status ​

Problem: Jobs can get stuck in Pending state when you commit and deploy configuration while the job is running. This prevents the job from running again, and if multiple jobs freeze, the job queue can fill up. Note that Worker upgrade jobs may fail when upgrading from versions earlier than 4.8.

Workaround: Restart the affected Worker and the Leader. To restart the Worker:

1. Log into Cribl Stream as an Admin.
2. Go to the Workers page.
3. Select the affected Worker’s GUID. You will teleport to this Worker’s UI.
4. On the colored top bar, select Restart Stream.

To restart the Cribl.Cloud Leader:

1. Get an API token for your Leader Node.

2. Use the following curl command to restart the Leader, after replacing <organizationID> with your Cribl.Cloud organization ID and $token with the API token for your Leader Node:

   curl -X POST 'https://<workspace-org>.cribl.cloud/api/v1/system/settings/restart' \
   -H 'Authorization: Bearer <token>' \
   -H 'Content-Type: application/json' \
   -d '{}'

   Copy

Incorrect Source URL in REST Collector events ​

Problem: The REST Collector incorrectly displays the Discovery URL as the source field in collected events, instead of the actual Collect URL. This issue occurs when using OAuth2/token API for authentication.

Workaround: Manually adjust the source field in downstream processing or use a custom script to update the field after data collection.

Stream Projects: Post-processing Pipelines defined in Destination do not work ​

Problem: When using Cribl Stream Projects, a post-processing Pipeline defined only in the Destination, not in the Project itself, will be ignored.

Workaround: Copy the post-processing Pipeline inside the Project:

1. In Cribl Stream’s Processing > Pipelines table, click the Pipeline you’re using for post-processing.
2. Select the Pipeline’s gear (⚙️) button, and then the “Manage as JSON” button at the top right of the Pipeline settings.
3. Select Export and save the Pipeline in JSON format in a convenient location.
4. Go back to the Project screen and select Pipeline at the top.
5. Select Add Pipeline, and then Import from File.
6. Choose your exported configuration and confirm.

Post-processing should now function correctly in your Project.

Already-existing Webhook Destinations show unexpected UI behavior upon upgrade to 4.4.4 ​

Problem: If you have a Webhook Destination in a Cribl Stream deployment, and then upgrade that deployment to version 4.4.4, the UI will show an error; the new Load balancing toggle will be turned on with associated settings displayed; and the UI will not allow you to save changes.

Workaround: First, toggle Load balancing off. This will return the UI to normal behavior. Then, if you want to use the Destination without load balancing, configure as usual and save. Or, if you prefer, update the Webhook URLs to include one or more valid endpoints you want to send to, and save the Destination with Load balancing enabled.

Resolved in 4.7 ​

The following issues were resolved in version 4.7 and its maintenance releases.

Splunk HEC Destination drops events when _raw is an empty string ​

Problem: When sending events to a Splunk HEC Destination, if the _raw field is an empty string, Cribl Stream sends an empty string as _raw instead of trying to send the full serialized event. This results in an error from Splunk and the event is dropped.

Workarounds:

• Do not set _raw to an empty string in any Pipelines feeding a Splunk HEC Destination.
• If you do not know or cannot change the _raw contents prior to the Destination, add an Eval Function to a post-processing Pipeline with the expression: _raw = _raw?.length ? _raw : undefined;.

Splunk HEC Destination drops events with null _time values ​

Problem: Events sent to a Splunk HEC Destination with a null _time field are dropped by Splunk, halting further processing of the entire payload (potential data loss).

Workaround: Assign the _time field to the current time in a post-processing Pipeline by adding an Eval Function before the Splunk HEC Destination with the expression: _time === null ? Date.now() / 1000 : _time.

Group and Fleet Naming Conflict ​