On This Page

Home / Stream/ Integrations/ Manage Backpressure/Persistent Queue Shared Storage

Persistent Queue Shared Storage ​

By default, persistent queues (PQs) write to each Worker Node’s local disk. If that Worker Node goes down before the queue drains, the data stays on that Worker Node until the same Worker Node returns.

Shared PQ storage configures a shared backing store for all Worker Nodes in a Worker Group. That store is either Network filesystem (NFS) or AWS S3. All Worker Nodes can read queue data on that store. When a Worker Node disappears, surviving Worker Nodes can recover and drain its queues. Pair shared storage with orphan management to control when that recovery starts.

Shared PQ storage and orphan management are available in Cribl Stream on customer-managed and hybrid Worker Groups only. They require an Enterprise license. Cribl Edge uses local filesystem PQ storage only. On Cribl.Cloud-provisioned Worker Groups, Cribl manages shared storage automatically. You cannot edit PQ storage or orphan management settings.

For PQ basics, backpressure behavior, and per-Source or per-Destination settings, see About Persistent Queues, Optimize Source Persistent Queues, and Optimize Destination Persistent Queues.

When to Use Shared Storage ​

Use shared PQ storage when:

  • Worker Nodes in the same Worker Group are replaced, scaled down, or fail during a backpressure event.
  • You need queued data to survive the loss of a specific Worker Node, not just a downstream outage.
  • You run in elastic environments (Kubernetes autoscaling, cloud instance replacement) where Worker Nodes come and go.

Use local filesystem PQ storage when:

  • Worker Nodes have stable identities and local disks you control (for example, fixed on-prem hosts).
  • You need the lowest write latency and do not need cross-Worker Node recovery.

For storage type trade-offs and sizing guidance, see Where to Store Data and Worker Group PQ Storage Fields.

How Shared Storage Works ​

Shared storage namespaces queue data by Worker Node so Worker Nodes in the same Group do not overwrite each other’s queues. Each Worker Node writes to its own area on the shared store. Surviving Worker Nodes can read and drain queues that belonged to a failed Worker Node.

  • Local filesystem and NFS: Configure the Queue file path (local) or Mount point (NFS).
  • AWS S3: Configure the bucket Key prefix. Cribl Stream uses a local cache on each Worker Node for fast writes, then uploads data to S3 asynchronously.

Migrate from Local Disk ​

When you first enable shared PQ storage and deploy the configuration, Cribl Stream automatically copies existing local PQ data from each Worker Node to the shared store. Drain queues or plan for this migration window before you switch storage types in production.

Switch Back to Local Storage ​

You can change Storage type back to Local filesystem at any time. Data is not migrated automatically. Drain all shared queues into your Destinations, or use Clear persistent queue, before you switch. Otherwise, data remains in the S3 bucket or NFS mount and is not attached to the new local queues.

Configure Shared Storage ​

Configure shared PQ storage at the Worker Group level. All Worker Nodes in the Group must use the same storage type. You cannot mix local and shared storage in one Group.

  1. Select Products, then Cribl Stream. Under Worker Groups, select a Worker Group.
  2. Select Worker Group Settings, then System, then PQ Storage.
  3. Set Storage type to Network filesystem (NFS) or AWS S3.
  4. Configure connection settings, limits, and compression. See Worker Group PQ Storage Fields for field descriptions and recommendations.
  5. Select Save, then Commit and Deploy.

After you configure shared storage, enable PQ on individual Sources or Destinations. When shared storage is active, Queue size limit and related capacity fields are managed at the Worker Group level and are hidden on individual Source and Destination PQ settings.

For Kubernetes deployments, mount shared volumes consistently on Worker Nodes and the Leader, and preserve Worker Node identity across restarts. See Persistent Queue Considerations for Kubernetes Deployments.

Orphan Management and PQ Revival ​

An orphan is queued data on shared storage from a Worker Node that is no longer active in the Worker Group. This can happen when a Worker Node is replaced, scaled down, or cannot reach the Leader for an extended period.

A restart that preserves the same Worker Node identity is not an orphan. The same Worker Node reconnects and resumes draining its own queues.

Revival Flow ​

When a Worker Node has been absent longer than the Orphan detection threshold, Cribl Stream can reassign its orphaned queues to surviving Worker Nodes for drain. Surviving Worker Nodes read the queued data from shared storage and deliver it through the configured Routes, Pipelines, and Destinations.

If the original Worker Node returns before reassignment completes, it resumes draining its own queues.

Configure Orphan Management ​

Orphan management controls when the Leader scans shared storage and reassigns orphaned queues to surviving Worker Nodes. It applies only when Storage type is Network filesystem (NFS) or AWS S3.

PQ Orphan Management appears in the UI only after you configure shared PQ storage. If Storage type is Local filesystem, the setting is hidden because orphan management does not apply to per-Worker Node local queues. The menu item also does not appear on Cribl.Cloud-provisioned Worker Groups.

To configure orphan management in the UI:

  1. Configure shared storage in Worker Group Settings > System > PQ Storage. Set Storage type to Network filesystem (NFS) or AWS S3, then select Save, Commit, and Deploy.
  2. Return to Worker Group Settings > System. PQ Orphan Management appears next to PQ Storage.
  3. Adjust