Fluentd

forward

The out_forward Buffered Output plugin forwards events to other fluentd nodes. This plugin supports load-balancing and automatic fail-over (i.e. active-active backup). For replication, please use out_copy plugin.

The out_forward plugin detects server faults using a φ accrual failure detector algorithm. You can customize the parameters of the algorithm. When a server fault recovers, the plugin makes the server available automatically after a few seconds.

The out_forward plugin supports at-most-once and at-least-once semantics. The default is at-most-once.

It is included in Fluentd's core.

Example Configuration

<match pattern>
  @type forward
  send_timeout 60s
  recover_wait 10s
  hard_timeout 60s

  <server>
    name myserver1
    host 192.168.1.3
    port 24224
    weight 60
  </server>
  <server>
    name myserver2
    host 192.168.1.4
    port 24224
    weight 60
  </server>
  ...

  <secondary>
    @type file
    path /var/log/fluent/forward-failed
  </secondary>
</match>

Please see the Configuration File article for the basic structure and syntax of the configuration file.

Supported Modes

• Synchronous
• Asynchronous

See Output Plugin Overview for more details.

Plugin Helpers

• socket
• server
• timer
• thread
• compat_parameters
• service_discovery

Parameters

Common Parameters

@type

The value must be forward.

<server> (at least one is required)

The destination servers. Each server has the following parameters:

• host
• name
• port
• shared_key
• username
• password
• standby
• weight

If you want to manage destination servers by flexible approach, use <service_discovery> instead.

host

The IP address or host name of the server.

name

The name of the server. Used for logging and certificate verification in TLS transport (when the host is the address).

port

The port number of the host. Note that both TCP packets (event stream) and UDP packets (heartbeat messages) are sent to this port.

shared_key

The shared key per server.

username

The username for authentication.

password

The password for authentication.

standby

Marks a node as the standby node for an Active-Standby model between Fluentd nodes. When an active node goes down, the standby node is promoted to an active node. The standby node is not used by the out_forward plugin until then.

<match pattern>
  @type forward
  # ...

  <server>
    name myserver1
    host 192.168.1.3
    weight 60
  </server>
  <server>  # forward doesn't use myserver2 until myserver1 goes down
    name myserver2
    host 192.168.1.4
    weight 60
    standby
  </server>
  # ...
</match>

weight

The load balancing weight. If the weight of one server is 20 and the weight of the other server is 30, events are sent in a 2:3 ratio. The default weight is 60.

<service_discovery>

Use service discovery plugin instead of fixed <server> list. See also Service Discovery Plugin Overview for more details.

<match pattern>
  @type forward

  <service_discovery>
    @type file
    path /path/to/servers.yaml
  </service_discovery>
</match>

transport

The transport protocol used to connect to the destination servers.

Set tls to use TLS. All the tls_* parameters take effect only with transport tls. See How to connect to a TLS/SSL enabled server? for a complete configuration.

require_ack_response

Changes the protocol to at-least-once. The plugin waits for the ack from destination's in_forward plugin.

ack_response_timeout

This option is used when require_ack_response is true. This default value is based on popular tcp_syn_retries.

If set 0, this plugin does not wait for the ack response.

read_interval_msec

The wait time in milliseconds before retrying to read the handshake response from a server.

The handshake is a part of the authentication, so this parameter takes effect only when the <security> section is configured.

read_length

The read size for the data which this plugin receives from a server.

It applies to the handshake response of the <security> section, the ack response of require_ack_response and the heartbeat response of heartbeat_type udp.

This is a low-level tuning parameter, so you rarely need to change it.

send_timeout

The timeout time when sending event logs.

connect_timeout

The connection timeout for the socket. When the connection is timed out during the connection establishment, Errno::ETIMEDOUT error is raised.

recover_wait

The wait time before accepting a server fault recovery.

heartbeat_type

Specifies the transport protocol for heartbeats. Set none to disable.

The default value transport means the protocol which transport specifies.

tcp is deprecated since v0.14.12. It is treated as transport.

heartbeat_interval

The interval of the heartbeat packer.

phi_failure_detector

Use the "Phi accrual failure detector" to detect server failure.

phi_threshold

The threshold parameter used to detect server faults.

phi_threshold is directly related to heartbeat_interval. If you are using longer heartbeat_interval, please use the larger phi_threshold. Otherwise, you will see frequent detachments of destination servers. The default value 16 is tuned for heartbeat_interval 1s.

hard_timeout

The hard timeout used to detect server failure. The default value is equal to the send_timeout parameter.

expire_dns_cache

Sets TTL to expire DNS cache in seconds. Set 0 not to use DNS Cache.

dns_round_robin

Enable client-side DNS round robin. Uniform randomly pick an IP address to send data when a hostname has several IP addresses.

heartbeat_type udp is not available with dns_round_robin true. Use heartbeat_type transport or heartbeat_type none.

ignore_network_errors_at_startup

Ignores DNS resolution and errors at startup time.

compress

Since v0.14.7, Fluentd supports transparent data compression. You can use this feature to reduce the transferred payload size.

Example:

<match debug.**>
  @type forward
  compress gzip
  <server>
    host 192.168.1.2
    port 24224
  </server>
</match>

You do not need any configuration in the receiving server. Data compression is auto-detected and handled transparently by the destination node.

Since v1.19.3, the receiving in_forward limits the decompressed chunk size to 256MB by default. See in_forward if you need to send larger chunks.

Supported values:

• text
• gzip
• zstd (Experimental) (since v1.19.0)

tls_version

The default version of TLS transport.

tls_ciphers

The cipher configuration of TLS transport.

tls_insecure_mode

Skips all verification of certificates or not.

tls_allow_self_signed_cert

Allows self-signed certificates or not.

tls_verify_hostname

Verifies hostname of servers and certificates or not in TLS transport.

If the following conditions are met, you must set tls_verify_hostname false explicitly to forward events correctly:

• specify host in <server> section with IP address, not hostname
• specify server certificate file with tls_cert_path which contains common name (CN) field with IP address, not hostname

tls_ca_cert_path

The additional CA certificate path for TLS.

This is the new name of tls_cert_path, which was renamed in v1.3.1 to clarify its meaning.

If both of this parameter and tls_cert_path are specified, tls_cert_path is used and this parameter is NOT used.

tls_cert_path

The additional certificate path for TLS.

This parameter was the one for the CA certificate until v1.3.0. tls_ca_cert_path took over that role in v1.3.1.

If both of tls_ca_cert_path and this parameter are specified, this parameter is used and tls_ca_cert_path is NOT used.

tls_client_cert_path

The client certificate path for TLS.

tls_client_private_key_path

The client private key path for TLS.

tls_client_private_key_passphrase

The TLS private key passphrase for the client.

tls_cert_thumbprint

The certificate thumbprint for searching from Windows system certstore. This parameter is for Windows only.

tls_cert_logical_store_name