Initializing
Overview
The firedancer configure command is used to setup the host operator system so Firedancer can run correctly. It does the following:
- iavf Configures VFs, clears and installs PF ntuple rules and loads the
vfio-pcikernel module required by the iavf network provider. - mlx5 Loads the
ib_uverbskernel module required by the mlx5 network provider and clears network configuration left by XDP. - hugetlbfs Reserves huge and gigantic pages for use by Firedancer.
- sysctl Sets required kernel parameters.
- hyperthreads Checks hyperthreaded pair for critical CPU cores.
- bonding Prepares bonded network devices for XDP networking.
- ethtool-channels Configures the number of channels on the network device.
- ethtool-offloads Modify offload feature flags on the network device.
- ethtool-loopback Disable tx-udp-segmentation on the loopback device.
- irq-affinity Prevents the kernel from routing IRQs to CPU cores used by Firedancer (via
/proc/irq). - irq-balance Prevents the irqbalance daemon from routing IRQs to CPU cores used by Firedancer (via
/run/irqbalance). - kworkers Prevents unbound kernel workqueue workers from running on CPU cores used by Firedancer (via
/sys/devices/virtual/workqueue/cpumask). - cpuset Creates an isolated cgroup cpuset partition over the CPU cores used by Firedancer, so no other process can be scheduled onto them (via
/sys/fs/cgroup). - console Quiets periodic kernel console rendering work that runs on the CPU that generated it, including CPU cores used by Firedancer (via
/proc/sys/kernel/printkand/sys/class/graphics/fbcon/cursor_blink).
The hugetlbfs configuration must be performed every time the system is rebooted, to remount the hugetlbfs filesystems, as do sysctl, ethtool-channels and ethtool-offloads to reconfigure the networking device, irq-affinity and irq-balance configure IRQ affinities for CPUs.
The configure command is run like firedancer configure <mode> <stage>... where mode is one of:
initConfigures the provided stages if they are not already configured.checkCheck if each stage is already configured. The command will exit with an error code if they are not.checknever requires privileges and will not make any changes to the system.finiUnconfigure (reverse) the stage if it is reversible.
stage can be one or more of iavf, mlx5, hugetlbfs, sysctl, hyperthreads, bonding, ethtool-channels, ethtool-offloads, ethtool-loopback, irq-affinity, irq-balance, kworkers, cpuset, console, and snapshots and these stages are described below. You can also use the stage all which will configure everything.
Stages have different privilege requirements, which you can see by trying to run the stage without privileges. The check mode never requires privileges, and the init mode will only require privileges if it needs to actually change something.
iavf
The iavf provider uses one SR-IOV Virtual Function (VF) per Physical Function (PF) on [net.interface]. This stage only runs when [net.provider] is iavf. The PFs must use Intel ice or i40e drivers, SR-IOV and IOMMU must be enabled in the system's BIOS, with Virtualization Mode set to SR-IOV in the NIC's BIOS section.
init loads the vfio-pci kernel module if needed, then reuses the highest index VF bound to vfio-pci. If VFs exist but none are bound to vfio-pci, init fails. If no VFs exist, it creates VF 0 and binds it to vfio-pci. It assigns the MAC address of [net.interface] to each VF used (this leads to all RX being sent to both the PF and VF, Firedancer adds drop rules on the PF so it processes less unnecessary FD packets), disables spoof checking and trust, and sets automatic link state. Repeated init does not add excess VFs.
The stage clears all existing PF ntuple rules and installs UDP port based drop rules for Firedancer's ports so unnecessary packets are not processed by the normal kernel (would otherwise waste CPU resources).
Changing VFs or PF filters requires root privileges. CAP_SYS_MODULE is also required when loading vfio-pci. If already configured, init does nothing. The check mode requires no privileges and makes no changes.
fini clears PF's ntuple rules and removes VF 0 if its PF has exactly one VF (PFs with multiple VFs retain all of them), it does not unload vfio-pci.
mlx5
The mlx5 network provider sets up the network device through the /dev/infiniband/uverbs* devices, which are provided by the ib_uverbs kernel module. The module is not always loaded at boot. This stage only runs when [net.provider] is mlx5.
If the uverbs interface is unavailable, the stage runs modprobe ib_uverbs to load the module. If the interface is still unavailable after loading the module, the stage fails and [net.provider] should be set to xdp.
The stage also clears network configuration left by XDP on [net.interface]. When leftover configuration is found, init automatically calls fini to reset the device before loading the module if needed. Cleanup deletes all ntuple rules, restores the default RSS indirection table, and restores the maximum channel count, capped by the number of online CPUs. For a bonded interface, it resets each bond member. Devices already at defaults are left unchanged.
Run this stage after boot if the uverbs interface is unavailable, or when switching from XDP to mlx5. If leftover XDP configuration is found during startup, Firedancer stops and recommends running firedancer configure init mlx5 to restore the device defaults. The stage has no dependencies on any other stage.
Changing device settings or loading the module requires root privileges. CAP_SYS_MODULE is also required when the module needs to be loaded. If the uverbs interface is available and the network devices are already at defaults, init does nothing and check succeeds without privileges. Otherwise, check fails without changing anything.
The fini mode performs the network cleanup described above and requires root privileges only when device settings need changing. It does not unload the module or restore the deleted ntuple rules and previous network settings.
hugetlbfs
The hugetlbfs stage is used to reserve huge (2MiB) and gigantic (1GiB) memory pages from the Linux kernel for use by Firedancer. See also the kernel documentation of these pages. Almost all memory in Firedancer is allocated out of these pages for performance reasons.
This is a two step process. First, the number of huge and gigantic pages available on the entire system is increased in the kernel by increasing /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages until the free_hugepages value is high enough for all the memory needs of the validator.
Once the pages have been reserved globally in the kernel pool, they are assigned specifically to Firedancer by creating a hugetlbfs mount at each of /mnt/.fd/.gigantic/ and /mnt/.fd/.huge for gigantic and huge pages respectively. These paths can be configured in the TOML file under the [hugetlbfs] section. Lets run it:
$ sudo firedancer configure init hugetlbfs
NOTICE hugetlbfs ... unconfigured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist
NOTICE hugetlbfs ... configuring
NOTICE RUN: `mkdir -p /mnt/.fd/.huge`
NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.huge -o pagesize=2097152,min_size=6155141120`
NOTICE RUN: `mkdir -p /mnt/.fd/.gigantic`
NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.gigantic -o pagesize=1073741824,min_size=150323855360`
$ cat /proc/mounts | grep \\.fd
none /mnt/.fd/.huge hugetlbfs rw,relatime,pagesize=2M,min_size=6155141120 0 0
none /mnt/.fd/.gigantic hugetlbfs rw,relatime,pagesize=1024M,min_size=150323855360 0 0This stage requires root privileges, and cannot be performed with capabilities. If the required hugetlbfs mounts are already present, with at least the amount of memory reserved that we required then the init mode does nothing and the check mode will return successfully without requiring privileges.
The fini mode will unmount the two filesystems, and remove them from /mnt/.fd/, although it will leave the /mnt/.fd/ directory in place. The fini mode will not succeed if memory from the mounts is mapped into a running process.
If fini succeeds, the huge and gigantic pages that Firedancer had reserved will be returned to the kernel global pool so they can be used by other programs, but the global pool size will not be decreased, even if it was earlier increased during init.
TIP
The hugetlbfs step should be run immediately when the system is booted. If run later, it may fail because the operating system memory is fragmented and a large contiguous block cannot be reserved.
sysctl
It is suggested to run Firedancer with certain kernel parameters tuned for best performance. The sysctl stage will check and configure these parameters. The stage will only increase values to meet the minimum, and will not decrease them if the minimum is already met.
| Sysctl | Minimum | Required | Description |
|---|---|---|---|
| /proc/sys/vm/max_map_count | 1000000 | Yes | Agave accounts database requires mapping many files. |
| /proc/sys/fs/file-max | 1024000 | Yes | Agave accounts database requires opening many files. |
| /proc/sys/fs/nr_open | 1024000 | Yes | Agave accounts database requires opening many files. |
| /proc/sys/net/ipv4/conf/lo/rp_filter | 2 | Yes | If sending QUIC transactions to Firedancer over loopback, this must be enabled to receive a response. Otherwise Linux will drop response packets due to limitations in the kernel eBPF networking stack. The sendTransaction RPC call will send over loopback. |
| /proc/sys/net/ipv4/conf/lo/accept_local | 1 | Yes | If sending QUIC transactions to Firedancer over loopback, this must be enabled to receive a response. Otherwise Linux will drop response packets due to limitations in the kernel eBPF networking stack. The sendTransaction RPC call will send over loopback. |
| /proc/sys/net/core/bpf_jit_enable | 1 | No | Firedancer uses BPF for kernel bypass networking. BPF JIT makes this faster. |
| /proc/sys/net/core/wmem_max | see desc. | Yes | With the XDP network stack in copy mode, every packet being transmitted is charged to the socket send buffer, which Firedancer sizes to the transmit ring: net.xdp.xdp_tx_queue_size times 2048 bytes (64 MiB by default). A lower maximum caps it and limits how many packets the kernel can transmit at once. |
| /proc/sys/kernel/numa_balancing | 0 | No | Firedancer assigns all memory to the right NUMA node, and rebalancing will make the system slower. |
Sysctls that are not required will produce a warning if they are not set correctly, but configuration will proceed and exit normally.
The init mode requires either root privileges, or to be run with CAP_SYS_ADMIN. The fini mode does nothing and kernel parameters will never be reduced or changed back as a result of running configure.
hyperthreads
Most work in Firedancer can be scaled with the number of CPU cores, but there are two jobs (tiles) which must run serially on a single core:
- pack Responsible for scheduling transactions for execution when we are leader.
- poh Performs repeated
sha256hashes, and periodically stamps these hashes into in-progress blocks when we are leader.
Because any interruption, context switch, or sharing of the CPU core that these jobs run on could cause skipped leader slots or unfull blocks, Firedancer expects them to get a dedicated core. This means on machines with a hyperthreaded CPU, the hyperthreaded pair of these tiles should be switched to offline.
This stage looks to see if the CPU is hyperthreaded, and will print a warning for the operator if the pair of these tiles are used or online.
A typical warning on a hyperthreaded system with auto layout looks like this:
$ firedancer configure check hyperthreads
WARNING pack cpu 1 has hyperthread pair cpu 65 which should be offline or in the isolated cpuset partition (`configure init cpuset`). Proceeding but performance may be reduced.When using the auto layout, Firedancer will avoid assigning tiles to run as hyperthread pairs, except when it is beneficial. But if using a manual layout, it is possible to assign another tile to the pair.
This stage has no dependencies on any other stage, but it is dependent on the topology specified in your configuration. It is recommended that you either turn off the CPUs specified in the warning, or run configure init cpuset so they are kept idle in the isolated partition.
bonding
Validator hosts commonly use bonded network devices to increase bandwidth and fault tolerance. Common Linux distributions pick defaults for bonding driver configuration that are incompatible with the XDP mode of some network drivers. When attaching XDP to a network device part of such an incorrectly configured bond, the bonding driver may erroneously consider that device as failed, leading to connectivity loss.
Firedancer reconfigures bonding driver parameters to tolerate brief downtime when configuring XDP networking. This is done by increasing the miimode, downdelay, and peer_notif_delay timeout parameters in /sys/class/net/bond0/bonding/ to five seconds.
$ sudo firedancer configure init bonding
NOTICE bonding ... unconfigured ... /sys/class/net/bond0/bonding/miimon is 100ms, want at least 5000ms
NOTICE bonding ... configuring
NOTICE RUN: `echo 5000 | sudo tee /sys/class/net/bond0/bonding/miimon`
NOTICE RUN: `echo 5000 | sudo tee /sys/class/net/bond0/bonding/downdelay`
NOTICE RUN: `echo 5000 | sudo tee /sys/class/net/bond0/bonding/peer_notif_delay`
$ cat /sys/class/net/bond0/bonding/miimon
5000Changing device settings with bonding requires root privileges, and cannot be performed with capabilities.
ethtool-channels
This stage only runs when [net.provider] is xdp. Firedancer uses receive side scaling (RSS) to improve network performance. This uses functionality of modern NICs to steer packets to different queues to distribute processing among CPUs. See the kernel documentation for more information.
In Firedancer, each net tile serves just one network queue, so the ethtool-channels stage will modify the network device [net.interface] configuration such that all packets needed by Firedancer are steered to the proper queue(s). There are three modes, selectable in your configuration, that govern this behavior:
simple mode modifies the combined channel count of the configured network device to be the same as the number of
nettiles,[layout.net_tile_count]. If your NIC does not support the required number of queues, you will need to reduce the number ofnettiles, potentially down to one for NICs which don't support queues at all. This is the default mode and should work for all network devices. Because the queue count is reduced system-wide, not solely for Firedancer, this can have a negative performance impact on non-Firedancer network traffic.dedicated mode reserves a dedicated hardware queue for each
nettile. This is the more advanced mode and may not work with all network devices. By modifying the RXFH indirection table and installing ntuple rules, Firedancer traffic is directed onto the dedicated queues and all other traffic is sharded amongst the rest. This has a performance benefit for both Firedancer and non-Firedancer traffic.auto mode attempts to initialize the device in dedicated mode and automatically falls back to simple mode if any failure occurs.
The command run by the stage in simple mode is similar to running ethtool --set-channels <device> combined <N> but it also supports bonded devices. We can check that it worked:
$ sudo firedancer configure init ethtool-channels
NOTICE ethtool-channels ... unconfigured ... device `bond0` missing ethtool-channels network configuration
NOTICE ethtool-channels ... configuring
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE RUN: `ethtool --set-channels enp1s0f0np0 combined 128`
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f0np0 start 2 equal 126`
NOTICE RUN: `ethtool --config-ntuple enp1s0f0np0 flow-type udp4 dst-port 8003 src-ip 0.0.0.0 m 255.255.255.254 queue 0`
WARNING error configuring network device (enp1s0f0np0), ioctl(SIOCETHTOOL,ETHTOOL_SRXCLSRLINS) failed (95-ENOTSUP-operation not supported)
WARNING error configuring network device (bond0), rss_queue_mode "auto" attempted "dedicated" configuration but falling back to "simple".
NOTICE RUN: `ethtool --set-rxfh-indir enp1s0f0np0 default`
NOTICE RUN: `ethtool --set-channels enp1s0f0np0 combined 128`