Ways to encrypt data on servers

At the 2026 edition of FOSSY, Romeo Solano gave a fast-paced, humorous presentation on what could have been a rather boring topic: server encryption. There are a number of threats that we face in today's world, from criminals, government overreach, espionage, and more, that can be thwarted with encryption. But encrypting data on a system that may live elsewhere, without any access to its keyboard at boot time, is rather more difficult than encrypting the disk of a laptop. Solano described the problems and gave a tour of some of the solutions in the talk.

The goal of the talk was to show attendees how to encrypt the data on their servers and to describe why they should do so. Encrypting the data at rest requires that the data be decrypted when the server is running, so he would be talking about how to do that at boot time, no matter whether the server was in the same room, another room in the house, or elsewhere on the globe. "You'll be able to accomplish this whether or not you can trust your hosting provider to lend you a helping hand."

The password prompt used when logging into a computer does "about as good a job at preventing access to your files" as a speed limit sign does to prevent speeding, Solano said. On the other hand, the disk-encryption-password prompt, "which looks incredibly similar but is slightly different", does actually prevent file access unless the decryption key is provided.

Encryption

Cryptography uses advanced math "beyond the understanding of most mortal humans, including myself". But that does not matter, users can still take advantage of cryptography without knowing or understanding the math underlying it. Because of that, it has become "standard operating procedure" for people to encrypt their computers and phones, especially for technical people like those in attendance at the talk (and the conference). "Why wouldn't my disks be encrypted at rest?"

It is more than just the risk of loss or theft, but to "protect against an increasingly snoopy and wants-to-know-everything-your're-doing international violence monopoly". It is a matter of course for many to power down their laptops and phones, thus encrypting their data, when going through a border checkpoint, for example. But much of the data users want to protect does not live on those devices, it is on a server somewhere.

When the server loses and regains power or needs to be rebooted for an urgent security update, though, the administrator may not be nearby to enter a password into the disk-encryption prompt. Maybe they are away on vacation; "I'm sorry honey, we have no Jellyfin for the next two weeks" is kind of a hard sell, he said. This is one of the reasons that people often leave their servers unencrypted.

The other main reason to not encrypt server data is because of a "real, tangible, potential performance impact" on reads and writes of data on encrypted disks. He showed numbers of 656MB per second versus 147MBps for the encrypted disk, which came from a 2020 Cloudflare blog post. The technical lead for the company, Ignat Korchagin, was looking into this performance degradation and found that a write to an encrypted filesystem ends up waiting in four separate queues before it reaches the disk. He patched the kernel to avoid that queueing and was able to get close to the same speed for unencrypted (696MBps) and encrypted (640MBps), though.

Those patches are part of the upstream kernel, so they can be enabled by anyone, Solano said. Those improvements only come for small I/O sizes, however, and performance can be even worse for large I/O. Each person needs to decide if the increased data security is worth the reduced performance; for him it is, but others may differ.

Threat model

It is unlikely that most users are facing movie-style heists in the data center, with people rappelling from the ceiling to grab servers or disks; "there's no laser beams or guard dogs". The real threat model that disk encryption is meant to thwart is someone taking a disk or server from the data center or a home lab. That means they now have the data—unless it is encrypted.

That threat may sound far-fetched, Solano said, but it happens frequently to small, community-run servers that are open to the public when a single person who uses it is accused of a crime. The police or other law-enforcement agencies confiscate the server for lengthy terms with little or no information about alleged crimes. It has also happened to regular people with their own servers that only they use; "they've had their homes ransacked and their equipment stolen", which lasts for years without any kind of charge being made. "This happens to the kind of people at FOSSY; this could happen to you."

There are other kinds of attacks, including cold boot attacks to recover encryption keys from the memory of a shut-down system. There are also evil maid attacks that swap out some hardware or software in the system, which allows the attacker to capture the decryption key. Those are real attacks, but they are "incredibly rare", targeted efforts that are outside of the scope of his threat model for the talk.

He reiterated that each administrator needs to decide for themselves whether encryption on the server makes sense for them and their users; he feels like he has a responsibility to protect his users' data, so the tradeoffs are worth it.

Key handling

There are ways to avoid the "inconvenience factor" of needing to provide the decryption key when the system restarts, so the trip to the data center—or even just down the hall to the server—can be eliminated. The first is "the worst possible method", which is a service called "remote hands" that most hosting providers offer; essentially it allows the use of the hands of an on-site technician, who can perform various tasks, including typing in a password. The dangers of that "should be immediately obvious". For one, the technician (and hosting company) have much less interest in keeping that key secure against bribery, theft, legal threats, and more; for another, many people have their server in a lab at home where there is no friendly technician available at all times.

A more desirable mechanism would be a way to remotely attach to the server system and type the password in. Server-class hardware generally has an extra network port, often labeled "IPMI" (for Intelligent Platform Management Interface). Connecting that to the local network, and creating a VPN connection to it, will allow accessing the system remotely, including its console, via a web browser. It is not a new technology, Solano said, but one that may not be all that well-known, especially among those with home labs.

The IPMI web server is running on a separate chip inside the system, not on the main CPU. That code only gets security updates infrequently, so it is important to have the VPN tunnel rather than exposing it on the internet. Hosting providers may not be willing or able to set that tunnel up—or they may promise that they can and then not actually do it for more than a year. "I don't know anything about that", he said with a sigh.

It would be even better to be able to use SSH to reach the server, but there is a chicken-and-egg problem in the way. The SSH daemon is normally on the root filesystem, which is encrypted and awaiting the decryption key from the console prompt. One solution to that is to use Dropbear SSH and to run it from the initial RAM disk (initramfs), which is the same place that the decryption-key console prompt is running from.

Depending on whether you believe the project's README file or its web site, Dropbear is "either 'smallish' or 'relatively small'". It lacks many of the features of the standard OpenSSH server, but "it is a lot smaller, and it's statically linked, it has no dependencies" so it can be easily placed in the initramfs. On Ubuntu systems (which is what he was using for his examples), it simply requires adding two packages (dropbear-initramfs and busybox-initramfs), modifying two configuration files, and doing a little more setup. The result is a system that allows SSH-key-based logins resulting in a BusyBox shell where the cryptroot-unlock command can be used to provide the disk-decryption key. Solano's talk notes have more information on setting up Dropbear in this fashion.

He uses both the IPMI virtual console and Dropbear solutions and likes them both—at least when he can get the VPN tunnel needed. Those mechanisms have something in common, though: they require that the administrator is available to type in the password. They cannot be asleep, hospitalized, or simply being inattentive to their systems because the system will not boot until the password is entered.

Unattended

It would be nice if there were an unattended solution where the server would only decrypt its disk if it is in a known safe state, but it would stay encrypted in the "cold, offline attack scenario that we are trying to protect against". As might be guessed, that is possible, by using two tools called Clevis and Tang, which are named after a two-piece fastener used in towing and farming, he said.

In the cryptography world, Tang is the server side that runs on some other internet host, while Clevis is the client side that runs from the initramfs as with Dropbear. Using Tang might sound like handing the key off to a remote technician, as mentioned earlier, but it is not. Tang does not know the key, nor does Clevis. He put up a slide (from his ODP slides) with a complicated-looking formula; "the way that it works is incredibly simple for some people, I am not some people", he said with a chuckle.

In order to explain, he used his cat and himself as metaphors for Tang and Clevis, respectively. When the two of them come together, because his cat jumps in his lap, say, they create a globally unique "shape" that corresponds to the decryption key. A different human or a different cat creates an entirely different key. Beyond that, Tang does not store anything about the encryption key (e.g. fingerprint) or retain any state about the Clevis on your system. "It's completely stateless, completely anonymous, no logs, no records, nothing, at least by default."

A single Clevis can use multiple Tangs, either individually or in groups. So a Clevis could require N Tangs in order to provide the disk-decryption key, for example. Installing Tang just requires a single package; it will run a server that listens on port 80, which is all that is needed. Clevis requires three packages and a command to bind a disk encrypted using Linux Unified Key Setup (LUKS) to the Tang server. Periodically, the Tang keys should be rotated, which can be done with a single command on the Tang server and another on the Clevis client. Once again, his notes provide more details.

Clevis and Tang can be used for more than just LUKS keys; they provide a general mechanism to derive secrets without storing the secret, which makes it different from what key escrow, or friendly technicians, require). In addition, Clevis can be bound to the Trusted Platform Module (TPM) chip. He personally does not trust the security of the TPM and is concerned that placing the disk-encryption key into a device on the system could result in key disclosure if there is some TPM exploit someday. But, "if you want to trust your TPM, Clevis has you covered."

Encrypted directories

The solutions he presented so far were all based on fully encrypted disks, where the root partition and everything else was protected. That is the way that all of his systems are set up, but it may not be realistic for every scenario. For example, the system may need to stay up, so it cannot afford the downtime required to encrypt it or it could be storing a huge amount of data that, due to storage costs, is too expensive to duplicate in order to encrypt it. In addition, virtual private server (VPS) systems from hosting providers may not provide a mechanism to encrypt the disks. For situations like those, other encryption options may make sense.

For example, Ubuntu has provided an installation option to encrypt a user's home directory for over 12 years. The regular user password is also the decryption key for the directory. It uses eCryptfs, which is unmaintained and will probably be deprecated soon, Solano said. It has been replaced with fscrypt, "which does the same exact thing, except that it doesn't have a cool logo or a mascot". It does, however, have a maintainer, which is probably more important.

Fscrypt is easy to set up as well. On ext4, the tune2fs command can be used to turn on encryption for the filesystem. Once that is done, the fscrypt command allows encrypting specific directories on the filesystem. There are other fscrypt subcommands to lock and unlock the encrypted directory; that can be scripted to run at login time, for example. "You can even have it reach out to Clevis and Tang to derive a decryption key" and handle it all automatically. Fscrypt is only available on a few filesystem types, but there are equivalent features on other filesystems, such as XFS and ZFS, he said.

Another option is using systemd-homed to create block devices that are "space-limited, encrypted, portable, even, across filesystems and servers". Each directory that systemd-homed creates is associated with a new user account that can handle the encryption needs for the system. At that point, Solano was running out of time, so he referred attendees to his talk notes.

After a summary conclusion of the talk, it turned out he had a bit of time for questions. One was: If the "men in black" take a server running Clevis, can't they just have it contact the Tang server to construct the decryption key? The answer is that they can do so as long as the Tang server(s) are still running and are willing to reply to the Clevis client running in a new location. Various levels of protection can be applied such that Tang servers only reply to certain IP addresses, or over certain networks, which could thwart Clevis operations from the captured systems.

In just a little bit over half an hour, Solano took a bit of whirlwind tour through the encryption options available to server administrators. Interested readers can view the video (minus the short Q&A).

[I would like to thank the Linux Foundation, LWN's travel sponsor, for its assistance with my trip to Vancouver for FOSSY.]