LWN.net Weekly Edition for September 17, 2026

Welcome to the LWN.net Weekly Edition for September 17, 2026

This edition contains the following feature content:

• Ways to encrypt data on servers: a FOSSY talk on the why and how of encrypting data on servers.
• PostgreSQL 19's "scary patch contest": the PostgreSQL 19 release is delayed due to some patches that have not really stabilized yet.
• Accelerating the kernel's build process: some LLM-assisted changes that result in a significant speedup for kernel builds.
• Adding BPF to blk-iocost: a way to allow BPF programs to decide on the cost of I/O operations for the blk-iocost control-group controller.
• Lessons learned as the Debian Project Leader: Andreas Tille was the Debian Project Leader for two years and related what he learned at MiniDebConf Winterthur.

This week's edition also includes these inner pages:

• Brief items: Brief news items from throughout the community.
• Announcements: Newsletters, conferences, security updates, patches, and more.

Please enjoy this week's edition, and, as always, thank you for supporting LWN.net.

Comments (none posted)

Ways to encrypt data on servers

At the 2026 edition of FOSSY, Romeo Solano gave a fast-paced, humorous presentation on what could have been a rather boring topic: server encryption. There are a number of threats that we face in today's world, from criminals, government overreach, espionage, and more, that can be thwarted with encryption. But encrypting data on a system that may live elsewhere, without any access to its keyboard at boot time, is rather more difficult than encrypting the disk of a laptop. Solano described the problems and gave a tour of some of the solutions in the talk.

The goal of the talk was to show attendees how to encrypt the data on their servers and to describe why they should do so. Encrypting the data at rest requires that the data be decrypted when the server is running, so he would be talking about how to do that at boot time, no matter whether the server was in the same room, another room in the house, or elsewhere on the globe. "You'll be able to accomplish this whether or not you can trust your hosting provider to lend you a helping hand."

The password prompt used when logging into a computer does "about as good a job at preventing access to your files" as a speed limit sign does to prevent speeding, Solano said. On the other hand, the disk-encryption-password prompt, "which looks incredibly similar but is slightly different", does actually prevent file access unless the decryption key is provided.

Encryption

Cryptography uses advanced math "beyond the understanding of most mortal humans, including myself". But that does not matter, users can still take advantage of cryptography without knowing or understanding the math underlying it. Because of that, it has become "standard operating procedure" for people to encrypt their computers and phones, especially for technical people like those in attendance at the talk (and the conference). "Why wouldn't my disks be encrypted at rest?"

It is more than just the risk of loss or theft, but to "protect against an increasingly snoopy and wants-to-know-everything-your're-doing international violence monopoly". It is a matter of course for many to power down their laptops and phones, thus encrypting their data, when going through a border checkpoint, for example. But much of the data users want to protect does not live on those devices, it is on a server somewhere.

When the server loses and regains power or needs to be rebooted for an urgent security update, though, the administrator may not be nearby to enter a password into the disk-encryption prompt. Maybe they are away on vacation; "I'm sorry honey, we have no Jellyfin for the next two weeks" is kind of a hard sell, he said. This is one of the reasons that people often leave their servers unencrypted.

The other main reason to not encrypt server data is because of a "real, tangible, potential performance impact" on reads and writes of data on encrypted disks. He showed numbers of 656MB per second versus 147MBps for the encrypted disk, which came from a 2020 Cloudflare blog post. The technical lead for the company, Ignat Korchagin, was looking into this performance degradation and found that a write to an encrypted filesystem ends up waiting in four separate queues before it reaches the disk. He patched the kernel to avoid that queueing and was able to get close to the same speed for unencrypted (696MBps) and encrypted (640MBps), though.

Those patches are part of the upstream kernel, so they can be enabled by anyone, Solano said. Those improvements only come for small I/O sizes, however, and performance can be even worse for large I/O. Each person needs to decide if the increased data security is worth the reduced performance; for him it is, but others may differ.

Threat model

It is unlikely that most users are facing movie-style heists in the data center, with people rappelling from the ceiling to grab servers or disks; "there's no laser beams or guard dogs". The real threat model that disk encryption is meant to thwart is someone taking a disk or server from the data center or a home lab. That means they now have the data—unless it is encrypted.

That threat may sound far-fetched, Solano said, but it happens frequently to small, community-run servers that are open to the public when a single person who uses it is accused of a crime. The police or other law-enforcement agencies confiscate the server for lengthy terms with little or no information about alleged crimes. It has also happened to regular people with their own servers that only they use; "they've had their homes ransacked and their equipment stolen", which lasts for years without any kind of charge being made. "This happens to the kind of people at FOSSY; this could happen to you."

There are other kinds of attacks, including cold boot attacks to recover encryption keys from the memory of a shut-down system. There are also evil maid attacks that swap out some hardware or software in the system, which allows the attacker to capture the decryption key. Those are real attacks, but they are "incredibly rare", targeted efforts that are outside of the scope of his threat model for the talk.

He reiterated that each administrator needs to decide for themselves whether encryption on the server makes sense for them and their users; he feels like he has a responsibility to protect his users' data, so the tradeoffs are worth it.

Key handling

There are ways to avoid the "inconvenience factor" of needing to provide the decryption key when the system restarts, so the trip to the data center—or even just down the hall to the server—can be eliminated. The first is "the worst possible method", which is a service called "remote hands" that most hosting providers offer; essentially it allows the use of the hands of an on-site technician, who can perform various tasks, including typing in a password. The dangers of that "should be immediately obvious". For one, the technician (and hosting company) have much less interest in keeping that key secure against bribery, theft, legal threats, and more; for another, many people have their server in a lab at home where there is no friendly technician available at all times.

A more desirable mechanism would be a way to remotely attach to the server system and type the password in. Server-class hardware generally has an extra network port, often labeled "IPMI" (for Intelligent Platform Management Interface). Connecting that to the local network, and creating a VPN connection to it, will allow accessing the system remotely, including its console, via a web browser. It is not a new technology, Solano said, but one that may not be all that well-known, especially among those with home labs.

The IPMI web server is running on a separate chip inside the system, not on the main CPU. That code only gets security updates infrequently, so it is important to have the VPN tunnel rather than exposing it on the internet. Hosting providers may not be willing or able to set that tunnel up—or they may promise that they can and then not actually do it for more than a year. "I don't know anything about that", he said with a sigh.

It would be even better to be able to use SSH to reach the server, but there is a chicken-and-egg problem in the way. The SSH daemon is normally on the root filesystem, which is encrypted and awaiting the decryption key from the console prompt. One solution to that is to use Dropbear SSH and to run it from the initial RAM disk (initramfs), which is the same place that the decryption-key console prompt is running from.

Depending on whether you believe the project's README file or its web site, Dropbear is "either 'smallish' or 'relatively small'". It lacks many of the features of the standard OpenSSH server, but "it is a lot smaller, and it's statically linked, it has no dependencies" so it can be easily placed in the initramfs. On Ubuntu systems (which is what he was using for his examples), it simply requires adding two packages (dropbear-initramfs and busybox-initramfs), modifying two configuration files, and doing a little more setup. The result is a system that allows SSH-key-based logins resulting in a BusyBox shell where the cryptroot-unlock command can be used to provide the disk-decryption key. Solano's talk notes have more information on setting up Dropbear in this fashion.

He uses both the IPMI virtual console and Dropbear solutions and likes them both—at least when he can get the VPN tunnel needed. Those mechanisms have something in common, though: they require that the administrator is available to type in the password. They cannot be asleep, hospitalized, or simply being inattentive to their systems because the system will not boot until the password is entered.

Unattended

It would be nice if there were an unattended solution where the server would only decrypt its disk if it is in a known safe state, but it would stay encrypted in the "cold, offline attack scenario that we are trying to protect against". As might be guessed, that is possible, by using two tools called Clevis and Tang, which are named after a two-piece fastener used in towing and farming, he said.

In the cryptography world, Tang is the server side that runs on some other internet host, while Clevis is the client side that runs from the initramfs as with Dropbear. Using Tang might sound like handing the key off to a remote technician, as mentioned earlier, but it is not. Tang does not know the key, nor does Clevis. He put up a slide (from his ODP slides) with a complicated-looking formula; "the way that it works is incredibly simple for some people, I am not some people", he said with a chuckle.

In order to explain, he used his cat and himself as metaphors for Tang and Clevis, respectively. When the two of them come together, because his cat jumps in his lap, say, they create a globally unique "shape" that corresponds to the decryption key. A different human or a different cat creates an entirely different key. Beyond that, Tang does not store anything about the encryption key (e.g. fingerprint) or retain any state about the Clevis on your system. "It's completely stateless, completely anonymous, no logs, no records, nothing, at least by default."

A single Clevis can use multiple Tangs, either individually or in groups. So a Clevis could require N Tangs in order to provide the disk-decryption key, for example. Installing Tang just requires a single package; it will run a server that listens on port 80, which is all that is needed. Clevis requires three packages and a command to bind a disk encrypted using Linux Unified Key Setup (LUKS) to the Tang server. Periodically, the Tang keys should be rotated, which can be done with a single command on the Tang server and another on the Clevis client. Once again, his notes provide more details.

Clevis and Tang can be used for more than just LUKS keys; they provide a general mechanism to derive secrets without storing the secret, which makes it different from what key escrow, or friendly technicians, require). In addition, Clevis can be bound to the Trusted Platform Module (TPM) chip. He personally does not trust the security of the TPM and is concerned that placing the disk-encryption key into a device on the system could result in key disclosure if there is some TPM exploit someday. But, "if you want to trust your TPM, Clevis has you covered."

Encrypted directories

The solutions he presented so far were all based on fully encrypted disks, where the root partition and everything else was protected. That is the way that all of his systems are set up, but it may not be realistic for every scenario. For example, the system may need to stay up, so it cannot afford the downtime required to encrypt it or it could be storing a huge amount of data that, due to storage costs, is too expensive to duplicate in order to encrypt it. In addition, virtual private server (VPS) systems from hosting providers may not provide a mechanism to encrypt the disks. For situations like those, other encryption options may make sense.

For example, Ubuntu has provided an installation option to encrypt a user's home directory for over 12 years. The regular user password is also the decryption key for the directory. It uses eCryptfs, which is unmaintained and will probably be deprecated soon, Solano said. It has been replaced with fscrypt, "which does the same exact thing, except that it doesn't have a cool logo or a mascot". It does, however, have a maintainer, which is probably more important.

Fscrypt is easy to set up as well. On ext4, the tune2fs command can be used to turn on encryption for the filesystem. Once that is done, the fscrypt command allows encrypting specific directories on the filesystem. There are other fscrypt subcommands to lock and unlock the encrypted directory; that can be scripted to run at login time, for example. "You can even have it reach out to Clevis and Tang to derive a decryption key" and handle it all automatically. Fscrypt is only available on a few filesystem types, but there are equivalent features on other filesystems, such as XFS and ZFS, he said.

Another option is using systemd-homed to create block devices that are "space-limited, encrypted, portable, even, across filesystems and servers". Each directory that systemd-homed creates is associated with a new user account that can handle the encryption needs for the system. At that point, Solano was running out of time, so he referred attendees to his talk notes.

After a summary conclusion of the talk, it turned out he had a bit of time for questions. One was: If the "men in black" take a server running Clevis, can't they just have it contact the Tang server to construct the decryption key? The answer is that they can do so as long as the Tang server(s) are still running and are willing to reply to the Clevis client running in a new location. Various levels of protection can be applied such that Tang servers only reply to certain IP addresses, or over certain networks, which could thwart Clevis operations from the captured systems.

In just a little bit over half an hour, Solano took a bit of whirlwind tour through the encryption options available to server administrators. Interested readers can view the video (minus the short Q&A).

[I would like to thank the Linux Foundation, LWN's travel sponsor, for its assistance with my trip to Vancouver for FOSSY.]

Comments (31 posted)

PostgreSQL 19's "scary patch contest"

PostgreSQL 19 was expected to be released in September, in keeping with the database project's longstanding tradition of a major release every year. However, some late-breaking concerns about several of the features slated for inclusion has some developers worried about the quality of the release. On August 25, PostgreSQL contributor Robert Haas sent an email with the subject "scary patch contest" about several patches that have required an unusually large number of bug fixes leading up to the release, which has raised questions about their readiness for a stable release. One of the patches has been reverted, but several are still under heavy revision, and an extra beta release has been slotted in to allow for additional testing.

Scary patches

Development on a major PostgreSQL version begins more than a year before its expected release date; the first CommitFest to review patches for PostgreSQL 19 was held during July 2025, and the first for version 20, which is expected in 2027, was held in July 2026. A release enters feature freeze several months before the final release is scheduled to allow time for the project to publish pre-releases for testing and to fix any bugs found. Typically there are three beta releases released by August, and then a release candidate in late August or early September, with the stable ("GA") release in late September. PostgreSQL 19 entered feature freeze on April 8—the project has been working since to stabilize the final release.

That stabilization phase seems to have involved more bug fixing than usual. In his scary patch email Haas said that he "asked Claude to evaluate which v19 patches were the scariest based on the number and type of bugs fixed post-freeze", and then shared the results "with a few particularly cutting remarks from the LLM edited out". There were several patches that had seen heavy revision after the feature freeze that Haas felt might be in bad enough shape to merit reverting before PostgreSQL 19 shipped, though he did not have a firm position on what to do.

"I think it's pretty clear that none of these were as robust at commit time as we would like, but that doesn't mean that they're still broken." He believed that some of the patches identified by his LLM inspection were low risk, "because they are only going to come up in fairly niche situations", but he worried that some of them could have "a long tail of bugs that we haven't found yet, in pretty critical areas".

One of his examples was a pair of patches from Amit Langote and Junwang Zhao designed to speed up PostgreSQL's foreign-key checks. The first patch provides a way to speed up foreign-key queries by bypassing the usual method of using PostgreSQL's Server Programming Interface and, instead, doing a direct index scan for foreign keys. The second patch builds on that to add batching. According to Haas's email, the two patches had required about 16 fixes after feature freeze, including "five distinct classes of incorrect [foreign key] enforcement (cross-type, domains, column order, nullable keys, non-btree indexes)" and a redesign of the feature's transactional model after the freeze.

Langote said that he had reviewed the fixes made since feature freeze; he was persuaded that the concerns about the batching code were justified and "considerably more concerning" than the per-row fast-path fixes. He acknowledged that both patches had required a number of fixes after the freeze, but the fast-path changes did not have "the same pattern of complexity as the fixes needed to manage the lifetime of batched checks". He sent patches to remove the fast-path batching from PostgreSQL 19, though leaving it in the PostgreSQL 20 development branch, while keeping the per-row fast path intact. On September 8, he promised an updated patch to deal with some comments from Amit Kapila about problems in the first patch to revert the batching feature.

Haas said that he generally agreed with Langote that "the batching stuff feels much riskier than the per-row fast path stuff". He was not entirely sure, though, that the batching patch was in good enough shape to keep for PostgreSQL 20, or that the per-row fast-path patch was good enough to keep in 19. "I think we should do at least as much as what you're proposing here, but possibly more. However, I'm not very sure what the right answer is at this point." For now, the per-row fast-path patch looks as if it will be included in the upcoming beta release.

Two patches contributed by Antonín Houska, and committed by Álvaro Herrera, were also identified as worrisome. The first patch adds a REPACK command to PostgreSQL, with a second patch that implements an option (CONCURRENTLY) for the command. Haas said that the two patches had a cumulative total of 28 bug fixes since the feature freeze, "including data loss [...] and two security-flavored ACL [access-control list] fixes". There is at least one outstanding patch for the CONCURRENTLY option that is still being discussed, which would disallow the use of REPACK CONCURRENTLY on user catalog tables.

Haas said he was most concerned with the two sets of patches described above as well as one from Daniel Gustafsson and Magnus Hagander that allows data checksums to be enabled or disabled in a running PostgreSQL cluster without restricting access to the cluster during processing. Before this patch, enabling or disabling data checksums was only possible as an offline operation. According to the LLM analysis, the patch had required about 25 "substantive fixes" since feature freeze, including false checksum failures. "A corruption-detection feature producing false positives is exactly the wrong failure mode." Gustafsson replied that he would prepare a revert for the feature.

The intent, Haas said, was not to jump straight to reverting the patch: "My sincere intent was to start a conversation, not jump straight to a conclusion." If Gustafsson was already considering reverting the patch then it might make sense to do so, but otherwise Haas was just looking to discuss the patches.

Since then, Gustafsson has continued to work on the feature. On September 4 he addressed the case for reverting the feature and noted that he had posted a patch to revert the feature "to give anyone interested a chance to see what it would look like, and what I propose leaving behind in v19". For now, it seems that the data checksums feature will be in the next beta. Whether it survives to the GA release remains to be seen.

LLM effect

PostgreSQL 19 release management team member Melanie Plageman said that she was not sure if there was consensus that Gustafsson's patch needed to be reverted. She also wondered "if the ease with which LLMs allow people to pressure test features means we are finding more bugs sooner than we have in the past." Nathan Bossart, also a member of the release-management team said that he had thought use of LLMs would make new features more stable by feature freeze, "but it seems that that hasn't turned out to be the case".

However, Haas replied to Plageman that he was not convinced that the problem with Gustafsson's patch was as simple as finding more bugs with LLMs. He was unsure what the right course of action was; it might be fine if the remaining bugs "have few architectural implications and can likely be cleaned up in a minor release without causing much trouble". But he expressed concern that there could be remaining bugs that would not be easily fixed in a point release without bumping the catalog version ("catversion" in PostgreSQL parlance): in other words, making a change that would cause a database created with PostgreSQL 19.1 (for example) to be incompatible with 19.0.

Gustafsson noted that most of the large features in PostgreSQL 19, "were written, reviewed and tested, before AI tools were either available or even remotely as good as they are now". He thought it would be a different story when the feature freeze rolled around for PostgreSQL 20. Kapila agreed, and suggested that LLMs would be good for long-term stability of the project, even though the project was seeing more post-freeze fixes this time around.

LLMs may be better at spotting bugs, but having more time helps as well. Haas observed that all of the features that have had a large number of bug fixes post-freeze were committed close to feature freeze in April. He suggested that "there's a good chance that they would have had most of the bugs flushed out by now" if the features had been committed to PostgreSQL's master branch in August 2025.

PGQs pulled

PostgreSQL core team member Tom Lane said he was "quite afraid" of another patch from Haas's list, an implementation of SQL property graph queries (PGQs) by Peter Eisentraut and Ashutosh Bapat. That feature had seen 17 bug fixes that Haas had characterized as lower severity. Lane worried that the project was still discussing fixes for the feature that might cause a catversion bump. "At this point I'd be willing to bet dinner that if we ship it in v19 there will be post-release bug discoveries that are unfixable until v20."

Haas said, without taking a stance on that particular patch, that no feature that requires a catalog version bump belongs in PostgreSQL 19 at this point. "The time for working out what the catalogs should look like was sometime well in advance of feature freeze, not four months after it".

On September 2, Plageman said that the release-management team had been looking at the current state of the PGQ patch and had "serious concerns about the number of design issues with either recently posted patches or without consensus on behavior". She said that the team felt it would be better to revert PGQ for 19, and fix things properly in time for 20, "without having to be burdened by backwards compatibility and backpatching". Eisentraut agreed and PGQ was ultimately backed out of both PostgreSQL 19 and the master branch. Whether it will resurface in time for PostgreSQL 20 is unclear.

Andres Freund said he was worried about the patch from Paul A. Jungwirth, committed by Eisentraut, that extends PostgreSQL's UPDATE and DELETE commands to add a "FOR PORTION OF" clause to work with temporal tables. (Hubert Lubaczewski has a good blog post that explains the new features.)

Haas said that Jungwirth's patch had 17 fixes, including three security fixes, that had to be addressed after freeze. Freund said that several of the bugs seemed fairly obvious. "Not recognizing & testing some complicated interaction, race conditions, etc. are one thing, but the oversights here seem to be pretty ... broad." He felt that this patch was on the high-risk side because the feature was available to users with relatively low privileges, which meant an attacker did not need to have superuser privileges to do "something intentionally adverse".

Jungwirth reported that Noah Misch had found an additional set of bugs in his FOR PORTION OF feature, "with some LLM investigation". He later said that he had also received a report of a memory leak from Tomas Vondra off list. Between Misch's and Vondra's additional reports, the bug count for Jungwirth's patch is now north of 30 since the feature freeze, and there may be others. So far, there has been no indication it will be removed.

Unprecedented situation

Core team member Bruce Momjian expressed concern about reverting patches like Gustafsson's: "We are in a situation where we might be reverting multiple big features, and we don't even know if their removal will affect other changes that were made assuming those feature existed". He said another beta release would be needed and that it felt like the project was in "an unprecedented situation". Jonathan S. Katz announced the extra beta to be released on September 24, and the release candidate and stable release dates are still unknown.

On September 7, Joshua Drake said that with the number of reverts under discussion the confidence in the release would be lower, whether the features were actually reverted or not. He proposed shifting the release to spring 2027. That would allow more time for testing and remove what he considered a "long-standing issue with the development cycle": the fact that PostgreSQL's testing window takes place during summer (and presumably vacation season) in the Northern hemisphere.

He argued that none of his customers had told him they needed PostgreSQL 19: "most people don't even run 18 [yet] so why are we in such a hurry?" He would rather have 19 ship late than for "all these great features and hard work [to be] gutted". So far, no one has responded to the idea of shifting the release to next year.

It seems most likely that the developers will try to get the release out as close to the original schedule as possible. The project hasn't put out a fourth beta release in recent memory, so it's unclear how many weeks it will allocate for beta testing—at least two weeks seems likely. The stable release is usually two to three weeks after the release candidate, so it seems there's a good chance that PostgreSQL 19 will slide into November even if all goes well from here.

Comments (8 posted)

Accelerating the kernel's build process

Kernel developers do a lot of kernel builds. Since the kernel is not a small program, those builds can take a fair amount of time, even on a fast machine. The kernel also has a complex build system; it is probably fair to say that few developers truly understand it, and fewer still are willing to try to improve it. Lorenzo Stoakes, armed with LLM-based assistance, decided to give it a try, though, and has managed to reduce the time it takes to build a kernel — and not by a small amount.

In the cover letter to his 23-part patch series, Stoakes described his tool use for this work:

An LLM was used to first determine where the bottlenecks were then to figure out how to improve them.

It generated a lot of code, much of it hideous.

I extensively audited and rewrote a lot of it, and heavily edited commit messages, the cover letter and comments.

The results of this work were summarized as "allmodconfig builds up to 36% faster, incremental builds up to ~70% faster, and noop builds up to ~90% faster".

How to accelerate kernel builds

Getting to that point required a lot of changes to the build process. Consider, for example, the kallsyms program, which runs during the build to create a table of all of the symbols (function and variable names) in the kernel. There are a lot of symbols, so this table can be large. Since the table can reside in memory, there is a strong incentive to make it smaller; kallsyms thus implements a special compression algorithm, described this way in the source:

Table compression uses all the unused char codes on the symbols and maps these to the most used substrings (tokens). For instance, it might map char code 0xF7 to represent "write_" and then in every symbol where "write_" appears it can be replaced by 0xF7, saving 5 bytes.

As Stoakes (or his LLM) observed, though, kallsyms performs this compression in a highly inefficient way, trying to perform each token replacement in every one of over 150,000 symbols, even though any given token will only appear in a small subset of them. This patch from the series tracks which tokens appear in which symbols, so that substitutions are only attempted in places where they will succeed. This change alone speeds kernel builds by 2-6%.

Even then, though, kallsyms outputs its table as a 37MB assembly-language file that must then be passed to the assembler to be turned into binary data. This patch causes kallsyms to output the binary data in a form that the assembler can pull in directly, reducing build times by up to 11% in the best case. Linus Torvalds applauded this change, but suggested that it could go further: kallsyms could simply output an ELF object, taking the assembler out of the picture entirely. Nobody, he suggested, would normally want to write the code needed to make kallsyms behave that way but, since an LLM is involved, "the obvious step would be to just ask the little helper bot to switch the whole kallsyms to using libelf-devel and writing an object file directly".

The build process uses the nm tool at various stages to query the contents of object files. The output from nm is sorted by default, but the build process rarely needs sorted data. This patch disables the sorting when it is not needed, improving build times by another couple of percentage points. In another patch, the various intermediate linking stages are changed to not emit relocation data, which is unneeded at that point, again improving build times by 2% or so.

Sometimes, though, the use of nm can be avoided altogether. The build process currently runs that tool on kernel objects, feeding the output to the mksysmap sed script to create data suitable for feeding to kallsyms. This patch replaces that script with new kallsyms code that reads ELF files directly and extracts the needed data, bypassing nm entirely. In the best case, this change improves build times by 12%, though the improvement seen in most cases is significantly more modest. Another modest improvement is had by not allocating the .modinfo section in the kernel image. This patch, which caches information about whether each object file is part of a module, speeds build performance, especially in larger directories.

The kernel contains a lot of header files, with the result that object files can have a huge number of dependencies. The make utility, it seems, spends a lot of time checking these dependencies in a single-threaded mode, slowing down the build, especially in cases where little has changed. This patch adds a new utility, called depcheck, that performs dependency checking in a more efficient way, taking care to only stat() each file once. This change may not last, though: Torvalds suggested that much of the trickery used to make dependency checks work properly may no longer be needed: "That whole thing is from a quarter century ago, and nobody has dared look at that code since". So the better solution might be to just remove it entirely.

Then, there is this patch, which aims to reduce the number of times that the build process invokes the compiler to probe whether a specific feature is supported. The compiler and its supported options do not change over the course of a kernel build, so it makes sense to remember which options are supported after querying them once. A new set of configuration options is added as a way of remembering and testing that information.

The modpost tool postprocesses symbol versions in loadable modules; that work involves calculating a lot of MD4 hashes. In current kernels, the data to be hashed is accumulated one byte at a time; this patch does the hashing on a per-file basis, yielding a build-time improvement of up to 6%. However, this functionality is being removed from modpost, so this change will be dropped from the series. This patch adds more caching to modpost, reducing build times by up to 12% in some cases. Finally, modpost generates a C file containing a descriptor for each module it handles; that file must then be compiled into an object file. As described in this patch, the resulting overhead is not small:

These files are compiled like any other kernel C file with all of the -include preamble, as well as including linux/module.h, header dependencies generated by fixdep of a few hundred headers, an objtool run and if LTO is being performed, a link is performed to generate native code.

On an x86-64 allmodconfig build 11,189 *.mod.c files are built, each taking ~0.24s of CPU time to compile, and module finalisation as a whole 6,300 CPU seconds, or 64 seconds of wall time when run over 128 threads.

Changing modpost to output assembly code, thus avoiding the compiler step, removes this overhead, speeding an allmodconfig build by 11% (and no-op builds by much more). Nobody will complain about an improvement like this but, again, Torvalds suggested that an even better approach would be to generate the object file directly and remove the assembler from the process as well. Beyond any speedups, he said, that change would remove the .mod.c files from the build entirely.

Other changes include this patch, which causes the finalization step at the end of each module build (which runs in milliseconds for any given module) to be batched, overcoming the inability of make to dispatch thousands of little tasks efficiently. Another patch parallelizes the hashing of files in modpost. There is a rather more complex patch that adds parallelization to the objtool utility, with large effects on some builds.

Then, there is the matter of Rust. Currently, the kernel's Rust crates are compiled serially, one at a time. That slows the kernel build considerably now, and will only slow things further as the amount of Rust code grows. This patch uses the -Zthreads option to rustc to build the crates in parallel; that option was added in Rust 1.84, which is (slightly) older than the minimum version (1.85) required by the kernel. The patch will also attempt to use the future --jobs option if it is available. Björn Baron warned, though, that parallel builds in rustc have not fully stabilized, so it is not clear that this mode should be used by default. This other patch, which allows Rust code to be built in parallel with the building of C code, should be rather safer.

The final patch in the series causes the build system to use pigz to compress the kernel if it is available. pigz is able to perform compression in parallel, significantly reducing the time required to complete this step.

In summary

See the plot (above) for another look at how these patches affect kernel-build times. All told, they add up to a significant reduction in the time required to build a kernel. It seems unlikely that developers will complain much about that. As long as they do not introduce build problems (and only the parallel Rust change appears to have any risk of doing that), there should not be much opposition to their merging.

Naturally, it was necessary to try it out. On my system, which is better suited to article writing than fast builds, a full kernel build with an empty ccache reliably takes about 960 seconds. With this patch series applied, that time drops to about 860 seconds (both are wall-clock time, the CPU-time numbers are quite a bit higher). That is roughly a 10% decrease in build time. The clean-ccache case was chosen because it is easily reproducible, but it is also dominated by actual compile time and is the scenario likely to show the least improvement. With a fully populated ccache, the patches made a difference of a few seconds in a roughly 90-second build.

At any given time, there must be an impressive number of machines performing kernel builds all over the planet. Developers will be running some of those, but there are probably far more in the many continuous-integration systems that are running. So build-time improvements may save more than developer time; they will reduce the carbon impact of the development process overall. This work, it seems, should have been undertaken long ago. One wonders what other savings may be waiting for a suitably motivated developer to take a close look.

Comments (81 posted)

Adding BPF to blk-iocost

The scheduling of block I/O requests has long been a challenge for operating-system kernels. For many years, the performance characteristics of rotating drives meant that putting considerable resources into request ordering was worthwhile. In a world with fast, solid-state drives, scheduling is more concerned with enforcing fairness between competing users while being fast enough to keep up with drives that can perform millions of I/O operations per second. The blk-iocost I/O controller was designed for the solid-state world and generally performs well, but there is always a desire to do better. This patch series from Tao Cui aims to make blk-iocost more flexible by enabling the loading of a BPF program to make cost decisions.

The blk-iocost controller was first covered here in 2019, when it was still known as "io.weight"; it was revisited in 2024. To summarize, it works by allocating a percentage of a block device's bandwidth to each control group. Every requested I/O operation is then evaluated and an associated cost, expressed in terms of how much of the device's time the operation is expected to take, is assigned. Each device has a virtual clock ("vtime"), as does each control group; executing an I/O operation advances the requesting group's vtime by the assigned cost. Operations are only dispatched to the device when the group's vtime is far enough behind the drive's vtime to accommodate them.

See this paper for more details on how blk-iocost works. One key point is that its operation is split into two parts. The calculation of the cost for each operation uses a number of pre-computed parameters in order to be as efficient as possible and not slow down the I/O operations themselves. A separate "planning" phase, which runs every few milliseconds, adjusts the weights of all of the control groups in order to reallocate bandwidth from groups that are not using their full allocation to others that are more busy. The planning phase also tweaks the rate at which the drive's vtime advances in an attempt to keep the drive fully busy — but not so busy that response time begins to suffer.

Cui's cover letter describes how the current cost model works and its shortcomings. To summarize: requests that are deemed to be sequential are assigned a relatively low cost, while those that look random are assigned a much higher (112x) cost. Sequential access is faster, even on solid-state drives, so a group that is performing mostly sequential operations should be entitled to more of them than groups performing random operations.

The problem, Cui says, is that the heuristics for determining the randomness of any given request do not work well for all workloads. A group that has two processes performing sequential operations can, if those operations are interleaved, appear to be doing random I/O; the controller will penalize that group, causing its I/O performance to suffer. Meanwhile, random operations within a sufficiently small window can be judged to be sequential, leading to an unfair boost for the relevant group. Additionally, some devices can behave differently depending on the type or the placement of an I/O operation, but blk-iocost does not take that into account. Given the range of devices and technologies out there, it may not even be possible for a single, general-purpose controller to take all of the relevant factors into account.

The solution, as always it seems, is to enable the loading of a BPF program so that the cost calculation can be customized. Cui's patch set only addresses the cost-calculation part of the controller; the planning portion is unchanged.

Specifically, the series enables the loading of a new type of struct_ops program built around this structure:

    struct iocost_model_ops {
	u64 (*calc_cost)(u64 opf, u64 nbytes, sector_t sector,
			 struct blkcg *blkcg, u64 model_flags);
	void (*blkcg_online)(struct blkcg *blkcg);
	void (*blkcg_offline)(struct blkcg *blkcg);

	/* model name, used to select the model through io.cost.model */
	char name[16];
    };

The core of this structure, and the only required operation, is calc_cost(), which calculates the cost of an I/O request. The opf parameter is a set of flags describing the request, including whether it is a read or a write, whether a cache flush is needed, and so on. nbytes is the size of the request, sector is the location of the desired blocks on the drive, blkcg is the block control group that controls the requesting process, and model_flags may contain a single flag, IOCOST_COST_F_MERGE, indicating that the current request is being merged into another that already existed. The return value is the cost that will be assigned to this request.

The other two functions, blkcg_online() and blkcg_offline(), are optional; if they exist, they will be called when this controller is added to (or removed from) a control group, allowing the BPF program to manage any housekeeping it may need. Loading the program does not, in itself, attach the new cost function to any devices; that is done by writing to the io.cost.model file in the control-group directory using the name provided in the iocost_model_ops structure.

Included in the patch set is an example program that attempts to improve performance for control groups that contain multiple processes performing sequential I/O. It uses a BPF map to keep track of each I/O stream that it observes; if a request looks sequential within any of the tracked streams, it is charged as a sequential request. The example program is able to track up to four independent streams, though the number of streams actually tracked will be lower if there is also random I/O being done.

This is still early-stage work, in its third RFC revision. The interface has evolved considerably since the posting of the first version on September 8. So it would not be surprising to see further changes before this feature is merged. Eventually, though, a version of this work does seem likely to be accepted, adding yet another place where BPF can be used to modify the operation of a core kernel subsystem.

Comments (none posted)

Lessons learned as the Debian Project Leader

What is it like to be a Debian Project Leader (DPL), or a former one? According to Andreas Tille, who stepped down this year after two consecutive terms as DPL, you'd have to be one to know. At the recent MiniDebConf in Winterthur, Switzerland, Tille spoke about what he learned while serving as DPL, some of the initiatives he led, mistakes that he made, and his thoughts on the general resolution (GR) on large language model (LLM) usage in Debian.

I did not have the opportunity to attend the MiniDebConf in person, but I watched the video of the session shortly afterward. The slides are also available, as are the slides and videos for many of Tille's talks back to 2001.

A short changelog

Tille said he was often asked what it feels like to be a former DPL. The only way to find out is to do it—and he hoped that a future DPL was sitting in the audience. "I can recommend it. It was less demanding than expected. So just try it, and then you will know." His wife had told him that he had also developed as a person while being DPL, "and I take this as a compliment".

He wanted to start with "a short changelog" of some of the projects he had worked on during his tenure. One of the things he wanted to do as DPL was to attract newcomers to become Debian contributors. He began a Bug of the Day program to train new people to triage and fix bugs, with a web page that lists one or more bugs that might be good candidates for inexperienced people looking to become contributors.

It was an interesting project, Tille said, that helped him learn a lot about Debian, such as the need to address social and cultural issues as well as technical ones. "I felt like a janitor taking a torchlight in the cellar and looking for things that needed clean up [...] and I learned way more than expected myself. And what I learned is that doing something and talking to people is essential for the process". The need for the DPL to spend a lot of time talking to people was a recurring theme in the talk.

Archive and DFSG teams

Tille said that he had spent "a lot of time in video conferences" talking to contributors about the ftpmaster team, which was responsible for two distinct tasks. The team was responsible for both the technical work of caring for the servers and software used in the process of bringing packages into Debian's archive, as well as the non-technical work of ensuring that packages entering Debian's new queue met packaging requirements, including the Debian Free Software Guidelines (DFSG).

"We called it the new queue, but actually it was a new heap", he said. Debian developers had complained for many years about long wait times for packages in the new queue to be reviewed and approved. Tille said that there was "a hint that was born inside the team to just split the distinct tasks", which is what happened.

"Splitting up a core team right before a stable release is not a good idea", Tille said. He waited until a few months after Debian 13 ("trixie") was released to split the ftpmaster team into two new teams. He created the Archive Operations Team for the technical job of managing the infrastructure, and the DFSG, Licensing, & New Packages Team (DFSG team) to do the non-technical job of reviewing packages entering the new queue. He said that the separation of teams has led to faster processing of the new queue and a more transparent system. LWN covered Andrew McMillan's report on the DFSG team's progress in July.

How to step back

Something unusual happened during Tille's term that he wanted to highlight: all three members of Debian's Data Protection Team said that they no longer wished to do the work and quit, which led to an urgent call for new members. The interesting thing about this was that they actually said that they no longer wanted to do the job. "I think the very good thing, and not so typical thing, was that the data protection team told the world that they will not do their job anymore."

Debian is a volunteer organization, and people step up to do a task but then their circumstances change. People have valid reasons for stepping back, but then fail to tell their teammates that they are no longer available. By announcing they wanted to quit, the team enabled a smooth handover to a new team that is now doing the work.

That led Tille to thinking about how delegations are handled within Debian. Typically, a delegation to a team has no time limit attached—it is valid until the DPL revokes it or a person announces that they are stepping down. But, he said, people are reluctant to tell their friends that they are unable to do a task any longer. Stepping down from a position, "is an active process from a person who is burnt out" or otherwise in need of a break.

Tille thought that it might be better to ask delegates if they are still interested in remaining on a team. It would be easier for them to say "no" when asked, rather than proactively stepping down. He raised the idea of time-limited delegations in February 2026 in one of his regular "Bits from the DPL" emails.

In some internal discussion people have raised the question of how Debian should deal with delegates becoming inactive, and how to do so in a way that is both effective and non-confrontational. I have read multiple thoughtful suggestions that converge on a common theme: we should normalize renewal and rotation, rather than treating changes as exceptional or adversarial events.

Tille said that he did not receive any responses about the idea, so he thought there was consensus that it was a good one. Tille asked incoming DPL Sruthi Chandran if it was acceptable to make all delegations in Debian time-limited, to end in October this year unless she renewed them, and she said that was fine. But when the change was announced, it was more controversial than he expected.

Some of the delegates, "stood up and said, 'You should have asked us'", even though Tille had publicly announced the idea. They were offended by the change in delegation procedure because "they felt some kind of distrust". Tille said that the process would have gone better if there was more communication with the delegates. "This is a problem in Debian in general, that we don't find so much time to talk to each other [...] I learned that governance is hard, but we have to do it somehow, and silence is hard to parse". Tille said that there was some clean-up to do after the announcement, but thought that things were fine in the end. "I made failures in my DPL term, yes", he said, but hoped that he admitted to them all openly.

Collaborative maintenance

When Tille began his term as DPL, there were more than 4,000 packages that were not hosted on Debian's GitLab-based collaborative-development platform, Salsa; one of his goals was to reduce that to number to 2,000 or fewer packages. He said that, at the time of his talk, there were only about 1,700 packages not on Salsa "which means no version-control system, or GitHub, or whatever". (The Debian Trends web site tracks the VCS hosting field for packages in the archive.) Even though Debian is a volunteer organization, he believes that it is good to have a shared collaboration platform. "I think this kind of professionalism is what makes volunteer projects scale".

Debian also has strict maintainer ownership of packages. "So if your name is in the maintainer field of the package, it's your package." Historically that was sensible, he said, "because we had experts for, say, 500 or 1,000 packages, and these were really the experts who know what this package is doing". Now Debian has more than 41,000 packages, "and the maintainer field is not always filled with a person who is really competent". He felt he could say that because he knows that he is maintaining packages where he is not an expert, and needs help from other colleagues to ensure he's on the right track. He works on Debian Med packages related to medicine and bioinformatics, while he is a physicist by profession.

But, Tille said, "my name is not on any maintainer field of any Debian packages". Instead, the packages he works on are always maintained by a team of packagers rather than a single packager. He is an uploader for packages, which means that he can make and push changes to the archive, but others can as well. He believes that is the best way to go for the future of packages in Debian.

"This historical strict maintainership is an increasing source of friction in Debian, because its hard to know whether a package is orphaned or just needs no work". Some packages may not be touched for years, but there is no indication why that may be. Silence, he noted again, is hard to interpret, "and also hard to accept for me, because there are maintainers who simply don't answer". The maintainer may be uploading from time to time, but "if you send them an email, they are not answering". He stressed that silence was not the rule, but an exception. "But it's a consequence of the fact that we have this strict maintainership that we can't touch these packages."

Tille had a goal of loosening the strict ownership of packages during his terms as DPL, but he failed to do so. "Debian is a great community, but we are a community of individuals. We have a lot of good teams, but we also have individuals who do not interact." That is fine, he said, but he thought that the project would benefit from stronger collaboration.

He mentioned some of the attempts to improve collaboration on packages and loosen ownership over the years, such as the introduction of non-maintainer uploads (NMUs), the Missing In Action team that checks to ensure maintainers are still active, and the package-salvaging process (more frequently referred to as "intent to salvage" or ITS) for packages that have not been officially orphaned but may be poorly maintained or completely unmaintained.

In late 2024, Tille suggested flipping the default assumption of package ownership to an open maintainership model by introducing a Debian control file file for packages, perhaps called "debian/dont_touch_my_package". A maintainer could indicate that they wanted to discourage others from making changes to a package by documenting their reasons. "This file could include a timestamp, and we could establish an agreed-upon timeframe for refreshing the statement to ensure its continued validity." If the file is not present, then it would be assumed that the package was open for maintenance by other Debian contributors.

That idea has not caught on, at least not yet, but Tille said that he would try to pick it up later. "You don't need to be the DPL to change anything in Debian. It's absolutely not necessary."

Responsible use of LLMs

LLMs and other generative-AI tools were just starting to make their presence felt in open-source communities when Tille began his first term as DPL in 2024. By the time he finished his second term, they had become an important and divisive topic within Debian. Not surprisingly, Tille had some thoughts about Debian's LLM usage GR, its vote to allow "responsible use of generative AI", and his own relationship with LLM tools.

Tille is German, and English is not his first language; thus, he is forced to communicate in a foreign language as part of participating in Debian. "And it's not only the language; you have also a foreign culture because Germans are quite straight. You usually tell what you think, and it's not very well perceived in other cultures." That led him to explore the use of LLMs to try to communicate more effectively as DPL.

He said, in what may have been the least-surprising disclosure in his talk, "you don't always get friendly emails as a response" when serving as DPL. That could lead to an angry response in return, but Tille opted to use LLMs as a buffer to prevent him from "responding quite emotionally", and instead try to discover the underlying problem and if the person had a point. He added that, separate from LLM usage, his advice was to avoid answering on the same day. "Draft your answer, sleep on it, send it the next day, or even better: delete it the next day." LLMs, Tille said, can help to focus on a topic, find new interpretations of an ambiguous response, and so forth.

So for me, LLMs are a legitimate tool; a tool for just clarity, not a ghostwriter. I never say to an LLM, "Please draft an email for me". No. It's kind of a depersonalization, but if you are in a responsibility situation like the DPL is, it makes sense to think wider than, or think beyond your own personality.

I own the intent of what I'm writing, and the LLM helps me with the right expression. So what I always can recommend: LLMs can be kind of an editor. Use this [as an] editor, but not as the author of your words.

He stressed that he drafted his own responses, reviewed all changes, and thought that he owned the final results. "There are people who might disagree, but this is my view on how to use [LLMs] responsibly."

Tille said that his view on the GR was that he perfectly understood the people who wanted to restrict LLM usage. "I know all of the drawbacks of LLMs. I think I'm aware of them". He thought it would have been better for those who wanted to restrict LLM usage to discuss possible restrictions, for those who wanted to enable LLM usage to propose their ideas and to find common sense ways to propose restrictions properly. Instead, there was a "heap" of proposals that "nearly made our software for voting [explode] because we have so many, admittedly interesting, proposals".

In the end, the "Responsible Use of Generative AI" option won, which he interpreted as Debian's voters saying "'we want to stand [by] the existing standards for quality, correctness, and maintainability', which is the core of Debian, in my opinion". He believed that it is not well-perceived within Debian if "someone is telling you, 'you are not permitted to do that'". That is why the non-permissive suggestions failed to even meet the majority requirements to pass, as well as failing to beat the permissive options. "I understand the reasons why the people wanted to have some restrictions, but this is not the way how you can push this on the Debian maintainers".

Lessons learned

Tille said he would cover some of the lessons he learned as DPL, starting with focusing on specific and reachable goals. This meant listening carefully to the community, and trying to find the input that indicates a support for change. He thought he could reach more of his goals, but learned that he had to concentrate on what was really important and leave other things to future DPLs.

"You also need the courage to make hard decisions." He said that he had to make the hardest decision of his life during the DPL term, though he did not specify what that decision was. "My specific goal required decisions which I'm absolutely not happy about, but they were necessary anyway". He added that a DPL should not fear pushback, but it will come.

Tille said that he had also learned that it was important to ensure cooperation from volunteers who will back up a decision. He again stressed the importance of talking to people in the community to find out their opinions and whether the "silent majority" agrees. A lot of people, he said, may be "on your side, but they are too busy to pronounce it on the mailing list".

It is not normal to acknowledge mistakes openly in "normal politics", Tille said, but that is wrong. "I think it was a good idea that I acknowledged my mistakes openly."

His favorite lesson, he said, was that the Debian Med team was able to function without him. In his 2024 DPL platform, Tille had talked about starting and leading the team, which had been "a significant part of half of my life". Being DPL was not as time-consuming as he was afraid it would be, but he had stepped back from his work with the Debian Med team while serving as DPL. "The good news is, the team kept on working." It made him happy to learn that the team could survive "at least for a couple of years" without his involvement. That was not, he noted, always the case for Debian teams when a main contributor steps back for some reason. It is harder, he had learned, for a contributor to step back gracefully than it is to find contributors to step up in the first place.

He had three "final" lessons. The first was that the opinions of Debian contributors are more diverse and divergent than he expected. The second was that—no matter what he said—the interpretation was "it's the DPL speaking". He wanted to "keep on being Andreas Tille talking to his friends", but he had to deal with treating him as the DPL anyway.

The third final lesson was that he couldn't do what he wanted to do in one term. It took him a year to understand the job before he could make the changes he wanted to make. He thought that it was a good thing that the DPL term was only for one year, because if he had known in advance that it would take two "maybe I would not have stepped up for the position. So do it one year and see what happens".

He finished his talk with his favorite quotation, which he credited to Giuseppe Tomasi di Lampedusa: "Everything must change so that everything can stay the same". Tille said that he liked that a lot, and it was fitting for Debian.

Questions

Marc Haber asked the first two questions. He wanted to know Tille's opinion about redefining "or putting some more definition" on the duties of package maintainers. The second question was what could a DPL do to bring more technical leadership into Debian.

Tille responded that the first question related to his own opinion that the historical model of package ownership is "not really meeting the current standards we need". He said that Debian needed to talk about it and that there was a lot of agreement that it needed to change, but "if you propose something, people stand up and say, 'No, I think it's good and we should keep on doing this'". The problem is that they may not represent the "about 1,000 Debian developers, who are all very busy doing their own stuff and don't have time to read long threads and mailing lists". He felt that the LLM GR illustrated this; the people who take part in discussions may not represent the majority view.

As to Haber's second question, Tille said that Debian did have technical leadership in some fields, but not all. The way to attract more was by "doing a good job, being open, being friendly, and inviting for them. But I also have no short and clear path to do this".

Another audience member said that Tille had talked about his Bug of the Day initiative, but he had not talked about whether it had succeeded. Did it bring in more people? Was it successful? Tille said that there were three people who are now Debian maintainers who were not before the program started, but overall "I was not able to break this barrier" of getting people who are "too shy" to participate and ask questions in order to become contributors. He admitted he also had no idea how to solve that problem.

Comments (none posted)