Rust, Linux, SLOs and All Things Performance at P99 CONF
P99 CONF is a free virtual conference that’s all about performance. Thousands of latency-obsessed engineers from around the world come together for P99 CONF each year. It’s purely technical, intentionally virtual, highly interactive and open source-focused.
ScyllaDB launched P99 CONF in 2021 to connect and foster the community of technologists who obsess over low-latency engineering, and it’s really caught on, thanks to many knowledgeable speakers sharing their optimization strategies, lessons learned and insights on what’s next for performance.
As we ramp up for P99 CONF 24 (more on that later), we thought it was a good time to share some of the most talked-about sessions from past P99 CONFs. You can also binge-watch more than 150 tech talks (free, ungated) in the on-demand library.
Get a complimentary P99 CONF 24 conference pass.
Rust and Low-Latency Systems’ Future – Bryan Cantrill
No recap can do justice to the information-packed and impassioned keynote by Bryan Cantrill, co-founder and chief technology officer at Oxide Computer Company.
In just 20 minutes, he took the audience on a history of computing to date, provided a blazing critique of where we are today, and gave the audience a glimpse of where the industry is heading next. We wrote it up in article format in “Bryan Cantrill on Rust and the Future of Low-Latency Systems,” but trust us — you want to watch the video for this one.
Here’s a tease: Cantrill’s conclusion, in his own words:
“Rust is actually the first language since C to meaningfully exist at the boundary of hardware and software. And this is what points us to the future.
“Wright’s Law means we’re going to have compute in more places. We are already seeing that. Those compute elements are going to be special-purpose. Don’t wait for your general-purpose CPU to be shoved down to a SmartNIC. It’s going to draw too much power. We can’t have memory that fast down there.
“But what we can put down there is Rust. Rust can fit into these places. We are going to see many more exciting de novo hardware-facing Rust systems that — thanks to no_std — will be able to build on one another.
“It’s a very exciting time to be developing high-performance low-latency systems, and the Rust revolution is very much here.”
Whoops! I Rewrote It in Rust – Brian Martin
The scalability and efficiency of Twitter (now X) services relies heavily on high-quality cache offerings. The team there developed Pelikan as a caching system when Memcached and Redis didn’t fully meet their needs. Their No. 1 priority for Pelikan was “best-in-class efficiency and predictability through latency-oriented design and lean implementation.” This was initially achieved with a C implementation. However, two subsequent projects introduced Rust into the framework with rather impressive development speed.
When they decided to add TLS support to Pelikan, Twitter software engineer Brian Martin suspected it could be done faster and more efficiently in Rust than in C. But to gain approval, the Rust implementation had to match (or beat) the performance of the C implementation.
Initial prototypes with the existing Rust-based Twemcache didn’t look promising from the performance perspective; they yielded 25% to 50% higher P999 latency as well as 10% to 15% slower throughputs. Even when Martin doubled down on optimizing the Rust prototype’s performance, he saw minimal impact. After yet more frustrating performance test results, he considered several different implementation approaches. Fortunately, just as he was weighing potential compromises, he came across a new storage design that made it easier to port the entire storage library over to Rust.
Martin went all in on Rust at that point, with a simplified single-threaded application and all memory allocations managed in Rust. The result? The 100% Rust implementation not only delivered performance equal to — or exceeding — both the C implementation and memcached: It also improved the overall design and enabled coding with confidence thanks to “awesome language features and tools,” which Martin then dove into.
Keeping Latency Low and Throughput High – Avi Kivity
Throughput and latency are in constant tension. ScyllaDB CTO and co-founder Avi Kivity focused his keynote on discussing how high throughput and low latency can both be achieved in a single application by using application-level priority scheduling.
Kivity began by outlining the stark contrast between throughput computing (OLAP) and latency computing (OLTP) and explaining the scenarios where it makes sense to mix these two types of jobs in a single application. When mixing is desired, two core actions are essential:
- Isolate different tasks for latency jobs and throughput jobs so you can measure and control them.
- Schedule them in a way that allows the latency jobs to be completed quickly, without interference from the throughput jobs.
But the devil is in the details. Do you take the common approach of isolating the tasks in threads and letting the kernel schedule them? It’s generally easier, but it doesn’t yield sufficient control or efficiency to achieve the optimal results.
The alternative is application-level task isolation. Here, every operation is a normal object and tasks are multiplexed on a small number of threads (ideally, one thread per logical core, with both throughput- and latency-sensitive tasks on the same thread). A concurrency framework assigns tasks to threads and controls the order in which tasks are executed. This means you can fine-tune everything to your heart’s content, but all that fine-tuning can be addictive, drawing your attention away from other critical tasks. More advantages: low overhead, simpler locking, good CPU affinity and fewer surprises from the kernel. It’s a less mature (though improving) ecosystem, but Kivity feels strongly that the extra effort required pays off immensely.
After visualizing what the execution timeline looks like, Kivity went into the finer details of switching queues, preemption techniques and using a stall detector. To wrap up, he explained how it all plays out in a real-world example: the ScyllaDB database.