Zymtrace Launches End-to-End GPU and CPU Observability
While detailed instrumentation of CPUs has been available for some time, GPUs are still commonly treated as black boxes. Zymtrace, a new profiler designed to provide detailed introspection for both CPUs and NVIDIA GPUs, as well as the interactions between them, aims to solve this problem. It comes from zystem, a young startup and member of the NVIDIA Inception program on a mission to accelerate the world’s transition to cost-efficient and sustainable computing.
Following stints at Microsoft (Yammer), Cisco (AppDynamics) and Elastic, zystem co-founder and CEO Israel Ogbole realized existing tools for GPU observability were inadequate.
“It dawned on me and my co-founder and CTO, Joel Höner, that the future of compute is heterogeneous. Compute is no longer CPU-centric. Enterprise workloads increasingly — and will continue to — rely on GPUs and AI accelerators, distributed across cloud and on-premises environments,” he explained in an interview. “Existing observability tooling relies on surface-level NVIDIA Data Center GPU Manager (DCGM) and NVML (NVIDIA Management Library) metrics, which lack CPU context as well as detail insights into the inner workings of the GPU; so we decided to take on the challenge to help enterprises optimize GPU and CPU workloads based on what we’ve learned on the CPU side.” Ogbole and Honer were part of the team that created, open-sourced, and donated the eBPF CPU Profiler to OpenTelemetry.
They founded zystem last year and, following an initial release in December, they’ve just announced the public availability release of GPU support for zymtrace, the company’s distributed profiler with a modern web frontend built using a combination of ReactJS and Rust, compiled into WebAssembly (WASM) with rendering powered by WebGPU/WebGL.
While there are other GPU profiling tools on the market, Ogbole believes zymtrace fills a critical gap.
“Our research indicates that GPU idling cycles are primarily caused by CPU bottlenecks, where applications become partly CPU limited when the processor cannot feed data to the GPU fast enough. This occurs because the CPU is switching contexts and wasting time waiting for new context to load from memory, which in turn causes the GPU to stall,” Ogbole said. “Our unique selling point is that we can show users the entire code path, providing end-to-end visibility from the CPU down to the GPU, so you can see where inefficiencies lie on both sides.”
The zymtrace GPU profiler builds stack traces, from high-level PyTorch and JAX code and user-space libraries through native code and CUDA kernels, all the way down to the Linux kernel. It also captures GPU instruction mnemonics, GPU stall reasons and memory offsets, correlating them back to the exact CUDA kernel function that triggered the execution.
Ogbole also said zymtrace provides more detailed GPU information than many competitors.
“Most GPU tools offer only basic NVML metrics, which show utilization percentages without revealing what’s actually happening inside your code,” he said. “NVIDIA has Nsight Compute, which does provide good code introspection. But it comes as a clunky desktop application you have to install, and because it isn’t distributed, you have to collect data from one workstation at a time, all while incurring extremely high performance overhead.”
As an example, the zymtrace AI flamegraph below shows a JAX neural network training job that is inefficiently utilizing the GPU due to excessive small operations and kernel launch overhead. The training is dominated by numerous tiny CUDA kernel calls and memory transfers rather than sustained compute-intensive operations, resulting in math pipeline throttling on the GPU.
Beyond GPU Observability
This level of detail allows zymtrace to solve a number of tricky use cases. Alongside being able to detect why GPUs are idle and the CPU bottlenecks that impact GPU utilization, it can help with the notoriously challenging problem of finding the optimal batch size for inference.
Optimal batch size is dependent on application-level requirements, and it’s essentially a trade-off of latency against throughput. Finding the correct batch size typically involves a great deal of trial and error. However, in cases where maximum throughput is required, “zymtrace can reveal where your model shifts from memory-bound to compute-bound, reducing inference costs and maximizing throughput on your specific hardware configuration,” Ogbole told us.
Another place where zymtrace can improve performance is around GPU kernel launches. Not to be confused with an operating system kernel, “compute kernels” are routines that are compiled for high-throughput accelerators, such as digital signal processors (DSPs), field-programmable gate arrays (FPGAs), or, in this case, GPUs.
Each compute kernel launch introduces latency. This is because the GPU sits idle while waiting for the CPU process to launch the next step. Multiple GPU compute kernels can, however, be combined into one using a technique called kernel fusion.
Kernel fusion allows the reuse of data that’s already in registers, eliminates redundant load-store operations, and reduces overall memory bandwidth pressure. Ogbole said, “zymtrace identifies these fusion opportunities in your specific workloads. This helps achieve up to 30% performance improvements without changing your underlying models or algorithms.”
How Zymtrace Works
Zymtrace consists of two main components: a profiler and a set of backend services.
The profiler is packaged as a single binary/container image. Users can enable the GPU/CUDA module on demand through zymtrace’s configuration. Itcollects information about kernel launches and completions. It also “samples high granularity information about precisely which GPU instructions are running on the GPU’s compute cores, the reasons preventing the kernel from making progress (stall reasons), and the performance profiles of resource-intensive processes on the node,” Höner told The New Stack.
If GPU profiling is enabled, zymtrace uses eBPF to hook into the kernel launch. This is possible because it uses uprobes, a Linux mechanism that can capture user-space function calls with minimal friction. Zymtrace also has built-in low-level engineering components that do not rely on eBPF to efficiently profile general-purpose and accelerated workloads with minimal overhead.
How Zymtrace Aims to Reduce Developers’ Roadblocks
Zymtrace runs entirely on-premises. There is no software as a service (SaaS) offering, although it can be deployed and orchestrated using the zymtrace Helm charts in under five minutes.
Currently, zymtrace is also available only for Linux systems — neither Windows nor MacOS support is available. The CPU profiler supports AMD64/x86_64 on Linux Kernel 5.4 or higher and ARM64 on Kernel 5.5 or higher. The GPU profiler is supported only on NVIDIA GPUs and requires CUDA 12.x or higher.
Standard pricing currently starts at $1 per core per month plus $0.25 per TFLOP per month for data center GPUs. Volume discounts are available for enterprises profiling 30k+ cores.
The product is already impressive, and Ogbole was keen to emphasize that zystem is just getting started. Though not available in the initial release, he showed me a demo developer build that offers recommendations to guide engineers on where to focus their efforts. Ogbole expects this should improve efficiency and “reduce the barrier for developers to understand flamegraphs,” a visual representation of software stack traces portraying the execution flow of a program.
Zymtrace is also working on providing CO2 estimates based on the type of CPU (ARM or x86), the cloud provider, the region where the server is located, CPU core usage, GPU power draw and the Power Usage Effectiveness (PUE) of the data center. The methodology and PUE data are based on research from Cloud Carbon Footprint.
You can try the product yourself, with the free trial plan.
