Codethink

What is Deadline Scheduling in Linux?

2026-09-08T00:00:00+01:00

Real-time operating systems are ubiquitous in industries such as automotive and aerospace where systems are required to operate under strict timing constraints with possible catastrophic consequences for failure.

In real-time systems, tasks must be executed both with the correct behaviour and within precise timing constraints. In other words, a task is only executed correctly if it produces the correct logical result before some deadline. In a multi-tasking operating system such as Linux, a scheduler is responsible for coordinating access to hardware resources, such as the CPU. The Linux scheduler supports different scheduling policies depending on the workload, and for non-real-time workloads, scheduling policies such as SCHED_OTHER and SCHED_NORMAL are sufficient. For real-time workloads, the Linux scheduler supports some additional policies and adds support for pre-emption: temporarily pausing a running task to allow a high priority one to run instead.

Due to the industry demands for real-time systems, this article explores Linux's capabilities to meet real-time schediuling constraints. This article details how real-time scheduling works in Linux, specifically deadline scheduling, including its limitations and how to mitigate those limitations.

To schedule real-time tasks Linux provides two schedulers: the POSIX real-time scheduler, and the deadline scheduler.

The POSIX real-time scheduler, which provides the first-in-first-out (FIFO) and round-robin (RR) scheduling policies, schedules each task according to its priority. The task with the highest priority is scheduled first. These scheduling policies differ only when tasks have the same priority. The FIFO scheduler will schedule the task which arrived first running it till completion or until it yields. The RR scheduler will schedule tasks in a round-robin fashion, executing each task up until a maximum amount of time, at which point the next task execution is started and the current task is moved to the back of the queue.

Deadline scheduling works very differently; it instead schedules each task according to the task’s deadline. The task with the promptest deadline will be served first. This means that when a task is requesting to be scheduled, it is required to provide parameters about its deadline and execution. These parameters are provided to the scheduler via the sched_setattr syscall:

Period: the time between providing the task it's runtime again, and the last time it was provided, in nanoseconds.

Deadline: the time from the start of the period by which the thread must have finished its execution for that period, in nanoseconds.

Runtime: the thread’s expected worst-case execution time in each period, in nanoseconds.

Reclaim (optional): whether the kernel should provide the thread with unused CPU time in a period if the thread hadn’t finished its execution

For example a process scheduled with:

#define SCHED_DEADLINE 6

struct sched_attr attr = {0}; attr.size = sizeof(attr);
attr.sched_policy = SCHED_DEADLINE;

/* Configure time parameters in nanoseconds */
attr.sched_runtime = 10 * 1000 * 1000;  /* 10 ms CPU runtime */ attr.sched_deadline = 20 * 1000 * 1000; /* 20 ms relative deadline */ attr.sched_period = 30 * 1000 * 1000;   /* 30 ms period */

Would repeat every 30 milliseconds, and would be finished by the 20th millisecond in each of those 30 millisecond blocks. The process would be allowed to run for 10 milliseconds. This can be visualised as:

Although the deadline schedulers parameters look complex, they attempt to remove the complexity from the calling process. The scheduler hides the complexity of deciding task priority and fitting the task’s execution around other scheduled tasks.

Because the deadline scheduler knows the required CPU time for each deadline task, it is able to stop the system becoming overloaded by implementing a check for each new schedule. This acceptance test by default works to keep the utilisation of a core at less than 95%, therefore allowing some CPU time for system tasks to run. This acceptance test is controlled by kernel.sched_rt_runtime_us and kernel.sched_rt_period_us sysctl configs, which default to 950000 and 1000000 respectively.

This is sufficient for a single-core system, however for a multi-core system this is not. Dhall’s effect shows how the global deadline scheduler acceptance test is unable to schedule a task even though there is sufficient CPU time.

This limitation can be avoided by manual partitioning of the system into ‘cpusets’ each containing one CPU. This is advantageous as it allows for deadline scheduling in a multicore system to have a higher overall utilisation, however it pushes the responsibility of managing this configuration onto the user.

Common questions at this point are: how do these different schedulers interface with each other? Should the user manage the interface between deadline scheduled tasks and other tasks?

The deadline scheduled tasks have a higher priority than any other tasks, including other real-time tasks such as FIFO or RR scheduled tasks. Therefore, for deadline scheduled tasks, there is no need to consider the current state of real-time or normal scheduled tasks. However, there is the risk of non-deadline tasks being pushed aside.

This concern can also be addressed using ‘cpusets’ and solid system architecture, by dividing the CPUs into different sets depending on the systems requirements. For an 8-core machine, that needs to run deadline, round-robin and normal tasks the cores could be divided as:

Core 1: Deadline tasks
Core 2: Deadline tasks
Core 3 and 4: RR tasks
Core 5 to 8: All other tasks

The deadline scheduler is heavily associated with the PREEMPT_RT patch set, which was a separate patch set but has since merged into the mainline Linux kernel. The aim of this patch set was to reduce the time that a lower priority task could block access to a core, while a higher priority task(s) waited in the wings. It works by reducing the time a task can run with IRQs and preemption disabled.

Overall this article has outlined Linux scheduling, specifically Linux deadline scheduling, some common pitfalls with it, and how it relates to the PREEMPT_RT patch set.

Real-Time Performance Optimisations for Multi-Axis Robotic Systems using copper-rs

2026-08-20T00:00:00+01:00

Real-time execution is essential for safety-critical robotics systems in autonomous vehicles, surgical robots, and industrial automation. The real-time performance of traditional middleware frameworks are often constrained by design limitations which are difficult to address with minimal modifications to application code.

Consequently, this article will explore the practicality of robotic frameworks outside of well-established ecosystems. Copper-rs is positioned as a direct competitor to ROS 2, offering OS-independent, real-time, deterministic scheduling. I discuss a case study on an open-loop, 18 degrees of freedom (DoF) hexapod robot, demonstrating copper-rs on a multi-axis robot that demands low latency in a high frequency message environment. I aim to provide empirical evidence justifying claims about determinism, observability, and runtime latency that make copper-rs uniquely suitable for real-time applications.

Real-time Considerations in Robotics

Operating System

For real-time applications, the fairness of scheduling principles in default Linux (CFS and Round-Robin) are unsuitable and liable to priority inversions. Furthermore, locking primitives such as spinlock_t disable preemption, driving sentiment towards two soft real-time platforms: Linux with a PREEMPT_RT patch and QNX.

Studies target QNX for multi-axis robotic systems which demand high bandwidth capability to drive numerous joints, though lack the empirical evidence necessary for widespread adoption. Conversely, the PREEMPT_RT patch in Linux uses forced-threaded interrupts, sleep spinlocks and real-time mutexes to construct a preemptive event-driven system kernel.This uncertainty in the current best solution leaves a lot to be desired.

copper-rs does not claim to be a replacement for Linux or RTOS. It exists within its own category, supporting both baremetal and GPOS targets. Accordingly, we consider copper-rs capacity to bypass OS constraints.

Middleware Frameworks

The supporting middleware may introduce latent behaviour. ROS 2 has maintained its position as the primary framework for robotic applications, offering modularity and distributed execution in a way that doesn't compromise on convenience.

Complex behaviour in ROS 2 can be described by a network of topics and nodes that emphasise flexibility and loose, composable design. As a result, the default model for inter-process communication using the Data Distribution Service (DDS) protocol is unfit for real-time performance, suffering from copying overheads, data serialisation, and asynchronicity which significantly reduce throughput.

Demand for time-critical solutions that guarantee performance and reliability have surged and copper-rs has emerged to bridge the real-time gap.

copper-rs: A Case Study into Rust-first Robotics

Where ROS 2 Struggles

In ROS 2, communication is facilitated by a Data Distribution Service (DDS) layer. Nodes are mapped to processes in the underlying OS and execute behaviour through callbacks on a single/multi-threaded executor. In asynchronous mode, the execution of the callback is freed by delegating the sending process to the DDS. This is the first fundamental flaw in the ROS 2 framework: callbacks are executed without strict time or sequence ordering. OS scheduling, thread preemption and system load all affect callback execution in a way that breaks determinism.

Figure 1 Illustration of ROS 2 system. Source: https://doi.org/10.1109/RTSS59052.2023.00026

The second fundamental issue in ROS 2 is inter-process data transport. In the DDS layer, DataWriters and DataReaders are registered for callbacks to publish and receive messages from a topic. Messages must be serialised (expensive) and transported via the underlying transportation protocol, by default set to UDPv4. Every step introduces runtime latency and despite community effort for enhancements like EtherCAT (the underlying network protocol) and Agnocast (zero-copy communication), latency and jitter remains an issue in DDS.

Figure 2 Interprocess communication over TCP/UDP. Source: https://doi.org/10.23919/ICCAS66577.2025.11301358

copper-rs

Beyond ROS 2, competing frameworks have redefined the robotics execution layer seeking proven real-time performance. The 3rd of July 2026 marked the first stable release of copper-rs: a Rust-first robotics runtime that is designed with determinism as a first principle.

The copper-rs runtime is categorically different to ROS 2, more closely resembling a time-triggered operating system than a middleware framework. In a typical workflow, tasks (nodes), payloads (messages), and bridges to middleware are statically described in a configuration file (copperconfig.ron) and checked at compile-time.

Based on user-defined configuration, copper-rs generates a task graph and custom deterministic scheduler to execute tasks in a pre-computed order. Two lists are generated which form the order of execution: CopperList and CuTasks.

Figure 3 copper-rs task lifecycle. Source: https://github.com/copper-project/copper-rs/wiki/Task-Lifecycle

You can read the rust docs about how copper-rs provides deterministic replay by serialising the internal state of tasks (freeze) here.

Figure 3 illustrates the task lifecycle and usage of the scheduler. All tasks in CuTasks are executed synchronously and communicate over shared buffers to reduce runtime latency. In the best effort section, high throughput jobs are executed to minimise the interference with the critical path. This is essential for visual perception, motion planning, and control to run at low-latency in the time-critical section. copper-rs' low latency and time-triggered scheduling principles make it ideal for real-time, multi-axis robotics systems. Consider the table below which provides an overview of optimisations to latency and jitter.

Consideration	copper-rs	ROS 2
Execution order	Synchronous task execution in a determined order	Asynchronous, callback-oriented architecture mean that execution order is uncertain
DDS overhead	Data is only serialised at keyframes for resimulation. Zero-copy between tasks	DDS suffers from copying overheads and expensive data serialisation that affect runtime latency
Data transport	Tasks write directly into a buffer for the next task. Memory is organised to optimise cache usage and prefetches	DDS relies on its underlying network protocol for data transport
Scheduling	Internal unified scheduler and filesystem: supports baremetal	Scheduling on GPOS platforms struggle to guarantee deterministic execution
Logging	Binary serialisation using bincode, pre-allocated buffers and no string construction overhead	Suffers from string formatting overhead. By default rosbag indexes an SQlite database affecting runtime latency
Memory	Zero allocation along the hotpath	Publishers and subscribers allocate memory on the heap when handling variably-sized messages

Modularity and Observability

Complex behaviour in robotics often relies on sensors, cameras, and safety monitors on the critical path. In high data frequency environments we observe the declining performance of ROS 2. Developers choose to generalise behaviour and reduce log capture points to achieve lower-latency in production at the cost of modularity and observability. The same scenario is significantly less likely in copper-rs because it is significantly less latent by design, and has 12x higher logging throughput than ROS 2.

Thread Pools and Real-time Scheduling

copper-rs exposes parallel-rt and rt-scheduling for std targets, offering multi-threaded execution and real-time scheduling without sacrificing deterministic behaviour. Each thread pool's processor affinity, scheduling policy and on_error behaviour can be configured.

Available scheduling policies are:

Fair - Fair time-sharing (SCHED_OTHER / CFS)
Nice - Fair scheduler with a niceness bias
Fifo - SCHED_FIFO hard real-time
RoundRobin - SCHED_RR, same as Fifo except threads at the same priority are time-sliced in turn

You can read more about parallel-rt and rt-scheduling in the copper-rs documentation.

copper-rs on a Hexapod Robot

This section presents the findings of a 4 week case-study using copper-rs to drive an open-loop, 18DoF hexapod robot. Time constraints meant compromises were made to some of copper-rs core design principles. I provide an overview of copper-rs performance on multi-axis robotic systems based on anecdotal experience, evaluating the runtime quantitatively in a high message frequency environment.

Specification

Servo2040 (RP2040)
Raspberry Pi 5 8GB
Zeee 2S Lipo Battery 5200mAh 7.4V 80C
18 TD-8130MG 30 KG Digital Servos
Leg model by Emre Kalem

Figure 4 Hexapod robot model

How It Works

On initialisation, a source task (clock.rs) triggers the main loop at the target frequency and serialises (freeze) the current timeframe for resim.
The payload emitted by the source task is read from the buffer by the Gait task (gait.rs), which stores the description of the hexapod robot (Spyder) defined by the custom motion planner (spyder_ik).
gait.rs calls the motion planner to produce joint angles in radians based on the current command and emits the ServoFrame payload.
Finally, the ServoMapper task maps the joint angles to pulse widths, encodes to bytes and emits a PWMFrame payload that can be sent over the serial to a Servo2040 (rp2040) board.

Figure 5 Hexapod runtime performance metrics

Figure 5 shows performance metrics pulled live from the consolemon console monitor. This run represents the baseline without optimisation. Observe that the maximum latency for end-to-end is 730.364μs and the end-to-end jitter is 362.720μs. Furthermore, some tasks have achieved sub-microsecond latency. Average payload size is 144 bytes ([f64; 18]) at 30Hz. Note that the results have no direct comparison and are otherwise inconclusive.

Copper-rs' capacity for real-time performance is worth further investigation. The framework is 'metamorphic' by design: for any given task graph copper-rs will generate a deterministic scheduler that unblocks time-critical tasks along the hot path.

Combine scheduling with direct improvements to latency and it becomes abundantly clear why copper-rs should be considered alongside competing middleware. To raise areas for investigation:

Directly improve performance of RP2040 by rewriting Micropython firmware in Rust
Reduce data size by using smaller data types than f64

In terms of real-time performance, the following are applicable given the project does not move to a baremetal approach:

Use copper-rs parallel-rt to isolate latency-critical work onto dedicated cores with real-time priority
Define scheduling policies for thread pools with rt-scheduling

Logging

Figure 6 copper-rs DAG for hexapod robot

copper-rs offers various tooling that effectively demystifies the DAG concept. Run (just dag) to visualise data transportation as a series of tasks, bridges, and payloads.

Figure 7 Debug logs generated for serial bridge

Figure 8 Raw data sources in Foxglove using copper-rs MCAP export

Figure 9 No colcon build or launch files: execute with cargo run.

Serial Bridge and Zenoh Controller

copper-rs supports a plethora of middleware through the Bridges API. A bridge lets copper-rs talk to an external transport (serial, CAN, DDS, etc.) and facilitates communication with Zenoh, iceoryx, ROS 2 and more. Users should consider the effect on latency and determinism when using bridges, especially for real-time applications. A Zenoh bridge is used for teleoperation, broadcasting the command that dictates movement direction to the Gait task.

Using the same bridge API, I define a serial bridge in (bridges.rs) responsible for reading and writing from the RP2040 resource.

Rust-centric Workflow

Copper-rs feels purposefully ergonomic: having compile-time checks makes iterative development easier and eliminates some of the gripes I had debugging ROS 2. Additionally, copper-rs utilises the speed of Rust to offer significant reductions to latency over other robotics frameworks.

That being said, the adoption of Rust in the robotics ecosystem will be a long and arduous process. Comparatively, robotics has far fewer software engineers than any typical technological discipline. Familiarity with Python and C++ is widespread, so despite its drawbacks, ROS 2 wins in adoptability. The rigidity of copper-rs design principles mean it feels great in the final stages, but might take significant investment throughout the development cycle.

Hexapod Motion Planner

This project uses a custom, geometry-based motion planner that exposes a gait generator for hexapod robots with the following core features:

Home position, link lengths, angle resolution and fields that mathematically describe the robot are specified in the SpyderBuilder.
The user calls build to construct a Spyder, exposing standing position, LUT generation and export functionality.
Inverse kinematics, change of basis and LUT generation are handled for the end-user to simplify usage.

In the future, I plan to port from LUT generation to a live IK model and integrate sensor data. If you want to contribute, I would appreciate it!

Going forward, I hope to create a hexapod runtime that works straight out of the box, for aspiring roboticists who are uninterested in the complexities of robotics frameworks. copper-rs' Rust oriented approach is surprisingly well suited — simply define the hexapod parameters in copperconfig.ron and run it with cargo run. Logging naturally follows, and with more time I would like to have simulation (Bevy) integrated into the project.

Figure 10 Rviz simulation of hexapod robot

Plans for Future Development

Go baremetal with copper-rs

If the entire copper-rs runtime can be run on a RP2350 why use the Pi 5? This is an open-loop hexapod robot that has no object awareness or visual perception so compute is no issue. This could significantly reduce runtime latency and jitter: no serialisation overhead for USB CDC and full control over resources and scheduling.

Go 100% Rust with Zenoh controller (telemetry) and RP2040 firmware

The RP2040 firmware and controller could be written in Rust to remove Python dependency in this project. Micropython suffers from high memory usage and poor real-time performance which throttles the performance of the system.

Add a determinism regression test

Inclusion that guarantees the binary produced on resimulation is bit-for-bit identical which is a great addition to CI.

Conclusion

In this article, I present a case study on an open-loop, 18 degrees of freedom (DoF) hexapod robot, demonstrating copper-rs on a multi-axis robot that demands low latency in a high frequency message environment. I discuss claims about determinism, observability, and runtime latency, providing an initial framework for quantitative analysis of copper-rs.

Additionally, this article addresses key architectural limitations in the default ROS 2 framework, in particular:

The cost of the Data Distribution Service (DDS)
Asynchronous communication and the effect of callbacks on execution order
The impact of the underlying OS

I make direct comparisons between ROS 2 and copper-rs, evaluating each in the context of real-time robotic systems. The results of the case study exceeded expectations, achieving a baseline maximum end-to-end latency of 730.364μs on native Linux. Furthermore, subsequent improvements to application code continue to reduce the maximum latency and jitter. I provide a comprehensive overview of scheduling, memory-allocation, and key design principles which contribute to the remarkable performance of copper-rs.

The source code is available on CodethinkLabs: feel free to contribute!

Source code: https://gitlab.com/CodethinkLabs/spyder_robot

References

A. N. Nagiwale, P. K. Uttam and S. E. Talole, "A Real-Time Software Framework for Multi-axis Robotic Systems Using ROS 2 and EtherCAT," 2024 IEEE Pune Section International Conference (PuneCon), Pune, India, 2024, https://doi.org/10.1109/PuneCon63413.2024.10895565

AutomationDirect.com. “What Is EtherCAT? From AutomationDirect.” YouTube, February 16, 2024. https://www.youtube.com/watch?v=rt-ykjEL4Y8

J. -H. Kim, J. -H. Choi, Y. -C. An and T. -Y. Kuc, "Evaluation of RTOS for Robotic Applications with ROS2 on Embedded Systems," 2025 25th International Conference on Control, Automation and Systems (ICCAS), Incheon, Korea, Republic of, 2025. https://doi.org/10.23919/ICCAS66577.2025.11301358

ROS2 Real-time Performance Optimization and Evaluation. https://link.springer.com/article/10.1186/s10033-023-00976-5

Ye, Y., Nie, Z., Liu, X. et al. ROS2 Real-time Performance Optimization and Evaluation. Chin. J. Mech. Eng. 36, 144 (2023). https://doi.org/10.1186/s10033-023-00976-5

D. Casini, J. Chen, J. Li, F. Reghenzani, H. Teper. "A Survey of Real-Time Support Analysis and Advancements in ROS 2". arXiv:2601.10722. https://doi.org/10.48550/arXiv.2601.10722

X. Luo, X. Jiang, N. Guan, H. Liang, S. Liu and W. Yi, "Modeling and Analysis of Inter-Process Communication Delay in ROS 2," 2023 IEEE Real-Time Systems Symposium (RTSS), Taipei, Taiwan, 2023, https://doi.org/10.1109/RTSS59052.2023.00026

Zhang Z, He W, Wu F, Quesada L and Xiang L (2024) Development of a bionic hexapod robot with adaptive gait and clearance for enhanced agricultural field scouting. Front. Robot. https://doi.org/10.3389/frobt.2024.1426269

copper-rs. “Copper Tasks Lifecycle Overview.” Copper Tasks lifecycle overview - Copper Wiki. https://copper-project.github.io/copper-rs/Task-Lifecycle/

copper-rs. "Copper Robotics" Copper Robotics website. https://www.copper-robotics.com/

ROS 2. "ROS 2 Platform Support". https://www.ros.org/reps/rep-2000.html

“Bridging Robotics and Systems Programming: Why Copper-Rs Is a Game Changer.” 2026. Guillaume BINET 2026. https://fosdem.org/2026/schedule/event/SK8EGJ-copper-rust-robotics-runtime/

Copper Robotics. "What are we solving for with copper-rs?" Youtube, March 9, 2026. https://youtu.be/IuduPa_nyks?si=I7dVYDD7wVOc4FqY

Tokyo Rust. "Copper: The OS for Physical AI" Youtube, July 15, 2026. https://youtu.be/XxaSLWtGHxM?si=sttKzCPy9MlbVc6J

ROS 2 Agnocast: Supporting Unsized Message Types for True Zero-Copy Publish/Subscribe IPC. https://arxiv.org/pdf/2506.16882v1

Instantaneous CI: Reducing Development Lag Through Cicada with Local-ﬁrst Design and Caching

2026-07-16T00:00:00+01:00

Understanding Continuous Integration

Continuous integration and continuous deployment (CI/CD) enable the automation of building, testing, and releasing software. This approach streamlines collaboration between development and operations teams. CI/CD reduces large points of failure by providing immediate feedback on errors or bugs in an update cycle that can be made more efficient and reliable through being smaller and more frequent. Since the 1990s, CI has become a fundamental practice for devops, with CD following suit in the 2010s. At Codethink, we have a storied history of integrating CI/CD into our software development cycle, being early and prominent advocates of its use in the automotive industry. Our latest approach uses a self-hosted GitLab namespace, maintaining our open source ethos and enabling scalability.

Available Tooling for CI/CD

Tooling including Zuul, Jenkins, and GitLab CI offer framework support in the CI/CD process but each have benefits and drawbacks, which must be carefully considered when used for developing software. There is no perfect CI tool out there that will solve the problem for every use case. Often, modifications are required in the tooling to tailor and adapt the process for your project. This can result in unmanageable maintenance costs in the development cycle, which creates frustration, delays, and unwieldy dependency management.

GitLab CI makes local debugging difficult and does not offer a way to test jobs pre-merging. GitLab attempts to be an all-in-one tool and the successive feature creep increasingly makes it harder to manage.

Jenkins CI runs on a system of plugins, which too often are abandonware, increasing security risk and requiring a lot of overhead and dependency mitigation if an upgrade is required. The maintenance and hosting cost can become unreasonable when scaling a project.

Zuul CI still has a smaller uptake. While it is powerful and open source, which provides great project management flexibility, there are limited resources and documentations for its integration into a development workflow at this stage.

REAPI and CI

Remote Execution Application Programming Interface (REAPI) is an open and standardised gRPC-based protocol that enables build tools to offload actions onto remote servers for more efficient execution. REAPI operates on a client-server model where tasks are serialised by the build tool before being executed remotely, returning the output. Caching is added to this process in order to avoid delays from the duplication of task runs.

It has been used as a component part of CI/CD to make reproducible builds faster at scale with tools like Buildstream and Bazel. However, these benefits could be applied further to CI by incorporating REAPI in the CI system itself, rather than just within individual jobs.

By combining REAPI with integration tooling, remote caching could be shared and would produce a near instantaneous result on a job output after it has been run once if the input is exactly the same.

This would improve the efficiency of the integration process.

A common pain point with many existing CI systems is the inability to run jobs outside of the CI workflow. If the CI system is an REAPI client, we can develop a way to submit the same jobs to the REAPI server from outside of the CI itself. This local-first design improves the feedback loop when developing new CI jobs, as well as giving full confidence that a CI job will pass after pushing.

Using REAPI caching within a CI job allows reuse of old results of the same job for a given set of inputs. However, by using REAPI caching as part of the CI system itself we can also cache the job's environment construction, saving time spent fetching the git repository and dependencies.

This combination provides the opportunity for instantaneous CI.

The performance of our CI system's cache could be enhanced with speculative execution of CI jobs across multiple MRs, essentially pre-testing the potential merges to provide instantaneous CI when the target branch of an MR is updated.

Developing Cicada

Advancing from the theoretical application and implications of this variety of CI tooling, Cicada was built as a prototype.

Cicada jobs and pipelines are declared in YAML, then modeled as REAPI Actions to provide caching and offload job scheduling to REAPI server tooling. By using remote execution, the cache can be safely shared between locally invoked runs of Cicada jobs and jobs invoked during a CI run, making the actual CI run provide an almost instantaneous result.

You can read more about the development of Cicada on this blog post. There are real-world applications of this tool, notes on how this could work with Rust, and a list of ideas for future development.

Assessing the reproducibility workflow in freedesktop-sdk

2026-06-16T00:00:00+01:00

Introduction

Freedesktop SDK is a free, community-developed, and open-source project that simplifies the process of creating various software artefacts. Common use cases include building containers, Flatpak runtimes, snaps, and complete operating systems.

In this article, we explore a more lightweight approach to assessing reproducibility that prioritises speed, whilst providing meaningful validation to the engineer at the artefact level. Additionally, we provide metrics to help quantify the performance gap and discuss the tradeoffs between the two approaches. Note that this article focuses on Freedesktop SDK as an example repository to analyse the reproducibility workflow inside it and explores an alternative workflow strategy that aligns more closely with how upstream Linux distributions handle reproducibility.

Summarising reproducibility

Reproducibility in the context of Software Engineering can refer to different definitions depending on the environment and requirements. In this article, we consider bit-for-bit reproducibility, i.e., the ability to generate an identical binary given the same source code input and build environment.

The concept of reproducibility as a principle applies broadly to scientific and engineering disciplines, where the results from an experiment should be achievable with a high degree of reliability when the study is replicated. The Wikipedia article on reproducible builds provides a broad overview of the concept and its historical context within Software Engineering. It defines reproducible builds as:

"Reproducible builds, also known as deterministic compilation, is a process of building software which ensures the resulting binary code can be reproduced. Source code compiled deterministically will always output the same binary."

In theory, this sounds both useful and doable. In reality, achieving reproducibility is harder than it sounds. Build processes can be influenced by a range of factors, such as timestamps, environment variables, filesystem ordering, and differences within the toolchain behaviour. Unless you control these factors, you risk introducing stochastic variations in the final artefact, which decreases the confidence in the system as a whole.

The reproducible-builds project is one well-known community effort focused on improving reproducible builds. They have regular talks and events that take place each year. In addition, they publish monthly reports.

The official definition on their website is:

“Reproducible builds are a set of software development practices that create an independently-verifiable path from source to binary code.”

In short, reproducibility is widely regarded as an important topic within Software Engineering, particularly in environments that prioritise trustworthiness and long-term maintainability.

BuildStream plays a crucial role in reducing this problem because it enables you to declare software stacks in a declarative format and control the build environment, which helps reduce sources of non-determinism and makes reproducibility achievable in practice.

At Codethink, reproducibility is one of the key concepts integrated into our Eclipse Trustable Software Framework.

The current solution inside of freedesktop-sdk

The freedesktop-sdk repository (at this moment in time) uses the buildstream-reprotest library to establish the reproducibility of the BuildStream elements that reside inside it.

The buildstream-reprotest is a Python script that essentially uses the subprocess module in Python to invoke BuildStream and orchestrate a method for assessing the reproducibility status of a given BuildStream element. The script evaluates the entire dependency graph and essentially asks the question: Is the entire dependency graph reproducible at every level?

The script runs the following phases:

Phase 1: Build the target element and pull the remote cache where possible, and fall back to manually compiling if necessary.
Phase 2: Iterate through the dependency graph in reverse order, disable the remote cache and build each element manually from scratch.
Phase 3: Use the output from Phases 1 and 2 and execute diffoscope to identify any differences in the output.

Applying this methodology, by definition, assesses the reproducibility of the build process. Our argument is that, although there is nothing incorrect about this approach, it is time-consuming due to its very nature.

This approach introduces the following key challenges:

High computational cost: rebuilding all dependencies from scratch.
Time inefficiency: running comparisons at every layer of the graph.
Limited iteration speed: not ideal for rapid development feedback loops.

This raises an important question for us all:

Do we always need full dependency graph validation to get useful reproducibility signals?

A target level approach

To address this, we developed a Bash script that focused on target-level reproducibility rather than full graph validation. The reason for choosing Bash was primarily that it requires minimal boilerplate, is highly interactive, and has low resource usage.

Instead of asking:

Is the entire dependency graph reproducible?

We are now asking:

If we rebuild this element in isolation, do we get the same artefact?

The distinction between the two questions enables a faster feedback cycle, while also addressing many real-world reproducibility issues.

Workflow overview

The buildstream-reprotest-script executes the following phases:

Phase 1:
- Build the target element with all dependencies, using remote cache where possible.
- Fetch sources to ensure we can build without the cache/internet.
- Check out the resulting artefact into a temporary directory.
- Remove the target element from the local cache whilst keeping the dependencies.
Phase 2:
- Disable cache via a temporary configuration file.
- Rebuild the target element without rebuilding its dependencies.
- Check out the new artefact.
Phase 3:
- Run diffoscope between the two outputs.
- Generate a detailed HTML report.
- Return the comparison exit status.

The key design decision here is to reuse the previously built dependencies from Phase One into Phase Two. The benefit of reusing previously built dependencies means that we significantly reduce the time taken to assess the reproducibility of a project because we’re not devoting time to rebuilding each dependency from scratch. The time reduction is particularly useful when processing large dependency graphs, e.g., those within the freedesktop-sdk environment.

We should note that this workflow aligns closely with how upstream Linux distributions approach reproducibility workflows.

The pros and cons of buildstream-reprotest-script

Pros:

Ability to assess the reproducibility of a BuildStream element.
Faster than the buildstream-reprotest equivalent.
Helps expose situations where the target element is reproducible, but it’s a dependency that’s actually at fault.

Cons:

Does not assess the reproducibility of the dependencies and thus cannot assess the reproducibility of the entire Directed Acyclic Graph (DAG), which represents the build dependencies.
Does not detect reproducibility issues introduced by dependencies that are reused from cache rather than rebuilt.

Metrics

Element	No. of dependencies	Time for execution (s)	Time for diffoscope (s)	Run type
components/podman.bst	291	4716	372	Cold
components/podman.bst	291	472	365	Hot
vm/desktop/filesystem.bst	726	4669	118	Cold
vm/desktop/filesystem.bst	726	203	128	Hot
abi/sdk-lib-headers.bst	615	3205	10	Cold
abi/sdk-lib-headers.bst	615	42	11	Hot
extensions/mesa/mesa.bst	220	3801	44	Cold
extensions/mesa/mesa.bst	220	369	46	Hot
components/nasm.bst	154	2369	24	Cold
components/nasm.bst	154	62	22	Hot
oci/layers/flatpak-stack.bst	618	5630	288	Cold
oci/layers/flatpak-stack.bst	618	408	314	Hot

_{The run type is split into Cold and Hot, which simply refer to running the script without and with local cache,
respectively. A Cold run has no local cache, and a Hot run has local cache.}

The above table shows us a series of elements that we tested the buildstream-reprotest-script against. Whilst we did not execute the original buildstream-reprotest workflow for direct comparison, prior experience indicates that full dependency graph reproducibility testing can take between 5 and 15 hours, depending on the graph size.

The original Merge Request contains a detailed series of tests against a specific instance of freedesktop-sdk, which includes two interesting cases:

components/nasm.bst
oci/toolbox-oci.bst

The buildstream-reprotest-script helped us deduce that the non-reproducibility of [1] is due to a dependency within the pipeline. In addition, the script helped expose an issue with the cache keys associated with [2].

In summary, the metrics show us that our script is significantly faster and raises an interesting follow-up question:

Should the diffoscope step be optional in fast validation workflows?

Practical Impact

In practice, this approach:

Reduces testing time significantly.
Enables faster debugging cycles.
Provides actionable feedback early.
Helps isolate whether issues originate in the target or its dependencies.

Conclusion

Reproducibility testing doesn’t always need to be exhaustive to be useful, and in practice, we can alter the mechanism to enable engineers to detect non-determinism earlier in the development cycle, rather than discovering it during expensive full-graph validation.

By focusing on target-level validation, we can improve:

Speed.
Practicality.
Engineering insight.

The critical question for us all (at least within the Freedesktop SDK environment and ones like it):

Can target-level validation complement the existing full dependency graph validation within Freedesktop SDK (and similar environments)?

Future work

Potential improvements for our buildstream-reprotest-script include:

Making logging depth dynamic, i.e., removing our brittle assumption and reliance on caller 1.
Providing an option to perform full dependency graph reproducibility testing.
Making the diffoscope operation optional.
Capturing metadata for deeper diagnostics.

Deep Dive into Upstream RISC-V Boot Chain

2026-06-09T00:00:00+01:00

This deep dive into the upstream RISC-V boot chain was presented as a poster at the RISC-V Summit Europe 2026 in Bologna, Italy.

While porting Freedesktop SDK to the EBC7700 for my FOSDEM'26 talk, I encountered some UEFI boot issues, which motivated me to dig deeper and uncover all the mysteries about the RISC-V boot chain. We start by introducing a few important concepts/tools.

RISC-V Toolchain

First, you may need a toolchain to actually be able to compile any of the binaries involved in the boot chain. And if you are, just like me, still waiting for your long pre-ordered RISC-V laptop/workstation, and therefore still using an x86_64 machine as your daily driver, the easiest is to download a pre-built cross-toolchain. Throughout this tutorial, I used the nightly 2026.05.19 releases:

riscv64-glibc-ubuntu-24.04-gcc.tar.xz
riscv64-glibc-ubuntu-24.04-llvm.tar.xz

The setup for gcc:

⬢ [zim@toolbx ~]$ cat envsetup-gcc-riscv64-16.1.sh
export ARCH=riscv
export CROSS_COMPILE=~/Toolchains/riscv64-glibc-ubuntu-24.04-gcc-2026.05.19-nightly/bin/riscv64-unknown-linux-gnu-
export DTC_FLAGS="-@"
export LANG=en_US.UTF-8

Or alternatively for llvm, which is my preferred option nowadays:

⬢ [zim@toolbx ~]$ cat envsetup-llvm-riscv64-21.1.1.sh
export ARCH=riscv
export CC=clang
export CROSS_COMPILE=riscv64-unknown-linux-gnu-
export DTC_FLAGS="-@"
export LANG=en_US.UTF-8
export LLVM=1
export PATH=~/Toolchains/riscv64-glibc-ubuntu-24.04-llvm-2026.05.19-nightly/bin/:$PATH

Note: The -@ flag instructs the device tree compiler to generate symbol tables, which are mainly used should you ever play with device tree overlays.

Privilege Levels

Second, I want to briefly mention an important low-level RISC-V hardware concept called privilege levels, which define the processor's access rights to memory, I/O devices, and instructions. Understanding the basic concept of privilege levels is important for the overall understanding of this article. RISC-V knows 3 privilege levels:

M (machine) mode:
- highest privilege
- low-level firmware (e.g. OpenSBI), system boot, and security (e.g. secure execution environments)
- handles critical system traps
- return to lower-privileged levels using mret instruction
S (supervisor) mode:
- intermediate privilege
- used by operating system kernels to manage memory and hardware resources
- restrict direct access to machine-level configuration registers and physical memory protection (PMP)
- uses Memory Management Unit (MMU) and virtual memory
- ecall instruction to request kernel services
- return to U-mode using sret instruction
U (user) mode:
- lowest privilege
- cannot execute privileged instructions, access physical memory directly without virtual memory translations, or directly alter core control registers
- user space executing standard applications
- prevents incorrect or malicious application code from interfering with other running programs or the operating system itself

QEMU

Third, I want to introduce the Quick Emulator (QEMU), which is a free and open source machine emulator and virtualiser. The QEMU RISC-V emulation with its (riscv64) virt machine is the perfect playground for us to gain a deeper understanding of the RISC-V boot flow. It uses a hard-coded reset vector, and by default, loads the built-in OpenSBI binary into system memory space, unless a custom firmware binary is specified using the -bios argument. OpenSBI executes hardware-specific initialisation (in M-mode) and passes control (in S-mode) to the -kernel or bootloader payload at 0x80000000. The virt machine automatically generates a device tree blob (DTB) which it passes to the guest, if there is no -dtb option specified.

For QEMU I relied on the latest version shipped with my distro, rather than also compiling it from scratch:

⬢ [zim@toolbx ~]$ rpm -q qemu-system-riscv
qemu-system-riscv-10.2.2-1.fc44.x86_64

Boot Flow Overview

With those three things introduced, let us now look at the RISC-V boot flow:

Boot ROM aka ZSBL (Zero Stage Bootloader)

The ZSBL is what a RISC-V chip, or its emulated derivative, runs first at power-up out of reset. It initialises a few registers and jumps directly to some hard-coded address (in case of QEMU's riscv64 virt machine, that is 0x80000000), where the first user-provided code runs (by default, QEMU will load OpenSBI there).

Debugging

Let us use QEMU's inbuilt gdb server (debugger) to have a look at what exactly ZSBL does in the QEMU case.

In a first terminal, we launch QEMU with the -S argument, which instructs it to just wait for a gdb debugger session to connect to its gdb server and further specify what to do next:

⬢ [zim@toolbx ~]$ qemu-system-riscv64 -S -gdb tcp::1234 -machine virt -nographic

And in a second terminal, we launch gdb-multiarch (again, just using the distro-provided one), instruct it to disassemble the next line and connect to the above-launched QEMU. Note that it will automatically detect the riscv:rv64 architecture. We can use si a few times to step through ZSBL's instructions one by one, until we observe the final jump to OpenSBI at 0x80000000:

⬢ [zim@toolbx ~]$ gdb -ex 'set disassemble-next-line on' -ex 'target remote localhost:1234'
GNU gdb (Fedora Linux) 17.1-6.fc44
Copyright (C) 2025 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Remote debugging using localhost:1234
⚠️ warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0x0000000000001000 in ?? ()
=> 0x0000000000001000:  00000297            auipc   t0,0x0
(gdb) si
0x0000000000001004 in ?? ()
=> 0x0000000000001004:  02828613            addi    a2,t0,40
(gdb) si
0x0000000000001008 in ?? ()
=> 0x0000000000001008:  f1402573            csrr    a0,mhartid
(gdb) si
0x000000000000100c in ?? ()
=> 0x000000000000100c:  0202b583            ld  a1,32(t0)
(gdb) si
0x0000000000001010 in ?? ()
=> 0x0000000000001010:  0182b283            ld  t0,24(t0)
(gdb) si
0x0000000000001014 in ?? ()
=> 0x0000000000001014:  00028067            jr  t0
(gdb) si
0x0000000080000000 in ?? ()
=> 0x0000000080000000:  00050433            add s0,a0,zero
(gdb) exit
A debugging session is active.

    Inferior 1 [process 1] will be detached.

Quit anyway? (y or n) y
Detaching from pid process 1
Ending remote debugging.
[Inferior 1 (process 1) detached]

So QEMU's ZSBL execution starts at address 0x1000, and there are only 6 instructions executed, but what exactly are they doing? Turns out in the case of QEMU's virt machine, this is the riscv_setup_rom_reset_vec() function in hw/riscv/boot.c which is populated as part of its own bring up before booting the guest:

void riscv_setup_rom_reset_vec(MachineState *machine, RISCVHartArrayState *harts,
                               hwaddr start_addr,
                               hwaddr rom_base, hwaddr rom_size,
                               uint64_t kernel_entry,
                               uint64_t fdt_load_addr)
{
    int i;
    uint32_t start_addr_hi32 = 0x00000000;
    uint32_t fdt_load_addr_hi32 = 0x00000000;

    if (!riscv_is_32bit(harts)) {
        start_addr_hi32 = start_addr >> 32;
        fdt_load_addr_hi32 = fdt_load_addr >> 32;
    }
    /* reset vector */
    uint32_t reset_vec[10] = {
        0x00000297,                  /* 1:  auipc  t0, %pcrel_hi(fw_dyn) */
        0x02828613,                  /*     addi   a2, t0, %pcrel_lo(1b) */
        0xf1402573,                  /*     csrr   a0, mhartid  */
        0,
        0,
        0x00028067,                  /*     jr     t0 */
        start_addr,                  /* start: .dword */
        start_addr_hi32,
        fdt_load_addr,               /* fdt_laddr: .dword */
        fdt_load_addr_hi32,
                                     /* fw_dyn: */
    };
    if (riscv_is_32bit(harts)) {
        reset_vec[3] = 0x0202a583;   /*     lw     a1, 32(t0) */
        reset_vec[4] = 0x0182a283;   /*     lw     t0, 24(t0) */
    } else {
        reset_vec[3] = 0x0202b583;   /*     ld     a1, 32(t0) */
        reset_vec[4] = 0x0182b283;   /*     ld     t0, 24(t0) */
    }
...

So the 6 instructions are literally hard-coded right there in QEMU's source code!

QEMU populates a struct fw_dynamic_info with the information required to boot the next stage and passes its address in the a2 register of the RISC-V CPU. The address must be aligned to 8 bytes on RV64 and 4 bytes on RV32. The struct fw_dynamic_info is defined in include/hw/riscv/boot_opensbi.h:

/** Representation dynamic info passed by previous booting stage */
struct fw_dynamic_info64 {
    /** Info magic */
    int64_t magic;
    /** Info version */
    int64_t version;
    /** Next booting stage address */
    int64_t next_addr;
    /** Next booting stage mode */
    int64_t next_mode;
    /** Options for OpenSBI library */
    int64_t options;
...

The next_mode is set to S to select a CPU that supports S-mode.

QEMU also passes a unique per-hart ID in the a0 register. We skipped the discussion of harts here as it is not relevant for the discussion of the boot chain.

The fdt_load_addr, pointing to the flattened device tree, is passed via the a1 register.

Then it jumps to address 0x80000000 to execute the next element within the boot chain.

OpenSBI

SBI stands for Supervisor Binary Interface, which is the system call style calling convention between a Supervisor (e.g. S-mode OS) and Supervisor Execution Environment (SEE, e.g. the M-mode runtime firmware). SBI calls help to reduce duplicate platform code across OSes, provide common drivers for an OS which can be shared by multiple platforms and an interface for direct access to hardware resources (M-mode only resources). The specification is maintained by the Platform Runtime Services (PRS) Task Group (TG), which ratified the SBI specification v1.0 in 2022.

OpenSBI (Open Source Supervisor Binary Interface) is an open source implementation of that RISC-V SBI specification.

In OpenSBI, the FW_DYNAMIC approach dynamically obtains the runtime configuration and information about the next boot stage (e.g. ⁠U-Boot proper) from any previous boot stage via passing a pointer to a struct fw_dynamic_info structure in a CPU register, rather than using hard-coded compile-time values.

The assembly bootstrapping phase does early reset and sanity checks by validating its environment and setting up basic mtvec trap vectors. Subsequently, the dynamic information from fw_dynamic_info gets parsed, stack allocated, scratch space initialised (mscratch), and the C-runtime jumped into.

The core initialisation and platform setup phase (sbi_init) does early console setup, configuring the primitive serial ⁠UART block for debug log output. The platform hardware init (platform_init) invokes platform-specific functions, initialising critical hardware blocks. The physical memory protection (PMP) registers are provisioned, shielding M-mode memory from unauthorised S-mode accesses. The interrupt controllers, like core timers, inter-processor interrupts (IPIs), and peripheral controllers like the PLIC/APLIC get initialised.

The payload transition phase (sbi_expected_trap to S-mode) extracts the jump target from struct fw_dynamic_info, modifies the mstatus/misa registers, configuring the previous privilege level to S-mode, programs the target next_addr value directly into the machine exception program counter (mepc), restores the rest of the original dynamic information (e.g. FDT address) passed via CPU registers, and executes the mret instruction, which drops the processor privilege down to S-mode and jumps directly to the address staged inside mepc, passing system execution to U-Boot proper.

Rather than using QEMU's built-in OpenSBI, we can build our own for later debugging:

⬢ [zim@toolbx ~]$ git clone https://github.com/riscv/opensbi.git
⬢ [zim@toolbx ~]$ cd opensbi
⬢ [zim@toolbx opensbi]$ git checkout -b v1.8.1-test v1.8.1
⬢ [zim@toolbx opensbi]$ make BUILD_INFO=y PLATFORM=generic
...
 AR        platform/generic/lib/libplatsbi.a
 AS        platform/generic/firmware/payloads/test_head.o
 CC        platform/generic/firmware/payloads/test_main.o
 MERGE     platform/generic/firmware/payloads/test.o
 ELF       platform/generic/firmware/payloads/test.elf
 OBJCOPY   platform/generic/firmware/payloads/test.bin
 AS        platform/generic/firmware/fw_dynamic.o
 ELF       platform/generic/firmware/fw_dynamic.elf
 OBJCOPY   platform/generic/firmware/fw_dynamic.bin
 AS        platform/generic/firmware/fw_jump.o
 ELF       platform/generic/firmware/fw_jump.elf
 OBJCOPY   platform/generic/firmware/fw_jump.bin
 AS        platform/generic/firmware/fw_payload.o
 ELF       platform/generic/firmware/fw_payload.elf
 OBJCOPY   platform/generic/firmware/fw_payload.bin

Note: The latest master branch did not build for me with llvm, which is why I picked released v1.8.1 instead, which seems to work fine.

The resulting binary may be launched by QEMU (remember, without the -bios argument, it would just launch the built-in/default OpenSBI):

⬢ [zim@toolbx opensbi]$ qemu-system-riscv64 -machine virt -nographic \
-bios build/platform/generic/firmware/fw_payload.bin

OpenSBI v1.8.1-87-g547a5bbda7c3
Build time: 2026-05-29 13:51:26 +0200
Build compiler: gcc version 16.1.0 (g6afcc4f6d)
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name               : riscv-virtio,qemu
Platform Features           : medeleg
Platform HART Count         : 1
Platform HART Protection    : pmp
Platform IPI Device         : aclint-mswi
Platform Timer Device       : aclint-mtimer @ 10000000Hz
Platform Console Device     : uart8250
Platform HSM Device         : ---
Platform PMU Device         : ---
Platform Reboot Device      : syscon-reboot
Platform Shutdown Device    : syscon-poweroff
Platform Suspend Device     : ---
Platform CPPC Device        : ---
Firmware Base               : 0x80000000
Firmware Size               : 321 KB
Firmware RW Offset          : 0x40000
Firmware RW Size            : 65 KB
Firmware Heap Offset        : 0x47000
Firmware Heap Size          : 37 KB (total), 0 KB (reserved), 14 KB (used), 22 KB (free)
Firmware Scratch Size       : 4096 B (total), 1600 B (used), 2496 B (free)
Runtime SBI Version         : 3.0
Standard SBI Extensions     : time,rfnc,ipi,base,hsm,srst,pmu,dbcn,fwft,legacy,dbtr,sse
Experimental SBI Extensions : none

Domain0 Name                : root
Domain0 Boot HART           : 0
Domain0 HARTs               : 0x0*
Domain0 Region00            : 0x0000000080040000-0x000000008005ffff M: (F,R,W) S/U: ()
Domain0 Region01            : 0x0000000080000000-0x000000008003ffff M: (F,R,X) S/U: ()
Domain0 Region02            : 0x0000000000100000-0x0000000000100fff M: (I,R,W) S/U: (R,W)
Domain0 Region03            : 0x0000000010000000-0x0000000010000fff M: (I,R,W) S/U: (R,W)
Domain0 Region04            : 0x0000000002000000-0x000000000200ffff M: (I,R,W) S/U: ()
Domain0 Region05            : 0x000000000c400000-0x000000000c5fffff M: (I,R,W) S/U: (R,W)
Domain0 Region06            : 0x000000000c000000-0x000000000c3fffff M: (I,R,W) S/U: (R,W)
Domain0 Region07            : 0x0000000000000000-0xffffffffffffffff M: () S/U: (R,W,X)
Domain0 Next Address        : 0x0000000080200000
Domain0 Next Arg1           : 0x0000000082200000
Domain0 Next Mode           : S-mode
Domain0 SysReset            : yes
Domain0 SysSuspend          : yes

Boot HART ID                : 0
Boot HART Domain            : root
Boot HART Priv Version      : v1.12
Boot HART Base ISA          : rv64imafdch
Boot HART ISA Extensions    : sstc,zicntr,zihpm,zicboz,zicbom,sdtrig,svadu,f,d
Boot HART PMP Count         : 16
Boot HART PMP Granularity   : 2 bits
Boot HART PMP Address Bits  : 54
Boot HART MHPM Info         : 16 (0x0007fff8)
Boot HART Debug Triggers    : 2 triggers
Boot HART MIDELEG           : 0x0000000000001666
Boot HART MEDELEG           : 0x0000000000f4b509

Test payload running

Debugging

We can launch QEMU in the first terminal, but this time, in addition to instructing it with the -bios argument to load our self-built OpenSBI, again with the -S argument, which instructs it to just wait for the gdb debugger to connect to it and further specify what to do next:

⬢ [zim@toolbx opensbi]$ qemu-system-riscv64 -S -bios build/platform/generic/firmware/fw_payload.bin \
-gdb tcp::1234 -machine virt -nographic

In the second terminal, we start gdb, giving it the OpenSBI symbol file and instructing it to break at sbi_boot_print_banner. Then we can let it print the banner line by line:

⬢ [zim@toolbx opensbi]$ gdb -ex 'add-symbol-file build/platform/generic/firmware/fw_payload.elf \
0x80000000' -ex 'target remote localhost:1234'
GNU gdb (Fedora Linux) 17.1-6.fc44
Copyright (C) 2025 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
add symbol table from file "build/platform/generic/firmware/fw_payload.elf" at
    .text_addr = 0x80000000
(y or n) y
Reading symbols from build/platform/generic/firmware/fw_payload.elf...
Remote debugging using localhost:1234
⚠️ warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0x0000000000001000 in ?? ()
(gdb) b sbi_boot_print_banner
Breakpoint 1 at 0x8000b14c: file /var/home/zim/Sources/opensbi/lib/sbi/sbi_init.c, line 51.
(gdb) c
Continuing.

Breakpoint 1, sbi_boot_print_banner (scratch=0x80046000) at /var/home/zim/Sources/opensbi/lib/sbi/sbi_init.c:51
51      if (scratch->options & SBI_SCRATCH_NO_BOOT_PRINTS)
(gdb) n
55      sbi_printf("\nOpenSBI %s\n", OPENSBI_VERSION_GIT);
(gdb) n
62      sbi_printf("Build time: %s\n", OPENSBI_BUILD_TIME_STAMP);
(gdb) n
66      sbi_printf("Build compiler: %s\n", OPENSBI_BUILD_COMPILER_VERSION);
(gdb) n
69      sbi_printf(BANNER);
(gdb) c
Continuing.
[Inferior 1 (process 1) exited normally]
(gdb) exit

U-Boot

Das U-Boot (Universal Boot Loader) is the de facto open source bootloader used in embedded devices. Its main job is to initialise hardware, load the operating system (usually Linux) into memory, and boot. It supports multi-stage booting, often split into an SPL (Secondary Program Loader) that sets up basic RAM, and full U-Boot proper that takes over to load the next stage. It features an interactive command line interface (CLI) usually accessible via serial port UART used to interrupt the boot sequence, poke/test hardware, and configure various aspects of the system boot. As part of its hardware management, it allows loading files from file systems on various (boot) devices like NAND/NOR flashes, (e)MMC/SD cards/chips, NVMe drives, SD cards, UFS, or USB mass storage. But it also features a built-in network stack for downloading images over Ethernet using e.g. TFTP, which is very convenient for debugging/development. Last but not least, it does the OS hand-off, including preparing a device tree and loading the Linux kernel image into RAM before transferring execution. Alternatively, as we will see later on, it may also hand off to a next-stage UEFI boot manager.

The first phase of U-Boot's SPL consists of M-mode assembly entry. The hardware reset vector gets directly set to the entry point _start in arch/riscv/cpu/start.S. The hardware setup includes configuring fundamental control and status registers (CSRs), like disabling interrupts (mstatus), clearing delegation registers (medeleg/mideleg), and setting the trap vector base address (mtvec) to catch early faults. The hart ID is extracted by querying via the csrr a0, mhartid instruction. The primary executing core (boot "hart") continues initialisation, while auxiliary harts are placed in a secondary wait loop if symmetric multiprocessing (CONFIG_SPL_SMP) is enabled, as is the case on QEMU. An early runtime C stack is prepared in internal SRAM or cache-as-RAM (L2 cache). The gp (Global Pointer) register is set up to reserve space for the early global data structure pointer (gd).

The second phase consists of the pre-RAM initialisation (board_init_f), which now runs as a C runtime within arch/riscv/lib/spl.c. The spl_early_init() sets up a basic driver model (DM) tree structure to find essential SoC components configured via device tree, either passed by QEMU or baked into U-Boot SPL. The riscv_cpu_setup() issues early CPU core tweaks and configures the physical memory protection (PMP) boundaries. The preloader_console_init() prepares the system console (typically a standard 16550 UART or similar device), allowing debug log prints. And spl_board_init_f() is the board/SoC-specific implementation called to power up, configure clock phase-locked loops (PLLs), and explicitly initialise the external RAM.

The third phase consists of the post-RAM initialisation (board_init_r), which now runs within generic U-Boot SPL common/spl/spl.c. The operational stack and active global data structures get transferred out of early internal SRAM into RAM if configured to do so (CONFIG_SPL_STACK_R), however, on QEMU this is not the case. Storage media drivers and devices, such as MMC/SD cards or NAND/NOR flashes, get initialised as preparation for loading the next stage payloads, the exact source of which may be dynamically discovered.

The fourth and final phase of the SPL consists of image loading and executing the jump into OpenSBI, which does the transition from M-mode, where the SPL runs, to S-mode for U-Boot proper to accommodate an operating system like Linux. The FIT image gets parsed and may consist of U-Boot proper, OpenSBI and optionally a device tree blob (DTB). However, QEMU usually passes in its own DTB via FW_DYNAMIC mechanism. U-Boot proper gets loaded into its target execution RAM address, while OpenSBI is loaded into its respective baseline location. The instruction pipeline is flushed in invalidate_icache_all(), clearing caches. Last but not least, the jump gets initiated by loading the target execution entry address to point directly to OpenSBI's entry point. The information gets passed via regular RISC-V registers: a0 receives the boot hart ID (gd->arch.boot_hart), a1 the memory pointer referencing the DTB, and a2 the memory pointer referencing the struct fw_dynamic_info.

Once OpenSBI completes its M-mode initialisation, it transitions to S-mode, and ultimately branches to the pre-loaded U-Boot proper entry address in system RAM to continue the full boot sequence.

The first phase of U-Boot proper consists of S-mode assembly entry (start.S), where the incoming information from OpenSBI is parsed, the S-mode trap vector base address register (stvec) is set up to catch early faults, the S-mode interrupt enable flags (sie) are cleared to ensure sequential, deterministic initialisation, and the global pointer registers, thread pointer (tp) and global pointer (gp), get loaded.

The second phase consists of pre-relocation C initialisation (board_init_f), where a sequential array of early setup functions get executed, the driver model (DM) gets activated in dm_init_f from device tree information for early core devices (clocks, reset controllers, and pin multiplexers), the memory layout is calculated reserving space at the top of memory to store the relocated U-Boot binary, the master page tables, the permanent stack, and the new heap.

The third phase consists of the binary relocation relocate.S, where U-Boot migrates itself out of the way towards the top of memory leaving low memory addresses available for further boot stages, the .rela.dyn relocation tables are iterated updating all address offsets to point to the newly relocated ones, the RISC-V fence instruction fence.i gets called to flush any stale instructions out of the CPU pipeline execution cache.

The fourth phase consists of the post-relocation C initialisation (board_init_r), where the full suite of drivers as declared in the device tree gets bound to, the environment gets loaded from non-volatile storage with configuration scripts and parameters (e.g. bootargs, bootcmd), and further subsystems like Ethernet, PCIe/NVMe, and USB get dynamically probed.

The fifth and final phase consists of the main loop and OS handover, where user interaction may be established or the default boot initiated after a certain countdown time. The interactive shell features command parsing in a cli_loop, where user debugging commands are evaluated. The OS handover gets initiated by a booting call like booti, bootm or bootefi, where the system gets prepared to transfer control to the next S-mode stage, first disabling all active hardware peripheral timers, flushing caches, and triggering the next entry point. Just like in previous stages, information gets passed using the FW_DYNAMIC mechanism via CPU registers.

We get the U-Boot source code:

⬢ [zim@toolbx ~]$ git clone https://github.com/u-boot/u-boot.git
⬢ [zim@toolbx ~]$ cd u-boot

However, if we do like the OpenSBI banner to be printed, we need to change its configuration as follows:

⬢ [zim@toolbx u-boot]$ git diff
diff --git a/configs/qemu-riscv64_spl_defconfig b/configs/qemu-riscv64_spl_defconfig
index 04dec72ceac7..4212699c3dd3 100644
--- a/configs/qemu-riscv64_spl_defconfig
+++ b/configs/qemu-riscv64_spl_defconfig
@@ -18,6 +18,7 @@ CONFIG_DISPLAY_BOARDINFO=y
 # CONFIG_BOARD_INIT is not set
 CONFIG_SPL_MAX_SIZE=0x100000
 CONFIG_SPL_SYS_MALLOC=y
+CONFIG_SPL_OPENSBI_SCRATCH_OPTIONS=0x0
 # CONFIG_CMD_MII is not set
 CONFIG_ENV_RELOC_GD_ENV_ADDR=y
 CONFIG_DM_MTD=y

Guess how/where I found out about this OpenSBI scratch-options handling? Yes, the attentive reader noticed it during our previous debugging session (the terminal box just above the U-Boot section) at line 32, source code line 51!

Only now we configure/build U-Boot for SPL:

⬢ [zim@toolbx u-boot]$ make qemu-riscv64_spl_defconfig
⬢ [zim@toolbx u-boot]$ make OPENSBI=../opensbi/build/platform/generic/firmware/fw_dynamic.bin
...
  LDS     spl/u-boot-spl.lds
  LD      spl/u-boot-spl
  OBJCOPY spl/u-boot-spl-nodtb.bin
mkdir -p spl/dts/
  FDTGREP spl/dts/dt-spl.dtb
  COPY    spl/u-boot-spl.dtb
  CAT     spl/u-boot-spl-dtb.bin
  COPY    spl/u-boot-spl.bin
  SYM     spl/u-boot-spl.sym
  MKIMAGE u-boot.img
  COPY    u-boot.dtb
  MKIMAGE u-boot-dtb.img
  BINMAN  .binman_stamp
  OFCHK   .config

This actually builds the SPL binary spl/u-boot-spl.bin, as well as the U-Boot proper binary u-boot.bin packaged together with the OpenSBI binary fw_dynamic.bin, into a single flattened image tree (FIT) itb.fit.fit blob. Its former name was u-boot.itb, where ITB meant image tree blob. As I am more familiar with that former name, I will stick to u-boot.itb throughout this article. Such .fit or .itb files may be inspected using U-Boot's mkimage tool:

⬢ [zim@toolbx u-boot]$ mkimage -l u-boot.itb
FIT description: Configuration to load OpenSBI before U-Boot
Created:         Sat May 30 18:23:39 2026
 Image 0 (uboot)
  Description:  U-Boot
  Created:      Sat May 30 18:23:39 2026
  Type:         Standalone Program
  Compression:  uncompressed
  Data Size:    895160 Bytes = 874.18 KiB = 0.85 MiB
  Architecture: RISC-V
  Load Address: 0x81200000
  Entry Point:  unavailable
 Image 1 (opensbi)
  Description:  OpenSBI fw_dynamic Firmware
  Created:      Sat May 30 18:23:39 2026
  Type:         Firmware
  Compression:  uncompressed
  Data Size:    275768 Bytes = 269.30 KiB = 0.26 MiB
  Architecture: RISC-V
  OS:           RISC-V OpenSBI
  Load Address: 0x80100000
 Default Configuration: 'conf-1'
 Configuration 0 (conf-1)
  Description:  NAME
  Kernel:       unavailable
  Firmware:     opensbi
  Loadables:    uboot

The interested reader may find much more information in the Flattened Image Tree Specification.

But, now, let us actually launch this by QEMU, which is slightly more complex, as not only a -bios argument, but also a loader device with the additional parts needs to be specified:

⬢ [zim@toolbx u-boot]$ qemu-system-riscv64 -bios spl/u-boot-spl.bin \
-device loader,file=u-boot.itb,addr=0x80200000 -machine virt -nographic

U-Boot SPL 2026.07-rc3-00008-g987907ae4bcc-dirty (Jun 01 2026 - 04:26:09 +0200)
Trying to boot from RAM

OpenSBI v1.8.1
Build time: 2026-05-29 17:28:04 +0200
Build compiler: clang version 21.1.1 (https://github.com/llvm/llvm-project.git
 5a86dc996c26299de63effc927075dcbfb924167)
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name               : riscv-virtio,qemu
Platform Features           : medeleg
Platform HART Count         : 1
Platform HART Protection    : pmp
Platform IPI Device         : aclint-mswi
Platform Timer Device       : aclint-mtimer @ 10000000Hz
Platform Console Device     : uart8250
Platform HSM Device         : ---
Platform PMU Device         : ---
Platform Reboot Device      : syscon-reboot
Platform Shutdown Device    : syscon-poweroff
Platform Suspend Device     : ---
Platform CPPC Device        : ---
Firmware Base               : 0x80100000
Firmware Size               : 321 KB
Firmware RW Offset          : 0x40000
Firmware RW Size            : 65 KB
Firmware Heap Offset        : 0x47000
Firmware Heap Size          : 37 KB (total), 0 KB (reserved), 12 KB (used), 23 KB (free)
Firmware Scratch Size       : 4096 B (total), 1464 B (used), 2632 B (free)
Runtime SBI Version         : 3.0
Standard SBI Extensions     : time,rfnc,ipi,base,hsm,srst,pmu,dbcn,fwft,legacy,dbtr,sse
Experimental SBI Extensions : none

Domain0 Name                : root
Domain0 Boot HART           : 0
Domain0 HARTs               : 0*
Domain0 Region00            : 0x0000000080140000-0x000000008015ffff M: (F,R,W) S/U: ()
Domain0 Region01            : 0x0000000080100000-0x000000008013ffff M: (F,R,X) S/U: ()
Domain0 Region02            : 0x0000000000100000-0x0000000000100fff M: (I,R,W) S/U: (R,W)
Domain0 Region03            : 0x0000000010000000-0x0000000010000fff M: (I,R,W) S/U: (R,W)
Domain0 Region04            : 0x0000000002000000-0x000000000200ffff M: (I,R,W) S/U: ()
Domain0 Region05            : 0x000000000c400000-0x000000000c5fffff M: (I,R,W) S/U: (R,W)
Domain0 Region06            : 0x000000000c000000-0x000000000c3fffff M: (I,R,W) S/U: (R,W)
Domain0 Region07            : 0x0000000000000000-0xffffffffffffffff M: () S/U: (R,W,X)
Domain0 Next Address        : 0x0000000081200000
Domain0 Next Arg1           : 0x00000000812da8c8
Domain0 Next Mode           : S-mode
Domain0 SysReset            : yes
Domain0 SysSuspend          : yes

Boot HART ID                : 0
Boot HART Domain            : root
Boot HART Priv Version      : v1.12
Boot HART Base ISA          : rv64imafdch
Boot HART ISA Extensions    : sstc,zicntr,zihpm,zicboz,zicbom,sdtrig,svadu
Boot HART PMP Count         : 16
Boot HART PMP Granularity   : 2 bits
Boot HART PMP Address Bits  : 54
Boot HART MHPM Info         : 16 (0x0007fff8)
Boot HART Debug Triggers    : 2 triggers
Boot HART MIDELEG           : 0x0000000000001666
Boot HART MEDELEG           : 0x0000000000f4b509


U-Boot 2026.07-rc3-00008-g987907ae4bcc-dirty (Jun 01 2026 - 04:26:09 +0200)

CPU:   riscv
Model: riscv-virtio,qemu
DRAM:  128 MiB
using memory 0x86ef5000-0x87715000 for malloc()
Core:  27 devices, 14 uclasses, devicetree: board
Flash: 32 MiB
Loading Environment from nowhere... OK
In:    serial,usbkbd
Out:   serial,vidconsole
Err:   serial,vidconsole
No USB controllers found
Net:   No ethernet found.

Hit any key to stop autoboot: 0

Device 0: unknown device

Device 0: unknown device

Device 1: unknown device
scanning bus for devices...

Device 0: unknown device
No ethernet found.
No ethernet found.
=> poweroff
poweroff ...

Debugging

Now, it starts to get interesting. We can launch QEMU again in the first terminal like before, but again with the -S argument, which instructs it to just wait for the gdb debugger to connect to it and further specify what to do next:

⬢ [zim@toolbx u-boot]$ qemu-system-riscv64 -S -bios spl/u-boot-spl.bin \
-device loader,file=u-boot.itb,addr=0x80200000 -gdb tcp::1234 -machine virt -nographic

In the second terminal, we start gdb, giving it not only the OpenSBI symbol file as before, but also U-Boot's SPL and U-Boot proper symbol files as well. We instruct it to break at board_init_f and spl_invoke_opensbi in the U-Boot SPL, sbi_boot_print_banner and sbi_hart_switch_mode in OpenSBI and display_options in U-Boot proper. Then we can let it print the various banners line by line:

⬢ [zim@toolbx u-boot]$ gdb -ex 'add-symbol-file spl/u-boot-spl 0x80000000' \
-ex 'add-symbol-file ../opensbi/build/platform/generic/firmware/fw_payload.elf 0x80100000' \
-ex 'add-symbol-file u-boot 0x81200000' \
-ex 'add-symbol-file u-boot 0x87715000' -ex 'target remote localhost:1234'
GNU gdb (Fedora Linux) 17.1-6.fc44
Copyright (C) 2025 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
add symbol table from file "spl/u-boot-spl" at
    .text_addr = 0x80000000
(y or n) y
Reading symbols from spl/u-boot-spl...
add symbol table from file "../opensbi/build/platform/generic/firmware/fw_payload.elf" at
    .text_addr = 0x80100000
(y or n) y
Reading symbols from ../opensbi/build/platform/generic/firmware/fw_payload.elf...
add symbol table from file "u-boot" at
    .text_addr = 0x81200000
(y or n) y
Reading symbols from u-boot...
add symbol table from file "u-boot" at
    .text_addr = 0x87715000
(y or n) y
Reading symbols from u-boot...
Remote debugging using localhost:1234
⚠️ warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0x0000000000001000 in ?? ()
(gdb) b board_init_f
Breakpoint 1 at 0x80000a3a: board_init_f. (3 locations)
(gdb) b spl_invoke_opensbi
Breakpoint 2 at 0x8000167a: file common/spl/spl_opensbi.c, line 55.
(gdb) b sbi_boot_print_banner
Breakpoint 3 at 0x8010b14c: file /var/home/zim/Sources/opensbi/lib/sbi/sbi_init.c, line 51.
(gdb) b sbi_hart_switch_mode
Breakpoint 4 at 0x80104b68: file /var/home/zim/Sources/opensbi/lib/sbi/sbi_hart.c, line 779.
(gdb) b display_options
Breakpoint 5 at 0x8125bec0: display_options. (2 locations)
(gdb) c
Continuing.

Breakpoint 1.1, board_init_f (dummy=0) at arch/riscv/lib/spl.c:26
26      ret = spl_early_init();
(gdb) n
27      if (ret)
(gdb) n
30      riscv_cpu_setup();
(gdb) n
32      preloader_console_init();
(gdb) n
34      ret = spl_board_init_f();
(gdb) c
Continuing.

Breakpoint 2, spl_invoke_opensbi (spl_image=0x801fff60, spl_image@entry=0x801ffe90) at common/spl/spl_opensbi.c:55
55      if (!spl_image->fdt_addr) {
(gdb) n
60      if (!IS_ALIGNED((uintptr_t)spl_image->fdt_addr, 8)) {
(gdb) n
89      ret = spl_opensbi_find_os_node(spl_image->fdt_addr, &os_node, os_type);
(gdb) n
97      ret = fit_image_get_entry(spl_image->fdt_addr, os_node, &os_entry);
(gdb) n
98      if (ret)
(gdb) n
99          ret = fit_image_get_load(spl_image->fdt_addr, os_node, &os_entry);
(gdb) n
102     opensbi_info.magic = FW_DYNAMIC_INFO_MAGIC_VALUE;
(gdb) n
103     opensbi_info.version = FW_DYNAMIC_INFO_VERSION;
(gdb) n
105     opensbi_info.next_mode = FW_DYNAMIC_INFO_NEXT_MODE_S;
(gdb) n
106     opensbi_info.options = CONFIG_SPL_OPENSBI_SCRATCH_OPTIONS;
(gdb) n
107     opensbi_info.boot_hart = gd->arch.boot_hart;
(gdb) n
109     opensbi_entry = (opensbi_entry_t)spl_image->entry_point;
(gdb) n
110     invalidate_icache_all();
(gdb) n
123     ret = smp_call_function((ulong)spl_image->entry_point,
(gdb) n
126     if (ret)
(gdb) n
129     opensbi_entry(gd->arch.boot_hart, (ulong)spl_image->fdt_addr,
(gdb) c
Continuing.

Breakpoint 3, sbi_boot_print_banner (scratch=0x80146000) at /var/home/zim/Sources/opensbi/lib/sbi/sbi_init.c:51
51      if (scratch->options & SBI_SCRATCH_NO_BOOT_PRINTS)
(gdb) n
55      sbi_printf("\nOpenSBI %s\n", OPENSBI_VERSION_GIT);
(gdb) n
62      sbi_printf("Build time: %s\n", OPENSBI_BUILD_TIME_STAMP);
(gdb) n
66      sbi_printf("Build compiler: %s\n", OPENSBI_BUILD_COMPILER_VERSION);
(gdb) n
69      sbi_printf(BANNER);
(gdb) n
70  }
(gdb) c
Continuing.

Breakpoint 4, sbi_hart_switch_mode (arg0=0, arg1=2167253192, next_addr=2166358016, next_mode=1, next_virt=false) at /var/home/zim/Sources/opensbi/lib/sbi/sbi_hart.c:779
779     switch (next_mode) {
(gdb) n
783         if (!misa_extension('S'))
(gdb) n
794     val = csr_read(CSR_MSTATUS);
(gdb) n
795     val = INSERT_FIELD(val, MSTATUS_MPP, next_mode);
(gdb) n
804     if (misa_extension('H'))
(gdb) n
807     csr_write(CSR_MSTATUS, val);
(gdb) n
808     csr_write(CSR_MEPC, next_addr);
(gdb) n
810     if (next_mode == PRV_S) {
(gdb) n
811         if (next_virt) {
(gdb) n
817             csr_write(CSR_STVEC, next_addr);
(gdb) n
818             csr_write(CSR_SSCRATCH, 0);
(gdb) n
819             csr_write(CSR_SIE, 0);
(gdb) n
820             csr_write(CSR_SATP, 0);
(gdb) n
832     __asm__ __volatile__("mret" : : "r"(a0), "r"(a1));
(gdb) c
Continuing.

Breakpoint 1.2, board_init_f (boot_flags=0) at common/board_f.c:1036
1036        gd->flags &= ~GD_FLG_HAVE_CONSOLE;
(gdb) c
Continuing.

Breakpoint 5.1, display_options () at lib/display_options.c:50
50      display_options_get_banner(true, buf, sizeof(buf));
(gdb) n
51      printf("%s", buf);
(gdb) n
53      return 0;
(gdb) c
Continuing.
[Inferior 1 (process 1) exited normally]
(gdb) exit

The preloader_console_init in U-Boot SPL is what prints its banner and spl_invoke_opensbi is where it hands over execution to OpenSBI. The sbi_boot_print_banner as we learned already, prints OpenSBI's banner and sbi_hart_switch_mode does the mret instruction handing over execution to U-Boot proper. The display_options in U-Boot proper finally prints its banner.

The observant reader may have noticed that we instructed gdb to load U-Boot proper's symbols twice. This has to do with the fact that U-Boot proper actually re-locates itself during execution towards the end of RAM leaving as much RAM as possible free to load further artifacts to be booted. To find the relocation address one can use U-Boot's bdinfo command:

=> bdinfo
boot_params = 0x0000000000000000
DRAM bank   = 0x0000000000000000
-> start    = 0x0000000080000000
-> size     = 0x0000000008000000
flashstart  = 0x0000000020000000
flashsize   = 0x0000000002000000
flashoffset = 0x0000000000000000
baudrate    = 115200 bps
relocaddr   = 0x0000000087715000
reloc off   = 0x0000000006515000
Build       = 64-bit
current eth = unknown
eth-1addr   = (not set)
IP addr     = <NULL>
fdt_blob    = 0x0000000086ef28e0
lmb_dump_all:
 memory.count = 0x1
 memory[0]  [0x80000000-0x87ffffff], 0x8000000 bytes, flags: none
 reserved.count = 0x3
 reserved[0]    [0x80100000-0x8015ffff], 0x60000 bytes, flags: no-map
 reserved[1]    [0x85eef000-0x85ef1fff], 0x3000 bytes, flags: no-notify, no-overwrite
 reserved[2]    [0x85ef28d0-0x87ffffff], 0x210d730 bytes, flags: no-overwrite
devicetree  = board
serial addr = 0x0000000010000000
 width      = 0x0000000000000001
 shift      = 0x0000000000000000
 offset     = 0x0000000000000000
 clock      = 0x0000000000384000
boot hart   = 0x0000000000000000
firmware fdt= 0x00000000812da8c8

UEFI Boot Manager

Systemd-boot (sd-boot for short, previously known as gummiboot) is a simple, easy-to-configure UEFI (Unified Extensible Firmware Interface) boot manager. It provides a textual menu to select the entry to boot and an editor for the kernel command line. Its source code is located primarily within src/boot/ (previously src/boot/efi/) within systemd.

The first phase consists of the entry point handover, where control gets transferred to the primary UEFI payload with efi_main() in src/boot/efi/boot.c being the initial code hook. The UEFI firmware, in our case U-Boot proper, passes standard arguments providing access to core console interfaces, system tables, and basic file system drivers.

The second phase consists of execution environment sanity checks, where the configuration is validated, a watchdog is initialised to safely reset the platform if a hard freeze occurs during image parsing, and optional secure boot checks are invoked, including random seed provisioning. Note that secure boot is outside the scope of this article.

The third phase consists of driver and file system probing, where file-access routines get initialised to locate available OS targets. The EFI system partition (ESP) where the binary was run from gets probed, and supplementary storage or file system drivers are loaded by iterating over /EFI/systemd/drivers/. It dynamically handles and loads any such compiled specifically with the architecture suffix riscv64.efi.

The fourth phase consists of config processing and menu presentation, where /loader/loader.conf gets parsed to handle general preferences, /loader/entries/*.conf gets scanned for target detection, searching inside /EFI/Linux/ looking for Unified Kernel Images (UKIs), and drawing the interactive boot menu, usually automatically booting the default target after a timeout occurred.

The fifth and final phase consists of executing the Kernel via EFI stub, where systemd-boot hands execution directly off to a kernel image after selection. It calls the EFI_BOOT_SERVICES.StartImage() execution standard, as the Linux kernel acts like a second UEFI application. Hardware information gets passed through via flattened device tree (FDT) or ACPI tables. Note that ACPI is outside the scope of this article.

To build systemd to include systemd-boot, make sure to have python3-pyelftools available and proceed as follows:

⬢ [zim@toolbx ~]$ git clone https://github.com/systemd/systemd.git
⬢ [zim@toolbx ~]$ cd systemd
⬢ [zim@toolbx systemd]$ git checkout -b v260.2-test v260.2
⬢ [zim@toolbx systemd]$ cat riscv64.txt
[binaries]
c = '/var/home/zim/Toolchains/riscv64-glibc-ubuntu-24.04-gcc-2026.05.19-nightly/bin/riscv64-unknown-linux-gnu-gcc'
cpp = '/var/home/zim/Toolchains/riscv64-glibc-ubuntu-24.04-gcc-2026.05.19-nightly/bin/riscv64-unknown-linux-gnu-g++'
ar = '/var/home/zim/Toolchains/riscv64-glibc-ubuntu-24.04-gcc-2026.05.19-nightly/bin/riscv64-unknown-linux-gnu-ar'
strip = '/var/home/zim/Toolchains/riscv64-glibc-ubuntu-24.04-gcc-2026.05.19-nightly/bin/riscv64-unknown-linux-gnu-strip'
exe_wrapper = 'qemu-system-riscv64 -nographic'

[host_machine]
system = 'linux'
cpu_family = 'riscv64'
cpu = 'riscv64'
endian = 'little'
⬢ [zim@toolbx systemd]$ meson setup --cross-file riscv64.txt build/ && ninja -C build/
⬢ [zim@toolbx systemd]$ file build/src/boot/systemd-bootriscv64.efi
build/src/boot/systemd-bootriscv64.efi: PE32+ executable for EFI (application),
 RISC-V 64-bit (stripped to external PDB), 6 sections

Note: The latest main branch did not work for me, which is why I picked released v260.2 instead, which seems to work fine.

Testing/debugging sd-boot is slightly more involved, as one can't just easily deploy it on its own, plus what would be the point, as we want it to subsequently boot a full Linux system. So we need to build one for RISC-V first. We can do this using Freedesktop SDK:

⬢ [zim@toolbx ~]$ git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk.git
⬢ [zim@toolbx ~]$ cd freedesktop-sdk
⬢ [zim@toolbx freedesktop-sdk]$ bst --colors -o bootstrap_build_arch x86_64 \
-o target_arch riscv64 build vm/minimal/efi.bst
⬢ [zim@toolbx freedesktop-sdk]$ bst --colors -o bootstrap_build_arch x86_64 \
-o target_arch riscv64 checkout vm/minimal/efi.bst vm-minimal-efi/

We can now replace sd-boot inside the EFI partition of that disk.img with our previously built one:

zim@fedora:~$ sudo losetup -P /dev/loop0 freedesktop-sdk/vm-minimal-efi/disk.img
zim@fedora:~$ sudo mount /dev/loop0p1 /mnt
zim@fedora:~$ sudo cp systemd/build/src/boot/systemd-bootriscv64.efi \
/mnt/EFI/BOOT/BOOTRISCV64.EFI
zim@fedora:~$ sudo umount /mnt
zim@fedora:~$ sudo losetup -D

With that in place, we can now launch it in QEMU:

⬢ [zim@toolbx u-boot]$ qemu-system-riscv64 -bios spl/u-boot-spl.bin \
-device loader,file=u-boot.itb,addr=0x80200000 \
-device virtio-blk-device,drive=hd0 -device virtio-net-pci,netdev=n1 \
-drive file=../freedesktop-sdk/vm-minimal-efi/disk.img,format=raw,id=hd0,if=none \
-m 256M -machine virt -netdev user,id=n1 -nographic

Note: Without setting the -m 256M argument to QEMU, it will fail to load with an Unhandled exception: Load access fault.

Once we are on the U-Boot command line shell, we can poke it some more to find out about what it loads next:

=> echo $bootcmd
run distro_bootcmd
=> echo $distro_bootcmd
scsi_need_init=; setenv nvme_need_init; virtio_need_init=; for target in ${boot_targets};
 do run bootcmd_${target}; done
=> echo $boot_targets
nvme0 virtio0 virtio1 scsi0 dhcp
=> echo $bootcmd_virtio0
devnum=0; run virtio_boot
=> echo $virtio_boot
run boot_pci_enum; run virtio_init; if virtio dev ${devnum}; then devtype=virtio;
 run scan_dev_for_boot_part; fi
=> echo $scan_dev_for_boot_part
if env exists distro_bootpart; then setenv devplist ${distro_bootpart};
 else part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1;
 fi; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype;
 then part uuid ${devtype} ${devnum}:${distro_bootpart} distro_bootpart_uuid ;
 run scan_dev_for_boot; fi; done; setenv devplist
=> echo $scan_dev_for_boot
echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes};
 do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_for_efi;
=> echo $scan_dev_for_efi
setenv efi_fdtfile ${fdtfile}; for prefix in ${efi_dtb_prefixes};
 do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile};
 then run load_efi_dtb; fi;done;run boot_efi_bootmgr;
 if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootriscv64.efi;
 then echo Found EFI removable media binary efi/boot/bootriscv64.efi; run boot_efi_binary;
 echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfile
=> echo $boot_efi_binary
load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootriscv64.efi;
 if fdt addr -q ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};
 else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi
=> load virtio 0:1 $loadaddr efi/boot/bootriscv64.efi
176128 bytes read in 1 ms (168 MiB/s)
=> bootefi $loadaddr
Missing RNG device for EFI_RNG_PROTOCOL
No RNG device
Booting /efi\boot\bootriscv64.efi
systemd-boot@0x8de61000 260.2

And once we are fully booted, we can look at the partition layout and where sd-boot's binary and configuration got deployed to:

Freedesktop Platform 26.08beta localhost ttyS0
localhost login: root
Password:
[root@localhost ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
vda    254:0    0  489M  0 disk
├─vda1 254:1    0   36M  0 part /efi
└─vda2 254:2    0  452M  0 part /
[root@localhost ~]# ls -l /efi/EFI/BOOT/BOOTRISCV64.EFI
-rwx------ 1 root root 101888 Nov 10  2011 /efi/EFI/BOOT/BOOTRISCV64.EFI
[root@localhost ~]# file /efi/EFI/BOOT/BOOTRISCV64.EFI
/efi/EFI/BOOT/BOOTRISCV64.EFI: PE32+ executable for EFI (application),
 RISC-V 64-bit (stripped to external PDB), 7 sections
[root@localhost ~]# cat /efi/loader/loader.conf
timeout 3
editor yes
console-mode keep

Debugging

Finally, the culmination. We can launch QEMU again in the first terminal like before, but again with the -S argument, which instructs it to just wait for the gdb debugger to connect to it and further specify what to do next:

⬢ [zim@toolbx u-boot]$ qemu-system-riscv64 -S -bios spl/u-boot-spl.bin \
-device loader,file=u-boot.itb,addr=0x80200000 \
-device virtio-blk-device,drive=hd0 -device virtio-net-pci,netdev=n1 \
-drive file=../freedesktop-sdk/vm-minimal-efi/disk.img,format=raw,id=hd0,if=none \
-gdb tcp::1234 -m 256M -machine virt -netdev user,id=n1 -nographic

In the second terminal, we start gdb, giving it sd-boot's symbol file, and we instruct it to break at notify_debugger in sd-boot. Then, we can let it print sd-boot's banner:

⬢ [zim@toolbx u-boot]$ gdb -ex 'add-symbol-file ../systemd/build/src/boot/systemd-bootriscv64.elf 0x8de62000' \
-ex 'target remote localhost:1234'
GNU gdb (Fedora Linux) 17.1-6.fc44
Copyright (C) 2025 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
add symbol table from file "../systemd/build/src/boot/systemd-bootriscv64.elf" at
    .text_addr = 0x8de62000
(y or n) y
Reading symbols from ../systemd/build/src/boot/systemd-bootriscv64.elf...
Remote debugging using localhost:1234
⚠️ warning: No executable has been specified and target does not support
determining executable automatically.  Try using the "file" command.
0x0000000000001000 in ?? ()
(gdb) b notify_debugger
Breakpoint 1 at 0x8de7e004: file ../src/boot/util.c, line 366.
(gdb) c
Continuing.

Breakpoint 1, notify_debugger (identity=0x8de87ae0 "systemd-boot", wait=false) at ../src/boot/util.c:366
366         printf("%s@%p %s\n", identity, __executable_start, GIT_VERSION);
(gdb) n
367         if (wait)
(gdb) c
Continuing.
[Inferior 1 (process 1) exited normally]
(gdb) exit

The watchful reader might have noticed that the address 0x8de62000 we loaded sd-boot's symbols for differs from the address previously reported systemd-boot@0x8de61000 260.2. Yes, there is an additional text segment offset of 0x1000, which we have to keep in mind.

Initially, I had some trouble getting gdb to ever show me the correct symbols. I then stumbled over the Debugging systemd-boot section of systemd's hacking documentation, but I was unable to get gdb to ever stop at a reasonable location to figure out what is going on.

Digging deeper, I found notify_debugger, which was just missing the RISC-V case in order to ease stopping gdb at a sensible location. I patched the missing RISC-V assembly instruction in (also sent an Upstream pull request), and noticed the address just being off by 0x1000. There you go!

Linux Kernel

The Linux kernel needs to be built as an EFI executable (CONFIG_EFI_STUB) for the EFI boot manager to be able to execute it:

Freedesktop Platform 26.08beta localhost ttyS0
localhost login: root
Password:
[root@localhost ~]# ls /efi
config  dtbs  EFI  initramfs-6.19.6.img  loader  System.map  ubootefi.var  vmlinuz
[root@localhost ~]# file /efi/vmlinuz
/efi/vmlinuz: Linux kernel RISC-V64 EFI executable gzip compressed zboot Image

Note that the Linux kernel EFI stub exits the boot services by executing ExitBootServices(), terminating systemd-boot's operations and shifting control into the Linux S-mode kernel environment.

Conclusion

The RISC-V boot process is well understood, and we are ready to tackle applying our knowledge to any RISC-V system out there. Looking forward to getting our hands on the first RVA23 boards, making sure they have their upstream boot chains ready.

Porting an Automotive Operating System to RISC-V

2026-06-02T00:00:00+01:00

RISC-V in Automotive

RISC-V is an open instruction set architecture (ISA) based on reduced instruction set computer (RISC) principles. RISC-V has a growing ecosystem, supported by companies including NVIDIA and Qualcomm, and open-source groups, positioning it as a viable alternative to traditional proprietary architectures.

Equipped to be a major player in the automotive industry, RISC-V’s extensible architecture enables automotive-specific specialisations and optimisations to be integrated, ensuring the hardware meets the use case.

The unified architecture of RISC-V enables it to scale from simple sensors to high-performance autonomous driving systems on chips (SoCs). For example, a car might use RISC-V microcontrollers and powerful RISC-V-based SoCs in advanced driver assistance systems (ADAS), leveraging the same instruction set and tooling across very different compute needs.

RISC-V allows for multiple processing units all to use the same base ISA with different extensions, depending on the unit's purpose. Resulting in a chiplet architecture made up of graphics processing units (GPUs), neural processing units (NPUs) and central processing units (CPUs), all of which are based on RISC-V.

In addition, the open nature allows for multi-sourcing. For instance, an automotive supplier can source RISC-V chips from multiple vendors such as Infineon and SiFive, mitigating risks associated with supply chain shortages and reducing dependency on a single vendor. As a result, RISC-V is on the brink of transforming the automotive industry.

It may have already started. Infineon announced the release of an automotive microcontroller family based on RISC-V. This will be part of Infineon’s well-established AURIX microcontroller brand, with a virtual prototype already available. NVIDIA, one of the leading companies in the automotive industry, already shipped one billion RISC-V cores in 2024.

Porting Software

RISC-V, as a new ISA, requires software to be ported to support it. This work is being undertaken by an extensive community, including companies, working groups, and open-source community members. Due to RISC-V’s clear alignment with free and open-source software (FOSS) ideals, there has been a highly enthusiastic community response, including support for RISC-V added to the Linux kernel in 2017.

However, many components, both FOSS and proprietary, still do not support RISC-V.

Codethink Trustable and Reproducible Linux (CTRL) is Codethink's own safety-oriented platform development kit (PDK) used in automotive and industrial projects. Due to the clear benefits of RISC-V’s use in automotive and its alignment with FOSS principles, Codethink added RISC-V support to CTRL to demonstrate porting a large mature software project to RISC-V.

CTRL is built upon the foundation of Freedesktop SDK, which already supports RISC-V. Therefore, it is well placed to answer the question: how hard is it to port a Linux-based OS to a RISC-V target?

CTRL to RISC-V

Adding support for a new ISA can be more involved than just adding an additional target. Therefore, the porting of CTRL is a phased adoption of RISC-V, broken into three parts:

Supporting RISC-V target compilation in the build environment
Building CTRL for RISC-V and running on a virtual target
Adding support for actual RISC-V hardware target

Updating the build container to add support for RISC-V targets means ensuring all tools in the container support the RISC-V ISA, mirroring any additional dependencies for the RISC-V versions of the tooling, and adding the RISC-V container build as an optional, manual step as part of the containers' continuous development (CD) pipeline. This was fairly straightforward; the only challenges centred on components that didn’t produce releases for RISC-V, including OpenTofu and Jsonnet. For this initial implementation, these components could be excluded from the RISC-V container as they were not required for the later stages of porting CTRL.

The second stage of RISC-V adoption requires the support of virtual targets, which are crucial in automotive as they align with “shift-left” strategies from software-defined vehicles (SDV). Using a virtual target enables shorter cycle times and easier debugging during RISC-V porting. Since CTRL already supports both Aarch64 and x86_64 QEMU virtual targets, it is a logical extension to support the qemu-system-riscv64 target.

When porting to the QEMU RISC-V target, the main challenges faced were related to intricacies in the UEFI boot stage. UEFI expects the next-stage binary, in this case, grub2, to be named bootriscv64, which differs significantly from the x86 and Aarch64 naming patterns (bootaa64 and bootx64). Further on this front, the default edk2 vars meant that the UEFI stage was not attempting to run the grub2 binary. The final QEMU command used to run CTRL for reference was:

qemu-system-riscv64 -nographic -M virt -m 2G -smp 4 \
    object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 \
    nic user,model=virtio,hostfwd=tcp::2222-:22 \
    drive if=pflash,format=raw,file=edk2-riscv-code.fd,readonly=true \
    drive format=raw,file=riscv_image/disk.img

After porting to the virtual target, and confirming all the expected functionality was present, CTRL was ported to the RISC-V hardware target, the Banana Pi BPI-F3. This board was chosen as it is readily available and widely supported throughout the RISC-V ecosystem, containing a SpacemiT K1 octa-core RISC-V processor. The process was made up of three main steps:

Building the bootloader components (OpenSBI and U-boot)
Building the BPI-F3 Linux kernel
Constructing a flashable image for an SD card

Although the sources for the BPI-F3 versions of OpenSBI, U-boot and the Linux kernel are easy to find and construct, they are maintained in forked repositories, rather than being upstreamed to official open-source repositories. This is a common pattern among RISC-V hardware, and other architectures as well, that is detrimental to the industry. By maintaining these sources separately, these sources will eventually drift from the upstream, meaning they lag behind in functionality, and the patches applied will be tested less, increasing the chances of bugs.

Challenges Encountered

Throughout this work, there were two issues common across all stages.

The first issue is the comparatively poor performance of the currently available RISC-V hardware when compared to Arm and x86. Since CTRL is built with Apache BuildStream, which is primarily aimed at native compilation, CTRL had to be compiled on RISC-V hardware. For a full build of CTRL, with every component built from source, this took over 24 hours on the Milk-V Pioneer hardware. BuildStream has in-built caching of components, which supported a significantly quicker rebuilding process and reduced the time to around 2 hours for full kernel rebuilds.

The second recurring issue is that, since RISC-V is still relatively new compared to other architectures, many projects still do not produce RISC-V releases as standard, and some do not even support RISC-V. Luckily, all of the core components required for CTRL did have RISC-V support. It is important to note that the Linux Foundation have also produced a course on porting software projects.

Conclusions

So, how hard is it to port a Linux-based OS to a RISC-V target?

The answer for us was: trivial. A single Codethink engineer completed all three stages described above in under a week. The move to RISC-V has never been easier. Although hardware may still be under development and some software dependencies may not yet support RISC-V, there is considerable value in adding support for a new ISA sooner rather than playing catch-up later.

Understanding Codethink's IEC 61508 Mapping for the Eclipse Trustable Software Framework

2026-05-29T00:00:00+01:00

Codethink's mapping between the Eclipse Trustable Software Framework (TSF) and IEC 61508 shows how the methodology designed specifically for complex software system integration and modern continuous delivery workflows can be used to demonstrate compliance with the standard. We have achieved a process functional safety assessment of the Trustable Software Framework against IEC 61508, conducted by exida specialists in the assessment of software development tools for safety-critical real-time systems. Our goal is that software engineers will adopt the TSF methodology to record and organise compliance evidence across the full software lifecycle, whether they are building new software or integrating pre-existing components into safety-critical systems.

While resources like CASS provide assessment templates to guide auditors through mapping evidence to the standard’s clauses, Codethink’s mapping is designed for software engineering teams. We believe it’s the first independently assessed, openly available methodology designed for software integration at scale and speed. The mapping is available as an early access release while Codethink completes the functional safety assessment of CTRL OS, after which it will be contributed to the Eclipse Trustable Software Framework project.

What IEC 61508 Requires for Pre-Existing Software

IEC 61508 is the sector-agnostic base standard for functional safety of electrical and electronic systems. Sector-specific standards derived from it include ISO 26262 for automotive, IEC 62061 for machinery, IEC 62304 for medical devices, and EN 50128 for rail. Each adapts IEC 61508's framework to the particular demands of its domain, but the foundational requirements trace back to it.

The software part of IEC 61508 is written from the perspective that you are developing software for a defined purpose: you specify what that software must do, understand how it contributes to a system with particular safety goals, and build your assurance argument from that foundation. However, this creates a problem for open source software as it is typically intended for anyone to use in any way they like, licensed as ‘not fit for any particular purpose’, with no warranty, and no claim of quality. That determination is left to whoever integrates it. IEC 61508 recognises this reality and provides a specific Route 3S for pre-existing software that was not developed with a safety application in mind.

Route 3S requires organisations to provide evidence across a defined set of categories for software that was not developed for their specific safety application, including:

Software safety requirements specification
Justification for use
Design
Integration with the hardware
Verification and validation
Interference from functions not used by the safety function
Credible failure mechanisms and mitigations
Configuration and runtime environment
Assumptions of use

In doing so, it provides a structured basis for the evaluation of pre-existing software based on the same criteria used for purpose-built safety-critical software. The principles underpinning Route 3S are shared by other standards addressing pre-existing software, including ISO/PAS 8926.

The mapping Codethink has published addresses those requirements directly, covering Routes 1 and 3 of IEC 61508 Edition 2.

What the Eclipse Trustable Software Framework Is

The Eclipse Trustable Software Framework (TSF) is an open source methodology, hosted at the Eclipse Foundation. Originated by Codethink through its initial project contribution, and actively supported by Codethink as a continuing maintainer, TSF provides a structured, measurable foundation for building software that can be trusted by OEMs and regulators alike. The framework is domain-agnostic: it is suited to any software-intensive system, not only Linux-based or open source systems or IEC 61508 applications.

TSF begins with a set of Tenets and Assertions. These are concise, verifiable statements about what must be true for software to be considered trustable. There are six Tenets, each defining a distinct area of evidence:

Provenance captures where the software came from, what its dependencies are, and what is claimed about them, supported by mirrored dependencies, supply chain attestations, and regular reassessment of upstream risk.
Construction defines how the software is built, installed, and run reproducibly, demonstrated through hash-verified, hermetic builds in controlled environments that can be reconstructed and independently verified.
Changes tracks how the software is updated without breakage or regression and how upstream fixes are absorbed, maintained through CI-gated merge policies, vulnerability triage, and systematic tracking of upstream changes.
Expectations document what the software must and must not do, independently of any particular implementation, grounded in defined behaviours, identified misbehaviours, and runtime monitors.
Results establish whether the software does what it must and does not do what it must not, with test results linked directly to the specific Expectations they address in the Trustable Graph.
Confidence brings together the reasoning about how confident we are in the software based on all of the above, including where uncertainty remains and why, maintained as an explicit, living account alongside the code.

The methodology is git-native. Statements and Artifacts live alongside source code in version control. The intent behind engineering decisions can often be lost in document-driven processes, which is why the TSF keeps it current alongside the code itself. When a developer makes a change that affects an existing Statement, the continuous integration pipeline flags it for review before it can be merged, keeping the assurance argument aligned with the software at every step.

Because the review happens inside the tools and workflow that engineers are already using, compliance is integrated into the workflow, rather than it being a separate exercise. TSF is designed to make compliance something engineers see as valuable to what they do, rather than a process imposed on top of their work.

John Ellis, "Rebuilding Trust: From Open Source to Open Accountability," OCX 2026

What the IEC 61508 Mapping to the TSF Covers

The mapping is Codethink's interpretation of how the TSF's Tenets and Assertions address the objectives of IEC 61508 Routes 1 and 3, translating each area of evidence into the standard's requirements. For example, the TSF category of Expectations documents how the behaviours, misbehaviours, indicators and constraints are derived and specified for software, and how these are traced to supporting evidence, which satisfies the IEC 61508 objectives for software safety requirements.

The mapping covers a standard-to-framework layer. An organisation that uses it will need to build on it to construct a compliance argument for a specific product and context, using the mapping as a practical, independently assessed starting point.

To understand what this looks like in a working system, consider how it operates day-to-day in our CTRL OS project. When a developer makes a change to the build environment, checking the Trustable Graph immediately surfaces any linked Statements that need review. The developer confirms the change does not invalidate them, marks them reviewed, and commits. Committed changes are reviewed through daily change control processes via tools such as GitLab, with safety-related changes signed off by a safety expert. The safety argument stays current with the code, providing a practical method for achieving continuous compliance.

Create Your Own Mapping Between the TSF and a Compliance Standard

Codethink's goal is for the Eclipse Trustable Software Framework to become the common engineering basis on which organisations across industries demonstrate alignment with the safety, security, and regulatory standards that are relevant to them.

For a long time, organisations have tried to navigate safety compliance in siloes, solving similar problems confidentially and separately. By making this mapping openly available, Codethink is taking a stand against that opacity. The vehicles, medical equipment, and industrial systems that depend on safety-critical software are used by everyone, and the reasoning behind why that software can be trusted should be open to scrutiny. That is why Codethink is inviting the industry to build on this work. The same process of mapping the TSF to IEC 61508 can be applied to other compliance standards: ISO 13849 for machinery safety, DO-178C for airborne systems, IEC 62443 for industrial security, or the medical device regulations applicable in your market. An open, community-maintained portfolio of mappings between the TSF and the world's most important safety and regulatory standards makes a shared foundation for reasoning about software trust at scale possible for open source projects, OEMs, and regulators. If you want to contribute to making software trust an open, shared endeavour, the first step is understanding what an independently assessed TSF mapping looks like in practice.

Request the early access IEC 61508 mapping here, review it, and reach out to discuss building one for the standards that matter in your domain.

Resisting Hyrum's Law with Private Constructors in Python

2026-05-07T00:00:00+01:00

Hyrum's Law States:

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

However, by hiding implementation details and providing stable interfaces for operations your users want to do, the "sufficient" number can be increased.

Python doesn't support completely hiding implementation details but has a convention that names beginning with a single leading underscore (e.g. a function called _create) are classed as implementation details and that the library developer makes no stability guarantees.

Unfortunately, there are features that require implementing special methods with fixed names that can't be used to determine whether they are public.

__init__, the constructor special method, is a noteworthy challenge since the Python data model requires this method to always exist.

Why Might You Want to Make the Constructor Private?

You might want private constructors if:

Your object should only exist when returned from another object's method.

class Ref:
    def __init__(self, ...) -> None:
        ...

    @property
    def name(self) -> str:
        ...

    @property
    def version(self) -> str | None:
        ...

class Document:
    def get_element_by_name(self, name: str) -> Ref:
        ....

Your object can be constructed from different types and want to manage your API surface with named constructors instead, and there's either no "natural" constructor or you want to have all constructors named for consistency.

Supposing we have a class representing a config file that can be constructed in a number of different ways, you could have an overloaded constructor where the parameter determines how to build it like this:

from io import TextIOBase, TextIOWrapper
from pathlib import Path
from typing import BinaryIO, TextIO

class MyConfigFile:
    f: TextIO

    def __init__(self, f: str | Path | int | BinaryIO | TextIO) -> None:
        if isinstance(f, str):
            f = Path(f)
        if isinstance(f, Path):
            f = f.open()
        if isinstance(f, int):
            f = open(f)
        if not isinstance(f, TextIOBase):
            f = TextIOWrapper(f)
        self.f = f

    ...

This form of overloading is useful when you've already committed to having an API based on the passed in parameter, but with foresight you might find named constructors more manageable.

from io import TextIOWrapper
from pathlib import Path
from typing import BinaryIO, Self, TextIO

class MyConfigFile:
    f: TextIO

    def __init__(self, f: TextIO, /) -> None:
        self.f = f

    @classmethod
    def from_binary_file_object(cls, fobj: BinaryIO) -> Self:
        return cls(TextIOWrapper(fobj))

    @classmethod
    def from_file_descriptor(cls, fd: int) -> Self:
        return cls(open(fd))

    @classmethod
    def from_path(cls, path: Path) -> Self:
        return cls(path.open())

    @classmethod
    def from_name(cls, name: str) -> Self:
        return cls(open(name))

    ...

In this example there is a natural default constructor that wraps a pre-existing text file object, but exposing that constructor as API is a leaky abstraction that commits the implementation to wrapping the text file object.

You might find that the config file is rarely opened so lazily opening it on-demand would be a potential optimisation that is made more difficult by the constructor requiring an already open file.

So for consistency and API management it would be better to have another from_text_file_object named constructor.

Your object may be constructed from the same types but the values may be interpreted differently so require named constructors and there may not be a natural default constructor.

datetime has two named constructors for creation from a timestamp. datetime.fromtimestamp(timestamp) will create a datetime in the local timezone while datetime.utcfromtimestamp(timestamp) will create one in Coordinated Universal Time.

As with the constructor overloading example above the datetime constructor exposes that the internal representation is separate attributes for the year, month, day, hour, minute, second, microseconds and time zone.

This has the same problem of making it hard to change the representation if it would be more convenient to operate on time as a single number.

The only way to create an object may be expensive and a constructor may imply it's cheaper than it is.

The natural constructors of the last two examples are pretty cheap, they assign passed in values to the object (though datetime does some validation).

An object representing the contents of some remote file may be represented as:

from urllib.parse import SplitResult as Url

class RemoteFile:
    origin: Url
    contents: bytes

Since the only way to construct it is to fetch the contents of the URL it might feel natural to fetch in the constructor.

from urllib.parse import SplitResult as Url
from urllib.request import urlopen

class RemoteFile:
    origin: Url
    contents: bytes

    def __init__(self, url: Url):
        self.url = url
        with urlopen(url.geturl()) as f:
            self.contents = f.read()

However if RemoteFile(url) implies the object is cheap to construct then it may be accidentally called in a tight loop and cause performance problems in a way that is not obvious.

This would be more obvious with a named constructor.

from urllib.parse import SplitResult as Url
from urllib.request import urlopen

class RemoteFile:
    origin: Url
    contents: bytes

    def __init__(self, url: Url, contents: bytes):
        self.url = url
        self.contents = contents

    @classmethod
    def fetch(cls, url: Url) -> Self:
        with urlopen(url.geturl()) as f:
            return cls(url, f.read())

However this then means that the constructor is public and objects could be created directly, potentially causing confusion from the contents not matching what could be fetched from the URL.

How Do I Make My Constructor Uncallable Then?

Uncallable is a relative term in Python, but there are two important things we can achieve:

Misuse of an object you shouldn't have created.
A warning from tooling that you shouldn't use the constructor.

So How Do We Prevent the Object Being Created?

Conventionally NotImplementedError is raised from methods in Abstract Base Classes to indicate that the subclass is incomplete if the method is not overridden, which is similar enough to "this method should not be called".

class MyClass:
    def __init__(self):
        raise NotImplementedError("Don't use the MyClass constructor")

    @classmethod
    def _private_constructor(cls, foo):
        ...

Of course this then makes the constructor unusable for all users, so how do we implement our private constructor?

Private Constructors With Private Tokens

One approach is to require that the constructor requires a special value be provided and that the value is explicitly marked as private.

This value is sometimes called a sentinel value due to their historic use as a special value at the end of a sequence to identify that it's over e.g. the NUL byte at the end of a C string.

A historically popular way of creating these in python is to use the object() constructor since it creates a unique value that is different from any other value that can be created.

_PRIVATE_TOKEN = object()

class MyClass:
    def __init__(self, foo, *, _private_token):
        if _private_token is not _PRIVATE_TOKEN:
            raise NotImplementedError("Don't use the MyClass constructor")

        self.foo = foo

    @classmethod
    def _private_constructor(cls, foo):
        return cls(foo, _private_token=_PRIVATE_TOKEN)

Since _PRIVATE_TOKEN is explicitly private, and so is _private_constructor the user can't unintentionally create the object incorrectly, and will discover they've created it wrong when they test it.

However, this additional parameter check is extra overhead that could cause worse performance in a tight loop.

How Can I Create Objects Without Using `init`?

__init__ is not the only special method that plays a part during object construction.

When you use an object's constructor such as our MyClass, the general process is:

self = MyClass.__new__(MyClass)
MyClass.__init__(self, foo)

So there is a separate step for creating an empty object of the appropriate type and initializing it, and __init__ is expected to call the constructors of its parent classes.

Which means if we don't want to use the constructor we can do this ourselves.

class MyClass:
    def __init__(self):
        raise NotImplementedError("Don't use the MyClass constructor")

    @classmethod
    def _private_constructor(cls, foo):
        self = super().__new__(cls)
        self.foo = foo
        super().__init__(self)
        return self

So now the constructor is unusable but we have an explicitly private constructor that we can use to implement our public named constructors or functions or methods on other objects that might want to return our object.

Getting Tooling to Warn

You may have read this and thought "why don't we just document that the constructor shouldn't be used" and it's a good idea to do that, but time is finite and developers are typically working under pressure so will reach for what looks like the most obvious thing to do and the documentation may only be read after it doesn't work as expected.

Developers know this about themselves though, so invest in tooling like IDEs, test suites, pre-commit check hooks and CI pipelines to catch issues.

Type annotations allow IDEs and static type checkers like mypy allow library developers to annotate their types and functions to support additional checking before the code is evaluated.

Here is our previous solution with type annotations:

from typing import Self

class MyClass:
    foo: str

    def __init__(self) -> None:
        raise NotImplementedError("Don't use the MyClass constructor")

    @classmethod
    def _private_constructor(cls, foo: str) -> Self:
        self = cls.__new__(cls)
        super().__init__(self)
        self.foo = foo
        return self

As you can see there's nothing indicating that __init__ should not be called in the type system.

Now that we are discussing annotations, it may have caught your attention that we have an annotation for functions that can only raise an exception or infinitely loop called NoReturn and it may be tempting to annotate the constructor with it.

However this will confuse mypy.

from typing import NoReturn, Self, reveal_type

class MyClass:
    foo: str

    def __init__(self) -> NoReturn:
        raise NotImplementedError("Don't call the MyClass constructor")

    @classmethod
    def _private_constructor(cls, foo: str) -> Self:
        self = cls.__new__(cls)
        super().__init__(self)
        self.foo = foo
        return self

reveal_type(MyClass)  # Revealed type is "def () -> Never"

Other type checkers like ty may have different results, but constructors are expected to return None and there's other ways to accommodate them.

NoReturn is a bottom type indicating a value that can't exist.

There's also the Never¹ type that represents unreachable code, a placeholder in generic types for states that don't exist or parameters to functions that cannot be called, which is precisely what we want to do with our __init__.

from typing import Never, Self

class MyClass:
    foo: str

    def __init__(self, *, _never: Never) -> None:
        raise NotImplementedError("Don't call the MyClass constructor")

    @classmethod
    def _private_constructor(cls, foo: str) -> Self:
        self = cls.__new__(cls)
        super().__init__(self)
        self.foo = foo
        return self

MyClass()  # error: Missing named argument "_never" for "MyClass"

So we must provide the _never parameter, it's indicated to be explicitly a private parameter and there's no value you can pass that satisfies the type checker².

MyClass(_never=object())  # error: Argument "_never" to "MyClass" has incompatible type "object"; expected "Never"

This doesn't prevent the constructor from being called at runtime, but the raised exception will handle that.

What Are My Alternatives if This Doesn't Work for Me?

The simplest alternative is to rename your whole class with a leading underscore (e.g. _MyClass) to indicate that it is private but that will result in confusing documentation about whether it should be used.

Another, which follows a common pattern in Java, is a private implementation of a public interface.

Python doesn't have Java-like interfaces, but it does have Abstract Base Classes (ABCs) (and supports Java interface-like behaviour by allowing multiple inheritance) which raises an exception when attempting to create an object when the class doesn't implement all the required methods and support run-time checking of whether an object is the required class.

Private construction can then be implemented by only making the ABC part of your public API and give the subclass that implements the ABC a private name like this:

from abc import ABC, abstractproperty

class MyClass(ABC):
    @abstractproperty
    def foo(self) -> str:
        ...

class _MyClassImpl(MyClass):
    def __init__(self, foo: str) -> None:
        self._foo = foo

    @property
    def foo(self) -> str:
        return self._foo

def make_myclass(foo: str) -> MyClass:
    return _MyClassImpl(foo)

The run-time check that the class has been implemented correctly may be a useful safeguard, but it introduces additional run-time overhead that may not be acceptable.

typing.Protocol is a newer alternative that supports static (or optionally run-time) duck-type checking of an object's behaviour without inheritance.

As a Protocol it would look like this:

from typing import Protocol

class MyClass(Protocol):
    foo: str

class _MyClassImpl:
    foo: str

    def __init__(self, foo: str) -> None:
        self.foo = foo

def make_myclass(foo: str) -> MyClass:
    return _MyClassImpl(foo)

Conclusion

I hope this advice has been interesting and helps you learn a new technique that you may find applicable in other circumstances.

If you are unfortunate enough to only be able to use an old version of Python and unable to use third-party modules then NoReturn is equivalent to Never and can be used instead. ↩
Strictly you could use MyClass(_never=cast(Never, object())) to force it to pass type checking since it's not foolproof, but you would have to be a mighty fool indeed to do it and it'll get caught at runtime. ↩

FOSDEM 2026

2026-01-27T00:00:00+00:00

FOSDEM week is back.

At the end of January, Codethink will once again be in Brussels, this time with more than 40+ Codethings speaking, volunteering, and presenting across FOSDEM itself and a set of pre-FOSDEM events and conversations that lead into the weekend.

From 31 January to 1 February 2026, FOSDEM will bring together the people who build, maintain, foster the open source community. We're looking forward to a few full days of deep technical talks, devrooms, hallway conversations, and long-overdue catch-ups. Just as importantly, the days leading into FOSDEM are where many of the most interesting discussions begin, and Codethink will be there too.

With the release of embedded-hal v1.0 almost two years ago, Rust is now well suited for bare metal embedded development.

This talk focuses on low cost RISC-V microcontrollers such as the CH32V003, in the same space as boards like the Raspberry Pi Pico or Teensy but costing under €1.30, and shows how to build an embedded system using bare metal Rust. It covers the full workflow from getting to main, to flashing and debugging with probe-rs, IDE integration, and using no-std to achieve the smallest possible footprint.

Talk: Generating SBoMs for BuildStream projects

Speaker: Abderrahim Kitouni

Track: SBOMS and supply chains

Room: UD2.208 (Decroly)

Date: Sunday 1st February

Start: 16:30

End: 17:00

Link here

Description:

BuildStream is a software integration tool that allows building software aggregated from multiple sources in a single pipeline to produce a final output. This final output could be a container image, an operating system image or anything that you can write a plugin for.

This talk presents buildstream-sbom - a tool that extracts information from a BuildStream project and uses it to generate an SPDX-formatted SBoM. It will also discuss some of the issues in translating from BuildStream concepts to SPDX.

Can't make it to Brussels this year?

If you won't be in Brussels, FOSDEM offers livestreaming for talks throughout the weekend - full details are available on the FOSDEM website. Recordings from related pre-FOSDEM events hosted by the Eclipse Foundation and OpenForum Europe will be published in the weeks following their respective events.

After the dust settles, we'll share a post-FOSDEM write-up with highlights, themes, and takeaways from the week. Until then, you can find more information about the events Codethink is sponsoring and participating in on our events page.

Building on STPA: How TSF and RAFIA can uncover misbehaviours in complex software integration

2026-01-23T00:00:00+00:00

Codethink Trustable Reproducible Linux (CTRL OS) integrates open-source software components to provide a stable but leading-edge platform for safety-critical designs.¹ We use the Eclipse Trustable Software Framework (TSF) to continuously quality-assure CTRL OS as it and its components evolve, and present this case study to show how TSF’s RAFIA methodology allowed us to close a feedback loop, by detecting and remedying defects in an upstream open-source component and, crucially, our initial analysis of that component.²

Setting the scene

TSF sets out a number of generic and high-level Trustable Assertions: testable statements which a robust software integration project should aim to fulfill through analysis and evidence. For this case study, we consider the TA-MISBEHAVIOURS assertion, which reads: “Prohibited misbehaviours for [CTRL OS] are identified, and mitigations are specified, verified and validated based on analysis”.³ TSF is not prescriptive, and instead encourages projects to think critically about their own unique circumstances and therefore choose appropriate methods to achieve and validate this.

TSF proposes, but does not require, RAFIA (Risk Analysis, Fault Induction and Automation) as an alternative process model for the development of complex safety-critical software systems.⁴ RAFIA is deliberately a cyclical and continuous process, making it well suited to TSF’s goal of continuous compliance, and setting it clearly apart from the V-model traditionally applied in these situations, where validation of test results against the original design may be a one-off activity.

Source: https://codethinklabs.gitlab.io/trustable/trustable/extensions/rafia/index.html

The RAFIA process has four ordered but cyclical stages: Analysis, Specification, Implementation, and Testing. Critically and uniquely, the Analysis stage involves analysing the results of testing from the previous iteration and using those results to re-analyse the system’s design to inform test specifications for the next iteration. For the first complete iteration of a RAFIA loop, however, an initial risk analysis of the system must be undertaken, typically without reference to test results. RAFIA strongly recommends that projects use the STPA methodology to enumerate the behaviours that the system should exhibit, and misbehaviours that should be either absent, or detected and mitigated. The results of this analysis are then used to drive testing of the system.

STPA (System Theoretic Process Analysis) is a technique for analysing complex systems and identifying modes of failure arising from ‘unsafe’ interactions between the components, which may involve otherwise correct behaviour from components as well as errors or faults. STPA is highly systematic in nature, and requires practitioners to sequentially identify:

The scope of their system
Losses (or harms) that the system should prevent
Hazards that might lead to a loss occurring
Interactions between the system’s components
How each interaction may be “unsafe” and lead to a hazard
Constraints that must apply to the system and its components to prevent or mitigate hazards

TSF provides guidance for using STPA in the Risk Analysis stage of RAFIA, and the STPA Handbook describes the procedure and its rationale in greater detail.⁵, ⁶ Yet, for the RAFIA procedure, its most important output is a set of constraints. We can close the RAFIA loop by writing a test specification to check these constraints are true for a real system, implementing that test specification, and then analysing the results of the tests to determine whether the STPA model is flawed or if our software misbehaves. In a TSF context, as STPA constraints are testable statements, they can be represented and continuously scored in a Trustable directed acyclic graph (DAG) for every iteration of the software, just like the overarching goal of TA-MISBEHAVIOURS itself.

Case study: integration of the Safety Monitor

For the integration of Codethink’s open-source Safety Monitor – a safety application that controls an external ‘mitigation of last resort’ – into CTRL OS, we chose to follow the RAFIA process to be certain that the mitigation is always triggered when the OS enters an unsafe state, but never triggered while CTRL OS is in a safe state.⁷

At the start of the RAFIA process, our safety engineer undertook an initial STPA analysis which started by considering the scope of the Safety Monitor system and its components. In summary we have:

Watchdog: a device external to CTRL OS which transitions the system to a safe state (for example, by transferring responsibility to another system, or simply rebooting) if it does not receive input (a “pet” signal) every 100 ms
Critical Processes: programs running on CTRL OS to provide the customer’s safety-related function(s)
Safety Monitor: a program running on CTRL OS which receives updates from Critical Processes (called Extrinsic Indicators) and sends pet signals to the Watchdog every 100 ms only if:
- All received Extrinsic Indicators are within predefined numerical bounds
- All expected Extrinsic Indicators have been received
- No internal corruption or malfunction as been detected

Detecting a defect in the software

Having taken our system requirements and system components, we followed the STPA procedure and eventually generated a list of testable constraints, amongst them Constraint 1: If any of the monitored Extrinsic Indicators are not updated at the specified cadence and frequency, the Watchdog-petting thread exits.

Continuing with the RAFIA process, we wrote a specification to test Constraint 1 on real hardware running CTRL OS both in pre-merge testing (to assert the constraint is still met for each proposed change to CTRL OS) and post-merge soak testing (to assert the constraint is still met for CTRL OS’s releases and mainline branch over thousands of repetitions). As part of this, we applied the next part of RAFIA: Fault Injection. This involves deliberately causing the misbehaviours that are identified by STPA, to determine whether the system either prevents them, or detects them and responds appropriately.

Implementing the specification produced a test in which a mock Critical Process was configured to not send the expected Extrinsic Indicators to the Safety Monitor, and the Safety Monitor’s logs were checked to see if the Watchdog petting ceased, and if so, the time that took. In a real system, the Watchdog would trigger an appropriate response at the system level, such as restarting a misbehaving component, but our tests in this case measure how long the system takes to respond at all when an unsafe condition is reported. However, when we came to analyse the results of the soak tests (around 500 samples each week), we were surprised to see aberrant response times from the Safety Monitor.

According to Safety Monitor’s published User Documentation, system integrators can configure the Timeout for an Extrinsic Indicator, after which the Watchdog petting action will stop if no such Extrinsic Indicators are received.⁸ Having consulted the upstream documentation and CTRL OS’s Safety Monitor configuration files, we expected this timeout to be 135 seconds for the Constraint 1 test case. When soak test results reported a mean timeout of 117 seconds for the Constraint 1 test case we started to investigate.

In the implementation of the Constraint 1 test case, the mock Critical Process’s configuration was temporarily overridden so that Extrinsic Indicator reporting would cease; this involved restarting the Critical Process. Following conversations with upstream Safety Monitor engineers and a joint interrogation of the test logs, we realised that Safety Monitor maintained an internal buffer of Extrinsic Indicators, did not empty this buffer when a Critical Process restarted, and actually calculated the Timeout starting at the last Extrinsic Indicator in the buffer rather than when it detected a restart of the Critical Process. As the system had been running in a “happy path” state before the test case induced the fault, the state of this buffer was not certain at the start of the test, so neither was the point at which Safety Monitor would start counting the Timeout or the measured response time itself.

The CTRL OS team raised an issue in the open, which was triaged, implemented, and tested by the upstream team.⁹ As a result, Safety Monitor now clears Extrinsic Indicator buffers when a process restarts, the measured response times in these scenarios are now consistent and predictable, and the expected behaviour is clearly documented upstream. This was verified through soak tests against a CTRL OS image containing a patched Safety Monitor.

In this way, the RAFIA process detected and resolved an obscure misbehaviour in an upstream software component of a complex system, but, more importantly, it also provided documentation so that future STPA analyses in the RAFIA loop will consider this buffering behaviour.

Detecting a defect in the analysis

While methodically inspecting soak test logs to determine the reason for the unexpectedly short response times, we noticed messages such as safety-monitor: Adding pause indicator, which prompted the CTRL OS team to check Safety Monitor’s upstream documentation and source code for an explanation. This revealed that an Extrinsic Indicator can be “paused” by any process sending the requisite message to the safety monitor.¹⁰ Moreover, Safety Monitor enabled this pausing behaviour for all monitored Extrinsic Indicators by default, meaning that if a rogue process A somehow sent Safety Monitor the “pause” signal for a critical process B, then B could have stopped providing Extrinsic Indicators altogether without petting ceasing.

We identified this behaviour as a pathway to a potential dangerous failure of CTRL OS, where a fault reported by a Critical Process was not mitigated externally. However, this failure mode was completely absent from the initial STPA. We handled this in two ways: firstly, according to the RAFIA process, we revisited our STPA analysis with the new information about pausing from the tests, and considered this as a Control Action to be systematically analysed according to the methodology. Secondly, we raised a further issue upstream so that pausing had to be explicitly enabled in CTRL OS’s configuration of Safety Monitor, allowing us to later argue through our TSF graph that new constraints around unsafe “pause” control actions were satisfied.¹¹ Here, we see that the thorough test data analysis that RAFIA mandated for one STPA Constraint revealed a different gap in the STPA itself, allowing immediate refinements to the model and neatly closing the development loop with Risk Analysis.

Conclusions

Historically, integrators have emphasised the "Theoretic” element of STPA, so when gaps in the analysis open up in the face of engineering reality, the analysis is re-done from first principles. As we have shown, TSF’s RAFIA process brings together safety, testing, and upstream expertise so that data (even anecdotal) from tests – with real hardware and software in the loop – can immediately be used to readjust the expectations of analysis and design, eliminating uncertainty early without necessarily starting again, just as software itself may evolve by incremental steps.

To find out more about how TSF, RAFIA, and CTRL OS can help you nail risks in complex software projects, please get in touch at connect@codethink.co.uk.

Adding big‑endian support to CVA6 RISC‑V FPGA processor

2025-10-23T00:00:00+01:00

Introduction

Big‑endian RISC‑V is an interesting area of experiments, and Codethink has previously demonstrated booted big‑endian RISC‑V Linux in QEMU. Building on that, we now achieved booting big‑endian Linux on an FPGA board. To do this we used the CVA6 open-source processor, which we investigated in one of our previous projects. CVA6 has its source code available on GitHub, written in SystemVerilog. It implements the RISC‑V instruction set and can target a few FPGA boards. We modified its code to add support for runtime configurable endianness. Read on to find out how we did it!

RISC‑V big‑endian specification

Let's start with RISC‑V specification. The endianness of the RISC‑V processor is controlled by the {M/S/U}BE bits in the Control-and-Status-Register mstatus. For example, if the processor is operating in M privilege mode and bit 37 of mstatus is set then the processor operates in big‑endian mode. Each privilege mode (M/S/U) endianness can be set individually. It’s also valid for SBE and UBE to just be mirrors of MBE, which is what we chose to implement. One interesting point of the specification is that processor instructions are always little‑endian, regardless of the current endianness mode.

How does different endianness look in practice? When CVA6 is operating in little‑endian mode, then everything happens as normal. But once the MBE bit is set in mstatus, then the processor switches to big‑endian mode, and all loads and stores to the memory are treated as being in big‑endian format. But because the CVA6 core still carries out operations in little‑endian format, we need to byte swap the data to convert between the two.

Assembly example

This is a simple assembly program that demonstrates changing endianness at runtime:

.section .data
.balign 32
    /* Variable stored in LE format */
    var1: .word 0x11223344

.section .text
.global _start

_start:
    /* Some initial vars */
    li        a1, (1 << 37)
    la        a2, var1
    li        a3, 0x1

    /* Add 1 to var1, keep the result in a4 (LE) */
    lwu a4, (a2)
    add a4, a3, a4

    /* Switch to BE */
    csrs mstatus, a1
    fence

    /* Add 1 to var1, keep the result in a5 (BE) */
    lwu a5, (a2)
    add a5, a3, a5

    /* Expected outcome:
       a4: 0x11223345
       a5: 0x44332212  */

Now let's see which CVA6 units we had to change.

Load/store unit and MMU

We started with load/store unit and added endianness flag and byte swapping there. An important point is that load/store operations can have different sizes – 8, 16, 32, and 64 bits – and they have to be byte-swapped differently. For example, 8‑bit sized operations don't need to be modified at all. Another place where memory accesses happen is memory management unit's (MMU) page table walker, so we added byte swapping there too. This was easier because we just replicated what we'd done for the load/store unit.

Atomic memory operations

Big‑endian support for atomic memory operations (AMOs) is more involved than for load/store unit, because in CVA6 AMOs are encoded as memory bus transactions. These transactions are then processed in another module called axi_riscv_atomics. AXI is the specification of the memory bus that CVA6 uses. Conveniently for us, AXI specifies an endianness flag as part of the transaction, it's just CVA6 doesn't utilise it. Having learnt that, we modified code that sends these transactions to set the endianness field depending on the current endianness mode, and we added code to the axi_riscv_atomics module to handle that flag correctly. axi_riscv_atomics module does AMOs calculations in its own arithmetic logic unit (ALU), so our changes there were mostly byte-swapping input and output of the ALU.

Testing

CVA6 supports GDB debug, the process of doing that is described in the documentation. So, for testing our changes we used GDB, with which we ran our small assembly programs on the processor and validated their results. We discovered that GDB doesn't connect to the processor when it's in big‑endian mode, so at the end of our test programs we switched the mode back to little‑endian to work around that.

In parallel we modified the CVA6 SDK to build Linux for big‑endian system. For that we mostly reused our work from a previous project, where we ran big‑endian Linux in QEMU. We published our CVA6 SDK modifications here. You can build it with make BE=1 FS=1 images, and then flash it with sudo -E make BE=1 FS=1 flash-sdcard SDDEVICE=/dev/sdX, where sdX is your SD card.

After successful big‑endian Linux boot, we celebrated by playing Tetris there :)

Conclusion

We modified CVA6 to allow runtime endianness switch and demonstrated big‑endian Linux running there. This demonstrates possibility of big‑endian RISC‑V systems and makes experiments on them easier.

You can read more RISC‑V articles from our blog here.

Are you looking for help with your RISC‑V projects? Get in touch with Codethink's team of experts by writing to us at connect@codethink.co.uk.

Bringing up a new distro for the CVA6 RISC‑V FPGA processor

2025-10-09T00:00:00+01:00

In this blogpost we will cover how to bring the CVA6 processor up with Linux on it and how we replaced the OS there with a different one, based on Freedesktop SDK, updating the kernel version on the way.

CVA6 is being developed by the OpenHW Group, part of the Eclipse Foundation, of which Codethink is a member. We thought it would be useful to see how we can combine our approach to building software with the processor developed by a fellow Eclipse Foundation member.

CVA6 has its source code available on GitHub, written in SystemVerilog. It implements the RISC‑V instruction set and can target a few FPGA boards. We found the best FPGA board with CVA6 support was the Digilent Genesys2, so we selected that to get going.

We also had a lot of experience with using Freedesktop SDK as a platform for building a Linux OS, so we wanted to try to run it on CVA6. Freedesktop SDK allows you to easily get reproducible builds, and be as close to upstream as possible.

During this project we managed to build and deploy a new OS, and solved a few open issues along the way. We've proposed updates and fixes to the CVA6 SDK. For QEMU, we proposed an initial CVA6 implementation with ethernet support. And we've published the build instructions for our OS on GitHub at CodethinkLabs/cva6-sdk-bst, which you can build yourself using BuildStream – more on that below. Read on to find out how we did it, and what we'd like to happen next!

Generating and flashing the bitstream

The configuration data that's loaded into an FPGA is called bitstream, and all FPGA vendors have their own software suites to generate bitstreams. Despite being an open-source processor, CVA6 requires one of these proprietary tools to generate its bitstream. There's an open-source alternative called Yosys, but it's lacking some functionality of commercial tools.

We are targeting a Xilinx FPGA, so we have to use the Xilinx's software suite called Vivado. CVA6 requires Vivado 2018.2 version to compile, which in turn requires old Ubuntu to run. To work around that we used a container. You can find instructions on how to install Vivado 2018.2 in a container here.

We also need to compile RISC‑V GCC toolchain following instructions in the CVA6 repository.

Now that we have working Vivado and RISC‑V GCC, we can generate the bitstream from the CVA6 repository as follows:

source /path/to/Xilinx/Vivado/2018.2/settings64.sh
LD_PRELOAD=/lib/x86_64-linux-gnu/libudev.so.1 CC="$RISCV/bin/riscv-none-elf-gcc" make fpga

The LD_PRELOAD is to work around this issue. The $RISCV variable comes from toolchain installation.

The next step is flashing the bitstream to the board. Add the udev rule to your system and then follow the flashing instructions.

After flashing you should see zero-stage bootloader output on the board's UART port. It tries to load something from the SD card, but since we haven't inserted the card yet, the bootloader just waits and does nothing.

Building cva6-sdk

Now let's generate an SD card image for our processor. The official way of doing it is to use cva6-sdk, which can generate SD card image with OpenSBI, U‑Boot, and Linux with busybox initrd. To do that, run make images in the cva6-sdk repository. After that insert an SD card into your computer, and run sudo -E make flash-sdcard SDDEVICE=/dev/sdX, where sdX is your SD card.

Now insert the SD card into the Genesys2 board, and power it on. On the UART port you should see booting logs and eventually get to the busybox shell. You can play tetris there if you run /tetris :-)

We now have Linux running on our processor, even though the kernel and userspace versions are somewhat outdated.

Using Freedesktop SDK userspace with old kernel

On our way to full Freedesktop SDK OS, we first aimed at an intermediate step where we wanted to use cva6-sdk's bootloader and kernel but replace busybox initrd with our rootfs.

We modified cva6-sdk to use a separate partition for rootfs and not include initrd in the kernel – this allowed us to easily use Freedesktop SDK based rootfs. To generate it we've written a few BuildStream “elements”, which are .bst files with YAML syntax that describe how to produce various software, in our case – rootfs image. As part of doing that we replaced default Freedesktop SDK kernel headers with the older ones from cva6-sdk, and disabled some profiling functionality in binutils, to fix a compilation error.

Our first attempt to boot the new rootfs failed because systemd was timing out waiting for the serial device to appear. It was a problem with the slow boot process – by the time systemd-udev-trigger.service started, it was too late, the serial device systemd unit had already stopped waiting. We solved this by increasing the default device timeout. We also disabled a few non-critical systemd services as they were taking a very long time to start. This is something to revisit later.

After doing that we were able to log into Freedesktop SDK userspace!

Using new kernel

With Freedesktop SDK userspace working, it was time to update the kernel. cva6-sdk used Linux v5.10, which is an LTS kernel with support until 2026, but we wanted the newest stable release – v6.14 in this case. Using newer version brings us better performance and some new features.

The first problem we encountered was the larger kernel size – U‑Boot's bootcmd wasn't reading enough data from the SD card, so we increased that amount.

The kernel started booting now but was triggering kernel panic at a later stage. We found out that it was happening because of the memory protections set up by OpenSBI. Turned out in newer kernel versions Linux doesn't reserve the first 2MB of memory for OpenSBI as it used to. Adding a reserved-memory node to the device tree fixed this issue.

Then we saw systemd failures. On startup it printed an error message about being unable to set its environment, and frozen execution. Also, the error messages were truncated. Unclear how to debug that, we turned to git bisect, with which we found that the errors were happening when Linux was doing unaligned memory accesses. When we disabled them with CONFIG_RISCV_EMULATED_UNALIGNED_ACCESS, the problem went away. We didn't get to the bottom of it, so it's something for future TODO.

Now we have the new kernel working. \o/

We were lacking some functionality though – cva6-sdk had a patch for v5.10 kernel with the board's ethernet driver, so we wanted to port it to v6.14. The porting itself was straightforward – fixing compilation errors, and one runtime error. We noted that the driver itself was poorly written and ideally needed to be rewritten from scratch, but we decided not to do that at the moment, so this is another thing for future TODO. Despite that, the ported driver worked and allowed us to use ssh to connect to the board.

You can find the BuildStream project with all our .bst files and instructions on how to use it to generate an SD card image there – https://github.com/CodethinkLabs/cva6-sdk-bst.

QEMU model

In parallel, we developed a QEMU model that simulates the CVA6. It was useful during initial kernel bring up, because with QEMU we could iterate quicker. Also having a working QEMU model should help in future developments for CVA6. The patches for QEMU were submitted upstream – 1, 2.

Contributing back to cva6-sdk

We wanted to contribute something back to the cva6-sdk, so we replicated our kernel work there and also updated the software versions in their initrd. This is the merge request with the updates – https://github.com/openhwgroup/cva6-sdk/pull/109. Creating it also required replicating everything we've done with 64 bit CVA6 to 32 bit CVA6, as cva6-sdk is common for both widths – this replication was straightforward though.

An interesting fact is that during testing the updated cva6-sdk version with tetris we noticed that it always generated the same shapes. After some investigation, it was obvious that we had a bug in the floating point unit. We confirmed that it was present before our updates and was not caused by us. And then we discovered that it was just fixed in the master branch of CVA6. Regenerating the bitstream and using the latest CVA6 version fixed this bug with tetris.

Conclusion

We demonstrated how to boot Freedesktop SDK on CVA6, created QEMU model of it, and updated the kernel and initrd in cva6-sdk. This brings the software on CVA6 closer to upstream and makes experiments on it easier.

Potential future work includes updating U‑Boot version, debugging why some systemd services time out, rewriting the board's ethernet driver, and debugging unaligned memory access issue.

Are you looking for help with your RISC‑V projects? Get in touch with Codethink's team of experts by writing to us at connect@codethink.co.uk.

Externally verifying Linux deadline scheduling with reproducible embedded Rust

2025-10-01T00:00:00+01:00

Exploring how Codethink used embedded Rust to measure Linux’s scheduling performance with an external clock, while maintaining bit-for-bit reproducibility of the testing firmware.

Before choosing a Linux-based operating system on a multi-core processor for a safety-critical task, we must first be certain of its real-time capabilities. These can be demonstrated through repeated long-duration “soak” testing of a running system, but such results are less valuable if all timing measurements are referenced to the system’s own clock. To help demonstrate the real-time scheduling capabilities of Codethink’s Trustable Reproducible Linux-based Operating System (CTRL OS) we employed custom embedded Rust firmware on an external microcontroller, with its own clock, to monitor the timings of Linux processes. Moreover, as our Trustable Software Framework (TSF) stipulates, the Rust toolchain lets us compile that firmware reproducibly and in a controlled environment, just like any other testing tool.

The Linux kernel provides the SCHED_DEADLINE mode to allocate CPU time to threads with hard real-time constraints running on the same processor core¹. SCHED_DEADLINE threads declare their mode and constraints to the kernel with the sched_setattr syscall, specifying 4 important parameters²:

Period: how often the kernel should provide the thread with its runtime, in ns
Deadline: the time in each period by which the thread must have finished its execution for that period, in ns
Runtime: the thread’s expected worst-case execution time in each period, in ns
RECLAIM flag (optional): whether the kernel should provide the thread with unused CPU time in a period if the thread hadn’t finished its execution

Using the earliest deadline first algorithm, the kernel then balances CPU time between SCHED_DEADLINE threads such that each meets its own scheduling parameters. If it would be impossible to allocate CPU time to a new thread without violating its own or existing threads’ requirements, the kernel rejects the sched_setattr call. SCHED_DEADLINE thus provides a theoretical basis for Linux as a powerful, open-source, and real-time operating system family.

However, to put this into practice, and work towards certifying Linux’s real-time scheduling as a safety function of CTRL OS, we must evidence that a set of SCHED_DEADLINE threads can be successfully scheduled on real hardware repeatedly, over many thousands of hours of soak testing. Codethink provides the userspace program rusty-worker for this purpose. rusty-worker has two threads: a SCHED_DEADLINE “worker” thread, and a SCHED_OTHER “monitor” thread. (SCHED_OTHER is the default scheduling mode for Linux threads, subjecting them to round-robin, not deadline, scheduling.) rusty-worker’s monitor thread initially configures the sched_setattr parameters of the worker according to user configuration. It then periodically reports the worker’s measured scheduling parameters, specifically runtime and period, to the systemd journal for post-test analysis and Codethink’s open-source safety-monitor running on the same system to trigger runtime mitigations³⁴.

rusty-worker’s SCHED_DEADLINE worker thread shares CPU time with other threads. It declares its period, deadline, and worst-case runtime to the kernel with the sched_setattr syscall, and the kernel ensures that it gets all its runtime before each period’s real-time deadline by balancing with other threads. Note that the rusty-worker’s real runtime and the position of that runtime in the period may differ between periods, however execution should always finish before the deadline expires.

The safety-monitor relies on rusty-worker’s extrinsic reports of scheduling period and runtime to make safety decisions (e.g. should the system be restarted by allowing a watchdog timer to expire?), and CTRL OS’s engineers rely on the post-test reports to investigate regressions and remedy them. If there were a problem with the device-under-test’s clock, a rusty-worker’s true runtime and period might differ from what it reported, without safety-monitor or engineers being any the wiser. To guard against this, we modified rusty-worker to send signals on a UART serial port when it started and completed execution for a period. As the UART protocol is, by definition, asynchronous, a device on the other end of the serial port can time these signals and derive its own external measurement of scheduling parameters without relying on the device-under-test’s clock: the time between a started and completed signal is the runtime, and the time between two started signals is the period.

We chose a Raspberry Pi Pico as our external measurement device. It is mounted on a custom printed circuit board with a second Pico, which acts as a probe to flash the first Pico with firmware before each test, and a logic-level shifter so signals from both RS232 and 3.3V level UART devices can be analysed. The Pico is based on the RP2040 ARM Cortex-M0+ microcontroller, and has first-class support for embedded Rust, without the overhead of an operating system. We thus set about writing Rust verification firmware (vfrs) for the Pico, utilising the rp-hal (Hardware Abstraction Layer) crate to build up an interrupt-driven architecture to provide accurate timings of signals on the UART line, and hence report its measurements of rusty-worker’s runtime and period on the device-under-test over USB to a CI runner⁵. The CI runner aggregates the external measurements with rusty-worker’s reports to the systemd journal, performing pairwise matching of measurement sessions, before uploading to an OpenSearch data lake for statistical analysis.

The custom PCB used for external scheduling measurement in CTRL OS’s test rigs. The left-hand Pico U1 serves as a Debug Probe, flashing the right-hand Pico U2 with a controlled build of vfrs in each test⁶. The right-hand Pico connects to the CTRL OS device-under-test via an RS232 UART (J3) or a 3.3V UART (J4), and the test runner via USB.

We chose Rust for vfrs despite the well-documented and extensive C/C++ SDK provided by the Raspberry Pi Pico’s manufacturers⁷. Our reasoning for this was threefold: firstly, HALs take advantage of Rust’s type and reference systems to ensure illegal hardware configurations and memory-unsafe operations are rejected at compile-time, not runtime, giving us more certainty about the Pico’s stability and accuracy over hours of soak testing. The only code exempted from the Rust compiler’s memory safety checks must be presented in explicitly marked unsafe code blocks, alongside human-written safety argumentation and justification.

Secondly, as the Rust code needed to bootstrap interrupts, USB and UARTs on a Pico is more verbose than the equivalent C, less information is hidden from engineers and safety assessors about how we use the RP2040’s registers and peripherals, giving more certainty around its exact behaviour and timing accuracy. For example, as we needed to calculate the Pico’s clock frequency (125 MHz) in code, we realised that our methodology would be inappropriate for the fastest rusty-worker processes with runtimes of hundreds of nanoseconds; this equates to just hundreds of Pico reference clock ticks.

Lastly, we chose Rust as its embedded toolchain can be used in a controlled build system just as easily as its desktop toolchain. CTRL OS is developed in accordance with the Trustable Software Framework (TSF), which asserts that “All tests for CTRL OS, and its build and test environments, are constructed from controlled/mirrored sources and are reproducible”⁸. We achieved this by defining vfrs as an BuildStream element integrated into the CI image for CTRL OS’s tests⁹. This element specifies vfrs’s source code, dependency crates, build commands and freedesktop-sdk’s Rust toolchain, with minimal patching to add the thumbv6m-none-eabi target and specify an LLVM rather than GCC backend. Since we build vfrs like any other testing component, we can make the same guarantees about its provenance (asserting the origins of source code, dependencies, build environment, and hence the final binary) and bit-for-bit reproducibility. We achieve this without the ARM toolchain wrangling associated with the C equivalent, such as handling vendor-specific compiler versions, nested Git submodules, and build scripts which can download arbitrary binaries from the Internet at compile-time.

We perform weekly analysis of the external and rusty-worker scheduling measurements stored in OpenSearch, both in version-controlled Jupyter notebooks and automatic validators to provide scores for CTRL OS’s Trustable compliance report¹⁰. With CTRL OS and rusty-workers running on an ARM64 Rockchip RK3588-based test system, we found exceptional correlation between rusty-worker and external measurements of runtime and period, with matching distributions and summary statistics to within 10 µs. These external measurement soak tests gave us great confidence in the many millions of samples of scheduling metrics collected in other soak tests on the same hardware. Furthermore, as external measurement requires that a rusty-worker’s worker thread interacts with a TTY serial device, we predicted and observed improved scheduling performance for the entire system when CTRL OS ran with the PREEMPT_RT kernel configuration option enabled¹¹. This was because, with PREEMPT_RT, the SCHED_DEADLINE worker thread may be interrupted (or pre-empted) while it works through the notoriously non-deterministic spin locks of the kernel’s TTY layer when sending signals to the Pico.

A histogram showing the mean elapsed runtime of a rusty-worker as measured by the rusty-worker itself and externally by vfrs on a Pico. Elapsed runtime plotted here is the time between the “started working” and “work completed” signals from rusty-worker, rather than the runtime specified in sched_setattr. These signals may have been sent in different blocks of runtime in the same period if the thread was pre-empted, so we see two peaks in the distribution: one at 710 µs without pre-emption, and one at 780 µs with pre-emption. Importantly though, the orange rusty-worker distribution maps well to the blue vfrs distribution.

In summary, external measurement, powered by embedded Rust firmware, allowed us to robustly evidence the real-time scheduling capabilities of a Linux environment, while maintaining tight control of test environment provenance and stability.

If you are interested in Codethink’s CTRL OS, Trustable Software Framework, or build engineering and embedded software expertise, then please get in touch at connect@codethink.co.uk.

https://www.kernel.org/doc/html/latest/scheduler/sched-deadline.html ↩
https://www.man7.org/linux/man-pages/man2/sched_setattr.2.html ↩
https://www.man7.org/linux/man-pages/man8/systemd-journald.service.8.html ↩
https://gitlab.com/CodethinkLabs/safety-monitor/safety-monitor ↩
https://github.com/rp-rs/rp-hal ↩
https://github.com/raspberrypi/debugprobe ↩
https://www.raspberrypi.com/documentation/microcontrollers/c_sdk.html ↩
https://codethinklabs.gitlab.io/trustable/trustable/doorstop/TA.html#ta-tests ↩
https://buildstream.build/ ↩
https://codethinklabs.gitlab.io/trustable/trustable/trudag/validators.html ↩
https://dl.acm.org/doi/10.1145/3297714 ↩

Engineering Trust: Formulating Continuous Compliance for Open Source

2025-09-22T00:00:00+01:00

Software has an increasingly important role to play in business, security and safety-critical applications. The complexity of the systems and hardware involved in these applications is also growing rapidly, and open source software is increasingly used in these contexts.

Established formal engineering approaches are not keeping up. They are often slow, difficult and costly to scale when applied to complex software and systems, and are not widely or consistently applied to open source software. They are often not intended, or not suitable, for managing rapid changes or continuous updates.

Requirements are a fundamental element in all of these established approaches, but requirements are broken. High-level requirements are a wish list; stakeholders just make a list of features. Low-level requirements are a narrative; developers just describe what is implemented.

Formal process models see the dynamic nature of FOSS projects as a problem, but this can be a key part of their value. However, because requirements are broken, the intent and expectations that inform the software are often lost in the noise.

So, how do we address these issues?

Codethink’s Paul Albertella and Kaspar Matas outline how the Eclipse Trustable Software Framework (TSF) — a lightweight continuous compliance framework, designed as and for FOSS — can help you to do just this.

Watch the talk to discover:

An overview of TSF and examples of how it is applied
How to organise and evidence your own objectives, including the intent and expectations that inform the software you are working on
How the framework is used to manage automated, transparent and traceable bodies of evidence, which can be quantified by a confidence score
How the results can be mapped to functional safety standards, to support certification and ongoing assessment.

Got your attention? Watch the talk in full below:

Interested in learning more about how to apply TSF to your project? Complete the form below and a member of the team will be in touch with you.

Email Please input your company email address.

Accept our Privacy Policy

Yes

Accept our Privacy Policy

Learn more about our Privacy Policy here.

By clicking submit, I agree to receive email updates from Codethink Ltd about our latest products and services.

Why Renting Software Is a Dangerous Game

2025-07-29T00:00:00+01:00

Software is never a temporary feature, especially in the fast-moving world of embedded systems. In industries like automotive, software is at the core of the product’s function, but time and time again, companies treat it like a short-term asset. It’s a common practice to license third-party components without proper planning for long-term ownership of the software.

But fast-forward 15 years and it is a situation ripe for disaster. The vendor might not exist anymore, or could be acquired and fully disassembled for parts. Then what? How can a car with unmaintained software stay on the road safely? It’s a dangerous game to rent your software. The best way to guarantee long-term maintainability? You have to own it.

Ownership = Control, Not Just IP

In software, ownership isn’t just about having intellectual property and rights to the product. When licensing software without access to the source code or perpetual rights, you have essentially put a timer on your product. Think about it: You are placing your bets that the vendor will (1) still be around, (2) be willing to support you, and (3) still be selling the same or a compatible version of the software 15 years down the line.

In other instances where projects are taking a longer time in the development phase, you also face the possibility of the vendor no longer existing when your product is ready to go to market. The vendor could get acquired by a competitor who refuses to license or support the product you’re hoping to back. Without access to the source code, you’re forced to go back to the drawing board to re-engineer.

15 Years Later

Focusing on the automotive world, many of the large markets legally oblige car manufacturers to support their software for 10 years after the last unit rolls off the production line. Considering development, certification, and testing time, these 10 years might turn into a 20-year timeline for full cybersecurity support.

Imagine when year 12 comes around, and your product faces a critical security vulnerability. The problem: you don’t own the software, and the vendor refuses to cooperate. What comes next? You can’t patch or rebuild and could even be forced to re-engineer the entire ECU. This includes changing the different hardware components to accommodate for the new OS. You’re looking at a product recall-level issue or, worse, having no option to buy back the remaining fleet.

With inflation, the cost of fixing or rebuilding these systems years later would also be considerably higher than if you had control from the start. Ownership doesn’t just mitigate risk but addresses cost predictability in the long term.

The Toolchain

Another often-overlooked risk is development tooling. You might license a proprietary application with no issues, but what about the compilers, SDKs, or development environments used to build and maintain it?

For example, could Creative Cloud Photoshop open a file made in Photoshop 1.0? The answer is ‘sort of’. While it may be able to open the older file format, this file likely contains features and tools not supported in the new version, meaning it wouldn’t be fully functional. Now imagine trying to rebuild 15-year-old software with tools that are no longer supported or that require new licences just to install.

When Does Proprietary Make Sense?

Like in the case of bleeding-edge technology, there are occasions where proprietary software is the only viable option as the products might only be available through a commercial provider. Still, there are options to negotiate access to the product’s source code under defined conditions. For example, some companies place code in escrow, with events like bankruptcy allowing the customer to take control if needed.

Unfortunately, a lot of companies assess this problem when it is too late. Getting ahead would be increasingly helpful in sectors like automotive where the scale of software continues to increase.

Where Does FOSS Stand?

FOSS thrives in this environment. The use of open technologies helps avoid vendor lock-in, while gaining traceability, and long-term maintainability. When problems arise, you have the source code and the right to modify it. In a FOSS setting, you’re also part of a community where there is a sustainable ecosystem that looks to maintain, patch, and evolve the software shared.

Questions Every Company Should Be Asking

To protect your future, ask these questions at the start of every project:

Will I be able to run this software exactly as-is in 15 years’ time?
Will I be able to make changes quickly if needed?
How hard will those two things be when time and pressure are working against me?

If you can’t confidently answer all three then you don’t own your software, you’re renting it.

Own Your Software, Own Your Future

Owning your software is about more than access. Think of control, sustainability, and resilience over time. If you lease every component of your stack, you’re betting that your vendors will still be around in 15 years, still offering the same support, and still aligned with your needs. That’s rare.

Futureproofing does not start at the top. It starts with the engineers, architects, and procurement teams that need awareness, tools, and authority to make long-term decisions today that protect the business tomorrow. Choosing open technologies, negotiating source access, and asking the right questions early on can mean the difference between a quick patch and a product recall 12 years after launch.

If your product needs to last 15 years, your software must last just as long. Start owning your software today. Talk to Codethink about building long-term, trustable systems that stand the test of time.

Linux vs. QNX in Safety-Critical Systems: A Pragmatic View

2025-06-17T00:00:00+01:00

In the ongoing debate between Linux¹,² and QNX,³,⁴ much of the noise comes from theoretical positioning and outdated assumptions—not from real-world engineering experience. It’s time to set aside the marketing spin and focus on what matters: architecture, performance, scalability, and long-term viability in production environments.

At Codethink, we are deeply invested in open source software, and we’ve seen Linux outperform alternatives in most safety-critical use cases. That said, QNX can offer specific advantages under certain conditions. This post outlines the key architectural, performance, and ecosystem trade-offs—grounded in production experience, not ideology.

1. Are Linux and QNX Even Comparable?

Not exactly. While both are used in embedded and safety-related systems, they’re typically not deployed in the same architectural roles. QNX aims for predictability and simplicity, while Linux is designed for performance, flexibility, and scale. Each comes with trade-offs.

2. Architecture: Microkernel vs. Macrokernel

QNX uses a microkernel architecture, where OS services (e.g., file system, networking) run as separate userland processes. This can offer fault isolation but introduces latency from inter-process communication.

Linux, as a macrokernel, integrates these services within the kernel itself. The result: significantly faster internal communication and higher throughput—critical in systems that demand responsiveness.

3. IPC and Real-Time Behaviour

QNX’s blocking IPC model supports tight real-time behavior by reducing priority inversion. However, this model introduces overhead when porting POSIX software, which is emulated using userland helpers.

Linux favours non-blocking, POSIX-native IPC designed for throughput and scalability. While this can introduce jitter in extreme cases, real-world latency is typically lower—and its scheduler is deeply optimized for modern CPU topologies.

4. Performance: Consistency vs. Throughput

QNX offers more consistent latency, but poor hardware utilisation and limited scalability.

Linux delivers vastly higher throughput, supported by advanced scheduling, isolation, and containerisation techniques, allowing critical systems to be prioritised deterministically.

With proper configuration, Linux can provide real-time performance under load—making the “Linux isn’t real-time” narrative outdated.

5. Scheduling and Resource Isolation

Linux goes far beyond simple priority queues:

Real-time scheduling modes (FIFO, round-robin, deadline) are widely used.
Cgroups and namespaces enable fine-grained CPU, memory, and IO partitioning.
Resource isolation lets critical functions maintain access without resorting to hypervisors.

This richness is crucial for mixed-criticality systems where multiple workloads share hardware.

Linux delivers verifiable confidence in real-time scheduling performance.

While many real-time operating systems—including proprietary ones—claim support for widely used scheduling algorithms (e.g., FIFO, round-robin, deadline), Linux stands apart in one critical respect: transparency. Because Linux is open source, its implementation of these algorithms is actively studied, verified, and continuously improved by a global technical community. This visibility fosters greater confidence that the real-world behaviour of the system aligns with theoretical guarantees. This assurance is difficult, if not impossible, to obtain with closed-source alternatives. This ability to independently verify system behaviour is a material strategic advantage in safety- or performance-critical environments.

6. Legal and Operational Realities

The conversation isn’t just technical—it’s strategic. Consider:

QNX tooling and SDKs are rarely perpetually licensed. In 15 years, can you still rebuild your product?
Linux is open, auditable, and vendor-independent. Most BSPs are public, and support is available from hundreds of organizations.
QNX binaries are opaque. No source, limited debugging capability, and long lead times for fixes.
Certification is contextual. An OS with a certificate means nothing if your integrated system isn't re-validated.

And finally—QNX has changed hands multiple times. There is no assurance it won’t end up owned by a competitor.

7. Common Misunderstandings

Let’s clear up some persistent myths:

Myth	Reality
Microkernel = safer	Faults in user apps—not kernel design—cause most issues
Closed = secure	Open source fixes vulnerabilities faster and more transparently
Linux isn’t real-time	It is—with higher throughput and flexible isolation
Vendor BSPs = better	Often less maintained and lower quality than open-source equivalents
Certification = ready	Pre-certified components still require full system validation

Conclusion: Choose Based on Outcomes, Not Labels

Both Linux and QNX have their place—but they are built on different philosophies and deliver different outcomes. Linux offers unmatched scalability, flexibility, and control—critical in environments where performance and long-term autonomy matter.

QNX may suit niche applications where extreme determinism is required and system complexity is tightly bound. But for modern, software-defined, connected systems—Linux is the platform of record.

If you're building for the future, trustability, flexibility, and computational throughput will define your competitive edge. And that’s where Linux leads.

Want to learn more about this topic?

Reach out to the Codethink team, using the form below:

Email Please input your company email address.

Accept our Privacy Policy

Privacy Policy & Consent

Learn more about our Privacy Policy here.

By clicking submit, I agree to receive email updates from Codethink Ltd about our latest products and services.

Is Rust ready for safety related applications?

2025-05-30T00:00:00+01:00

In late 2022, Codethink and exida showed that almost 90% of C (and C++) safety-related bugs were mitigated in some fashion by switching to Rust. We argued that the safety community must justify its continued use of C or C++ if not switch to Rust. Three years on, let's revisit what was said, and look at what has changed.

Is Rust ready for safety related applications?

You can download the slides and then the below will make a little more sense. The below is a reconstruction of content, which was delivered in person, from my slide notes, since unfortunately the video of the presentation no longer appears to be available. The reconstruction is written looking back at the deck, so if you happen to have been present at the symposium and remember the talk, just bear that in mind.

Abstract

As of October 2022, it was getting increasingly difficult to ignore the impact and capabilities of Rust. Rust brings a number of advantages, as a language, to the world of safety related application software. At the time, a commercially supported compiler was already available with support for several devices used in automotive projects.

Our goal on the day was to talk a little about Rust, and to hopefully convince the delegates to consider using it in their safety applications. We proposed to discuss a little about options around linking to existing C libraries, and porting code from C to Rust; as well as exploring the stated advantages of Rust as compared with C, and to not shy away from some of the new difficulties that might have arisen for programmers starting new projects in Rust or converting existing codebases.

Finally we explained that we hoped to give the delegates a basis on which they could start to construct an argument around using Rust, either alone or in conjunction with C, in their safety applications.

What do we do with C today

In order to make C suitable for use in safety applications we had to surround ourselves with large amounts of rules, cross-checks, guidelines, and tooling. There's the MISRA rules, AUTOSAR, if you get to C++, even more MISRA, AUTOSAR, and even JSF rules. Then there were tool qualification suites, including exida's ctools suite; coding guidelines, static analysis, etc. All of these tools were wrappered up with architecture guidelines, application specific tooling etc.

All of these things were, or are, valuable, and all of them address, in some part, things which the language is unable to help you with.

But this is, I'm sure you'll agree, a lot to deal with as we consider writing our arguments for using C in a safety context.

Background to Rust

Rust 1.0 was in late 2015, and it has been considered 'stable' in a backward-compatibility sense since then. The language's capabilities grow year on year, through a process called RFCs and MCPs, and though the rate of change is relatively high, the Rust project's commitment to compatibility tempers that somewhat.

Target support improves rapidly, once the compiler can build for something, library ecosystems catch up very quickly. Often wrappering C libraries until native Rust ones are written, and at the time it was evident that the Linux kernel was moving toward allowing Rust drivers to exist in-kernel.

The ecosystem is well documented with a number of 'books' around the core ecosystem, and every published library having documentation available in one place (docs.rs). The "official" language specification was the Rust compiler codebase itself; however there were efforts to write properly standardisable specifications, and indeed recently Ferrous systems donated their specification to the Rust foundation.

Contribution to the ecosystem, and getting support, was (and remains) similar to typical open source projects - there are community forums, Github or other code forges, and some consultancies offering Rust specific help. Governance was mostly handled by small teams within the community, though there is a Rust foundation tasked with stewarding the ecosystem and the language.

Core Rust

At the time, change to Rust was typically managed via the RFC or MCP processes where discussion is had to design things. Changes were then made in a way that is feature-flagged so that normal users are not affected by them. After changes were considered good enough, there would be a stabilisation proposal and if that goes through, the change becomes live for all users. This process remains largely unchanged to today, though there are more nuances than we wanted to go into in the presentation slot. There has been a lot of effort put into helping with this process, including documents such as Proposing a change to the language written up by the Rust language design team.

Rust libraries (and some programs), which are called crates, are published to a central package repository called crates.io used by both people and tooling to acquire them. All crates uploaded to crates.io are automatically documented on docs.rs.

The Rust packaging and build tool cargo uses crates.io by default to acquire dependencies for projects (though this can be overridden for software supply-chain management reasons). Cargo then builds your dependencies and your whole project managing the process for you. Testing is also generally handled, at least initially, via Cargo and Rust's built in testing.

But as an additional benefit to crates.io - there is a Rust service called 'Crater' which can run the entire crates ecosystem through a proposed compiler change allowing changes to be verified against a near-complete corpus of open-source Rust code. This leads to even more confidence in the stability of the tooling and ecosystem.

What makes Rust special

The primary thing which draws people to Rust is the fact that the language forces developers to consider the lifetime of data and data access. This, and a few other aspects of Rust's design allows it to make claims about safety in the sense of certain aspects of memory /safety such as data races, use-after-free, etc. which can plague other languages. Rust calls this the 'borrow checker' and it's effectively part of the early stages of the compiler. Lifetimes are a part of the Rust type system, so Rust's expressive type system is there to help you to write correct code.

Because the authors of Rust's standard library are aware of all the places that there could be problems, due in no small part to the Rust language helping them, there are many ways in which the standard library helps developers to do the right thing. One aspect of that is that all functions which could do something dangerous are more likely to panic rather than do the dangerous thing (such as indexing out of the bounds of an array). Panics are a controlled mechanism for dealing with such behaviour at runtime. Better, of course, is the APIs in Rust which make explicit the fallibility of such operations, requiring that the author of the code deal with such problems.

Outside of the language and the standard library, there are a number of pieces of Rust's tooling which really help - for example Rust compiler error messages try and act like a pair programmer, doing their best to assist the developer in fixing their problems - as mentioned above, there are runtime protections such as controlled panics which developers can take advantage of, or control the existence of, at their pleasure; and there is an ecosystem-provided extra-linting tool called clippy which helps with spotting bad idioms and offering fixes to the developer.

What do beginners struggle with

Newcomers to Rust often face a number of struggles depending on their past experience (or lack thereof) in Software. Rust is essentially a procedural language, with some fundamental rules around data lifetime and access lifetimes, with a behavioural modelling abstraction which it calls Traits - this forces a change in thinking around software design since most traditional design processes are object-oriented.

Sometimes Rust is forced on newcomers because of existing projects, but in this space, typically people come to Rust because of the promises it makes about helping programmers to not create certain classes of bugs in their code. Rust is low-level enough that we should consider using it in almost any safety application where we'd be using C "by default" and high-level enough that it offers many benefits once the learning curve is surmounted.

Because Rust is a low-level systems language at its heart, its interoperability with C code is very good; however there are also tools out there which help with converting C code into Rust which can be used as a starting point to understand how to write software in Rust. But do not let this lead you to false confidence, even very low-level Rust code feels distinct from C code and will require your engineers to rethink their approaches.

Rust is one of the recent neo-languages (at the time of the presentation it was only 7 years old and has only recently celebrated its 10th birthday) and as such it has a significant amount of modern tooling around it. This can cause a lot of confusion in engineers used to a more traditional C, C++, CMake/GNU Make environment. The neo-languages were, at the time, not at the same maturity of ecosystem support as the old guard, and as such newcomers also struggle with understanding which library is best to use; or what devices are supported well, which can complicate any decision to use Rust in a project.

What else is hard in Rust

While the compiler refusing to build bad code is more helpful than simply dealing with a runtime canary, the shift in cognitive load is not in a direction developers are used to, which can cause difficulty as new engineers come to Rust. Rust trades a lot of the pain of debugging something when a bug becomes apparent, for the work required to express to the compiler that your design is sound. This is very much something which many engineers are not used to. The corollary to this point is that not all syntactically valid Rust is semantically correct and no matter how helpful a compiler is, you cannot be certain you have written the right thing. There is a middle part to the Rust learning curve where developers will tend to assume that because the compiler said something was OK, that it was in fact correct. Overcoming this overconfidence in the tooling and returning to a disciplined approach to requirements-driven testing can be hard for newcomers who are experiencing the power of the Rust language for the first time.

Perfectly valid Rust code may yet not be considered idiomatic - there is a linting tool called Clippy which can catch a number of errors in your code, but which also guides developers toward a style of coding which aims at reducing the cognitive load of comprehending code written by others. Learning to obey these tools can be difficult when, again, newcomers are faced with the power of Rust and old-guard programmers who are more set in their ways with design and even layout of code.

Understanding the runtime canaries can be a sticking point as people try to integrate Rust into their products - Rust's "panic" approach is a controlled unwinding of the program state which can be hooked into. It is possible to force your program to fail to build if panics are happening in certain areas of your code, and learning to code in a way which cannot induce panics is important, but it is a step along the way to Rust mastery which is quite far along the learning curve and can take quite a while to get used to. Ultimately though, no amount of compiler, linter, and ecosystem tooling, will help you to avoid systematic semantic errors in your code. Even with Rust around, your developers will still need the support of your requirements team and a comprehensive testing approach in order to make good progress.

Coexistence with other code (C)

At the time the talk was given, a significant effort had already been put into permitting Linux device drivers to be written in Rust and used in the Linux kernel - this was a huge step forward for Automotive developers who need to create device drivers for custom Automotive hardware and who want to take advantage of Rust's language benefits. This coexistence is made possible primarily because Rust is designed to be a systems programming language and as such has interoperability with C built in. The Kernel has significant numbers of very well designed interfaces which are explicit about lifetimes of objects etc. permitting reasonably simple, or at least comprehensible, bindings into the Rust world. - This interoperability is achieved by means of Rust's safe/unsafe dichotomy, and its FFI.

The Rust project talks about "safety" but it's not safety in the same sense as the delegates at the symposium would have thought of it. Rust makes a few very constrained claims about what it considers 'safe'. But because computers, and the real world, are not built around Rust's safety model, it still permits developers to express to the compiler that they have manually verified that the Rust safety guarantees are still met by the codebase. These two worlds, safe and unsafe, live together by means of very constrained boundaries at which the assumptions Rust makes can be asserted back to the compiler.

This is done by means of a keyword unsafe which does double-duty, marking both things which the compiler cannot be certain about on its own and will need human assistance to confirm rules are not broken, and also by marking the places where a human has done the work to constrain the "unsafety" and reestablish the constraints to permit the compiler to continue to make the assumptions it needs in order to support the developer properly.

With all of the above, developers can then use Rust's ability to reference symbols in C code (or export symbols with C linkage) by means of Rust's FFI. FFI is inherently unsafe, and so your developers will need to understand and appreciate all of the safety rules before engaging in this kind of development. The Rust ecosystem provides documentation around safe/unsafe and the FFI to help you to develop this expertise.

As you'd expect, the Rust ecosystem provides tooling to assist with managing all this, there's cargo geiger which is a tool to determine how much of, and what kind of, unsafe code exists in your software and its dependency tree. And there's cargo miri which can actually interpret the intermediate form of Rust within the compiler and can be used to check for undefined behaviour at runtime (typically via your test suite), allowing you to assert the correctness of the second class of unsafe keyword usage mentioned above.

Testing in Rust

Rust's main build tooling (Cargo) comes with mechanisms for testing built into it. Unit tests are written as annotated functions in your codebase directly which tightly couples unit tests and the code they cover. Integration tests are written in separate source files alongside your codebase, in a tests directory; and any examples in your code documentation are automatically extracted, compiled, and run as tests, ensuring that documentation examples remain functional as you develop your software. The Cargo manual covers most of how these work; and the rustdoc manual covers code documentation testing.

In addition to the built-in testing framework, there is a rich ecosystem of additional testing libraries such as frameworks for mocking out parts of your design, benchmarking time-critical code sections, or fuzzing your software to help to ensure there are no crashes associated with unexpected or corrupt inputs. Each of these has good documentation to help you to use them in your project.

There is also a lot of other tooling out there to help. I mentioned miri briefly before, and it is there to help you run your tests in such a way that any undefined behaviour can be found and addressed; and there's Tarpaulin, a Cargo addon which assists with producing code-coverage reports to show if your tests are covering as much code as they need to in order to be confident of the codebase. Once again, documentation is a big thing in the Rust ecosystem and so there's plenty there too.

Finally you need to be able to test the tooling itself. As you'd expect, there is an extensive test suite for the compiler and core tooling built into their repositories and you could run all those tests yourself if you build your own Rust toolchain. Despite that, initiatives such as Ferrocene offer even more confidence and qualification should you require it. Whatever means you use to acquire your toolchain we recommend that you establish a baseline for confidence in the tooling yourselves.

C2Rust conversion

C2Rust is a project supported by DARPA. The C2Rust tool helps by transpiling C99 compliant code into Rust. It does not produce idiomatic Rust, nor necessarily nice Rust, but it does the syntactic conversion so that you can focus on the semantics of the software.

The Github repository contains instructions for downloading, building, and installing it on Ubuntu Linux, but there is a demo on https://c2rust.com/ which you can use to try things out on smaller pieces of code.

In terms of trust, you have the same problem you would have with any other transpiler or code generator - you need to develop trust in the tool for yourself. C2Rust explicitly states that they expect a human being to be involved in dealing with its output - it's the first step along a path to switching to Rust, not a silver-bullet.

What the tool does is take some C code, and effectively consumes unsafe C code, and produces the equivalent unsafe Rust code. This code should then be refactored and reengineered by your Rust engineers until the unsafety is isolated into places where it is necessary and well audited, and the rest of the code is in the Safe Rust language.

Tool support

The Rust community (often referred to as rust-lang) provides, free of encumbrance, all the tooling required to get going with Rust. The licences in use in the community are usually Apache 2.0 or MIT/BSD terms which permit use in proprietary situations. In contrast, organisations such as Ferrocene offer Long Term Support, qualified compilers, and other ISO 26262 and ASIL related assistance and tooling. Though there are costs associated with those.

Ultimately there is a question of how can you develop some trust in the tooling, and the community and ecosystem are working toward providing those assurances, though those of you familiar with Ken Thompson's seminal essay "Reflections on trusting trust" will be aware that this is a long and painstaking process. As with any switch in your approach to software in your safety applications; choosing Rust brings with it some amount of risk. Mitigations for those risks will vary depending on your particular applications, however the basic design and goals of the Rust language and core ecosystem are already mitigations for any number of C related risks you already carry.

Comparing ctools checks to stock Rust

In preparation for this talk, we went through the exida ctools repository, and considered around a hundred of the C test cases. ctools was presented by Michał Szczepankiewicz / Krystian Radlak / Piotr Serwa / Lukas Bulwahn at the exida 2020 Symposium Session 14: Tool-based code verification for C and C++ in dependable projects – systematic approach. In red you can see a snapshot of the uncaught language insecurities using 17 different C language static analysis tools.

Making the assumption that we do not want to use unsafe, unless it's literally impossible to demonstrate how Rust helps without it, we set about seeing how a default Rust toolchain does with these cases.

Rust, by construction, prevents many of the things checked for in the ctools suite such as use-after-free or leaked pointers. A full two thirds of the test cases are simply not compilable.
At runtime, Rust's controlled panics would catch a further 25% of the tests. With the use of the no-panic crate, the majority of these would be convertible to compile-time failures. Which could bring the "not possible by construction" percentage up as high as 90%
Three of the tests in the suite explicitly work with copying memory around, to do this we would need unsafe, but failure to use the correct copying semantics would likely be flagged by using MIRI to do testing around undefined behaviour.
There are a small number of things which Rust simply won't detect by default because they're defined clearly in things like the IEEE specification for floating point numbers. Further tooling work will likely be necessary to improve on this.
In most of the compile-fail cases, Rust gives a useful error message explaining why a certain pattern is not acceptable in the language. Rust's error messages tend to lead the way in the compiler world these days.

Given the spread of outcomes, we would not expect this distribution to have changed over the past couple of years.

ISO 26262 part six and Rust

This slide assumes reasonably intimate knowledge of ISO 26262. Since that is a proprietary document I cannot include it here for reference, so I'll summarise briefly.

If we take tables one, three, six, and seven from ISO 26262 part six as examples, we can look at each item in those tables and characterise them in terms of the Rust ecosystem and language. Many of the items in these tables are realistically things that your design process, review process and other human-related activities cover. These are not coloured.

A significant number of the items are either covered by built-in features of the Rust language or core tooling, or are at least human activities which can be supported by the tooling. As we would expect, there were still some aspects of ISO 26262 which needed tooling to be developed, however these are few and some tooling is already underway.

Briefly, table one considers modelling and coding guidelines, and everywhere Rust touches directly is covered nicely. Table three considers software design principles and while most of it is for humans, Rust at worst merely gets out of the way of the principles in question, and at best actively assists in following the principles. Table six is about software unit design and implementation. Rust gets much more involved here and other than one unit design principle (the one about each subprogram/function only having a single entry and single exit) Rust assists wholeheartedly. (Interestingly, because Rust encourages the use of the ? operator to neatly propagate runtime errors through the type system, it actively works against 1a.) Table 7 talks about unit testing. Tooling around data flow and control flow analysis has been developed since the talk, one example of which is Flowistry which is available as a VSCode plugin.

Is there an argument to use Rust

Up until this point in the presentation we were posing the question as "What argument can we make for using Rust in safety applications?", particularly given the experience and long-history of using C, and the effort invested into standards such as MISRA.

However, we'd much rather turn the question on its head, and that that point we did.

Is there an argument for not using Rust

Let's instead ask "What is your argument for not using Rust?"

There are some fairly simple reasons, such as AutoSAR MCAL availability, target availability or support tier; a particular C library you need for which wrappering would be inappropriate; or simply what batteries you get or do not get in a Rust distribution.

But - think also about what you actually need, in a lot of cases there is a good argument for originating the support you require, and contributing it to the now growing community of Rust users in the safety world.

And consider this - is the cost of wrapping a C library, or even rewriting it into Rust, greater than the cost of the risks associated with still using C.

Finally ask yourselves:- "What is my argument for using C? How can I justify the risks and dangers of continuing to work in C when there are alternatives like Rust to consider?"

The idea that the delegates ought to be justifying their use of C in their safety argumentation was one which very few at the conference had even considered before this point.

References

Several of the links are simply integrated above, however the rest from the slide deck are:

In addition, at the time there had been a Rust in adaptive AutoSAR presentation at the Elisa summit.

How has the landscape changed in the past years?

In the past few years, the fundamentals of Rust have not changed dramatically. Ecosystem support for various aspects of the language has improved and there are many new tools (some of which were mentioned above) which can be used to address aspects of concern the safety community may have had before. However the main shift has been in the understanding of just how poor of an idea it is to continue with using C (and to a lesser extent C++) without careful consideration of the risks.

There have been advances in possible C replacements such as Zig and possible updates/changes to C++ such as Circle and CPP2 though the latter two are very much bandaids/wrappers around the issues in C++. The C++ committees around safety and the language have worked on things such as a Safe C++ paper and C++ safety profiles though these also feel a little /like an addon rather than the paradigm shift which Rust provides.

The risks of C/C++ memory unsafety in particular continue to be talked about in increasing amounts by a more and more diverse set of people. For example, Sternum wrote in 2024 about memory safety from the perspective of IoT, and Runsafe wrote about memory safety vulnerabilities. While both of those articles are particularly focussed on cybersecurity, the close relationship between safety and cybersecurity means that they are relevant to us in the safety context as well. Furthermore, the continued evolution of sites such as the ISRG's Memory safety site point to the software community embracing the idea of shifting to languages such as Rust more and more as time passes.

When we then look at the safety community itself, we can also see a shift toward Rust. The Eclipse Kuksa project has embraced Rust as part of Eclipse SDV. The [ELISA][elisa] project recently held a workshop where our own Paul Albertella presented on the topic of the role of Rust in safety critical applications along with Daniel Krippner of ETAS.

Finally we would be remiss not to consider the direct approaches being taken to utilise Rust in safety critical areas which are emerging. The ongoing work on Ferrocene along with the Rust Foundation's Safety Critical Rust Consortium are good examples; the latter of which publishes the https://arewesafetycriticalyet.org/ website, the most useful page of which at the moment is probably the lists of members of the coding guidelines and tooling subcommittees. In addition, last year a paper about Bringing Rust to Safety-Critical Systems in Space was published.

Would Codethink recommend using Rust in safety applications?

Given the mention above about our talking about Rust at an ELISA workshop, it might be reasonable to assume that Codethink believes that Rust is ready for safety applications. In no small sense, we do believe that to be the case, but with reservations such as those we expressed a few years ago -- Rust won't be right for everyone or for every situation yet. There remains a lot of work to be done to ensure that the language specification is complete, then further work on the standard library and other pieces of the ecosystem before every traditional safety scenario could be comfortable with Rust.

Some of that work we have been involved with, and the rest we keep an eye on with enthusiasm.

However, we are already actively using Rust in our own safety work. In February, at FOSDEM, we open-sourced and spoke about our Trustable software framework (which is now an Eclipse project) as well as open-sourcing our Safety Monitor project which is written in Rust. This was a precursor to our announcement of a baseline safety assessment of Codethink's Trustable Reproducible Linux OS. We firmly believe that using Rust in the sort of situation where, rather than fully traditional ISO 26262-style safety, you are on board with the Trustable approach to safety, is completely viable and indeed desirable.

There is a long way to go, but where in 2022 we asked the question "What is my argument for using C? How can I justify the risks and dangers of continuing to work in C when there are alternatives like Rust to consider?" we now feel that the better question is "What is my argument for not choosing Rust." and since, for us, the answer to that question is "I have no good argument not to".

We choose Rust.

Download slides

The open projects rethinking safety culture

2025-05-16T00:00:00+01:00

In 2016, Codethink started out on a journey to discover how open source software can be safely used to build safety-critical systems — that is, in products where people might be harmed if the software fails to do its job correctly.

Free / libre open source software (FLOSS) projects like Linux have clearly demonstrated the value of collaboration in public when creating software that is — amongst many other things — trusted as the backbone of the web and millions of smart phones. FLOSS projects have also established the essential role of transparency and rapid software updates in dealing with cybersecurity threats. When it comes to safety, however, the difficulties of making a case for using FLOSS in a solution have long been a frustrating obstacle for product developers.

Immediately following Codethink’s announcement about our latest milestone in this journey, I took part in two workshops focussing on safety and open source. This gave me the opportunity to talk about the Trustable Software Framework (TSF) and how we are using it in our development of CTRL OS. I also learnt more from other open source projects about their approaches to creating software where trustability is as important.

The workshops were hosted by Volvo Cars in the Swedish city of Lund, and our hosts also provided several enthusiastic participants. The events were organised by two open source projects that have common goals and challenges, but approach these from different perspectives and with different focuses. The Eclipse SDV project aims to build an automotive software stack to provide “an open technology platform for the software-defined vehicle of the future”. In contrast, the ELISA project is concerned with the use of Linux-based operating systems for safety applications in a range of different domains.

→ Read: 'Codethink Limited Announces World’s First Baseline Safety Assessment for a Linux-Based OS to SIL 3 / ASIL D'

Day 1

Markus Bechter from BMW started the Eclipse SDV workshop by describing the approach to safety being developed for the Eclipse S-CORE or Safe Open Vehicle Core project. The intent is to establish a common set of development processes for components of this project, making the software amenable to safety certification using the ISO 26262 Automotive Safety Standard.

The Trustable Software Framework project was recently accepted into the Eclipse Foundation, so I gave the next presentation. TSF approaches the challenge of using FLOSS in safety more broadly: how can we make a case for using software that has not been developed following a process that conforms to an applicable safety standard? Since this describes the vast majority of existing FLOSS, including many of the tools and dependencies that S-CORE plans to use, an answer to this question is sorely needed, and TSF provides a methodology for making such a case.

After lunch, it was time to welcome a new set of participants and start the ELISA workshop. This began with an introduction to the project for newcomers (see my retrospective from last year’s workshop if you are also new to the project), followed by an Ask Me Anything discussion. Then we had a fascinating talk from David Cuartielles, a founder of the Arduino project who was recently honoured in the European Open Source Awards. After telling us about the latest Arduino (the Portenta x8) and the features of the boards that are relevant for trust, he went on to talk about a topic that he is passionate about: the DESIRE4EU project, which is exploring how to make printed circuit boards that are recyclable, in support of the European sustainable electronics goal.

The rest of the day focussed on the efforts of the ELISA Systems working group to describe and build systems involving Linux in combination with two other FLOSS components: the Zephyr RTOS and the Xen Hypervisor. This led naturally into a discussion of ELISA’s interactions with other adjacent open source communities.

Day 2

Philipp Ahmann and I started the second day with a discussion exploring some common misapprehensions about Linux and safety. We talked about some of the ‘routes’ to certification in the safety standards for pre-existing software, and why these are difficult to apply to open source software. We also explained why the notion of creating a ‘safe’ Linux is misleading, because safety can only really be understood in terms of a system, as opposed to an intrinsic property of a component. This led into discussions of various system models involving Linux, the use of complete redundant systems as part of a larger system design, and the role of hardware components in this, which was a perfect segue to the next session.

Olivier Charrier talked about the role of hardware integration in safety, describing how the responsibilities for achieving specific safety objectives as part of a system design are typically assigned to hardware and software components, and then refined or re-defined in a series of iterations to address the identified gaps. Alessandro Carminati then shared the results of a Linux Features working group investigation to build and analyse a minimal Linux configuration and identify a core set of features that must be considered for any Linux-based system.

After lunch we had a series of ‘special topic’ talks, beginning with interesting talks on PX4SPace — a flight control solution for drones that is being used to build robotic space vehicle solutions — and the SPDX Safety Profile, which extends the SPDX 3.0 ‘knowledge graph’ to include metadata relating to development processes for safety.

Håkan Sivencrona from Volvo then talked about Safe Continuous Deployment, emphasising the importance of building development processes that deliver an ongoing stream of ‘safe’ software deliveries using DevOps principles, not just one ‘blessed’ release that is never expected to change. Igor Stoppa’s talk on “Resilient Safety Analysis and Qualification” sparked a lively discussion, as he argued that any safety analysis of Linux must be based on a detailed understanding of the code, and that this might be a reason not to rely on more complex features or extensions of the kernel.

We then had a talk by Gustavo Padovan of the Kernel CI project, which recently became an associate member of ELISA. He explained that a key goal of the project is to enable projects and organisations testing the kernel to share their results with the wider kernel community by providing a common framework for reporting results. Recent developments include kci.dev, a command line tool enabling developers and maintainers to interact with Kernel CI, and a YAML config file format to enable Linux subsystems to share tailored test case executions for maintainers and the wider community.

The rest of the day focussed on requirements management and traceability, looking first at ELISA’s BASIL tool, and then at an initiative with the Linux Tracing subsystem to develop a low-level requirements specification approach. The latter involved documenting detailed requirements for each function in the kernel, which would be intended to support complete reimplementation of the functionality without reference to the code. One participant noted that this approach might enable the kernel to be re-written in Rust!

Day 3

I kicked off the last day by reprising my presentation about the TSF from the Eclipse workshop for the ELISA attendees. Once again, the enthusiastic engagement and insightful questions from the participants were very gratifying, and Daniel Krippner helped to illustrate how the framework may be applied in practice by talking through his use of it as part of the Eclipse uProtocol project. Daniel and I followed this with a quick discussion of how Rust is becoming increasingly relevant in the safety sphere, and how this may be relevant for ELISA.

The workshop wrapped up with a discussion on the Open Source Best Practices Standard, an initiative that was launched earlier this year. It included a live survey collecting input from the audience about their awareness of existing standards and suggestions for projects to be considered for examples of best practices.

Key Takeaways

I’ve attended numerous ELISA workshops since the first one in 2019, and it was wonderful to note how many passionate and enthusiastic newcomers we had attending this time. We also had participants from a variety of different backgrounds, including academics from the local university and engineers from the rail, medical and aeronautics industries, as well as the always-prevalent automotive specialists.

ELISA’s increasing engagement with other open source communities, including those from the Eclipse Foundation and Linux Foundation projects, is also good to see. The growing interest in safety-related topics in these communities, building on the already well-established awareness of cybersecurity topics, is also encouraging. After the enthusiastic reception that my talks had last week, I am hopeful that the Trustable Software Framework can help to continue this trend, giving all open source projects a way to start engaging with these topics and to share their thinking and strategies for building trust with other projects and communities.

→ Read: 'Codethink Limited Announces World’s First Baseline Safety Assessment for a Linux-Based OS to SIL 3 / ASIL D'

RISC-V Summit Europe 2025: What to Expect from Codethink

2025-05-08T00:00:00+01:00

RISC-V Summit Europe 2025 is just around the corner (12-15 May), and we’re excited to present a technical presentation and a poster exhibition. Learn out more about where and when you can find us below.

Talk: ‘Implementing Runtime-Configurable Endianness in RISC-V: Challenges and Solutions’

Join Lawrence Hunter, Roan Richmond, and Ben Dooks as they dive into the intricacies of implementing dynamic endianness support in RISC-V.

Talk information:

Session T2.3.06 (Submission #105)
Date: Wednesday 14 May at 12:45
Location: Gaston Berger (S2)

Abstract:

The RISC-V Privileged Specification introduced dynamic endianness switching in version 1.12, though no commercial hardware yet supports it. This work extends QEMU to enable big-endian execution, allowing the booting of a big-endian Linux system with OpenSBI. Modifications were required across QEMU, OpenSBI, the Linux kernel, and supporting libraries to ensure correct memory operations, instruction encoding, and runtime patching. The project demonstrates the feasibility of big-endian support for RISC-V, providing a foundation for future hardware and software development.

→ Sound interesting? You can read about the project in our blog post: ‘To boldly big-endian where no one has big-endianded before’

Poster Exhibition:

Come and chat with our engineers during the poster session on Tuesday, 13 May, when we’ll showcase this work in further detail.

Location: Island 2.1 (Level -2)

Learn more about our work on RISC-V: meet Codethink in Paris

If you’re attending RISC-V Summit and would like to learn more about Codethink’s work on RISC-V, let’s set up a meeting. See you in Paris!

Cyber Resilience Act (CRA): What You Need to Know

2025-04-22T00:00:00+01:00

Robust cybersecurity measures are becoming increasingly important in software design and development. Cyber threats continue to evolve and become more sophisticated, requiring governments to take action to safeguard vulnerable digital infrastructure.

Over the past decade, several significant cyberattacks have occurred across the public and private sectors. These include the WannaCry Ransomware Attack in 2017, which impacted more than 200,000 people worldwide, and the Kaseya VSA attack, in which hackers used insecure network administration software to disrupt over 100,000 businesses. These incidents (and others) underscored the need for a systemic solution, so the Cyber Resilience Act was developed.

This blog post provides a top-level overview of the EU Cyber Resilience Act, who it will impact, what happens if businesses don’t comply, and when it will be implemented.

What is the EU Cyber Resilience Act?

The Cyber Resilience Act (CRA) is an EU regulation that requires manufacturers of digital products to satisfy its standards before they can offer those products in EU member states.

Furthermore, the CRA is designed to protect consumers and businesses purchasing software products that include a digital component. This includes hardware products such as smartphones, tablets, smart home devices, computers, laptops, and software products such as mobile apps or cloud services.

The Cyber Resilience Act addresses the insufficient cybersecurity measures in products and the delays in security updates for software and hardware. In addition, the CRA establishes mandatory cybersecurity standards for manufacturers and retailers, regulating these products' planning, design, development, and maintenance. The act also mandates manufacturers to ensure proper support throughout their products' lifecycle.

→ Access the Cyber Resilience Act in all official EU languages

Who is impacted by the Cyber Resilience Act?

While the CRA covers a broad range of areas, its primary focus is on manufacturers of software-enabled devices. However, it will impact organisations throughout the supply chain.

The Cyber Resilience Act outlines three categories of businesses/organisations:

Manufacturers must ensure their products comply with the regulations.
Distributors need to verify that the products display the CE mark.
Importers are responsible for making sure only products that meet CRA requirements (and carry the CE mark) are made available.

When will the Cyber Resilience Act be implemented?

The Cyber Resilience Act came into effect on December 10, 2024, with the primary obligations set to take effect on December 11, 2027.

Cyber Resilience Act Enforcement and Penalties: What Businesses Need to Know

EU member states will appoint market surveillance authorities to enforce the Cyber Resilience Act. The respective national authorities will work alongside an EU-level supervisory structure to ensure the CRA is being observed.

Penalties include product withdrawals from the market. There is also the potential for fines up to €15 million or 2.5% of total worldwide annual turnover (whichever is higher).

How Codethink can help

Ultimately, the Cyber Resilience Act emphasises that manufacturers must integrate stringent cybersecurity measures into designing, developing, and implementing software-enabled digital products. Additionally, the CRA requires these products to maintain cybersecurity standards from production to end-of-life, with support for regular updates.

Codethink specialises in delivering secure and maintainable software systems, aligning with the Cyber Resilience Act's requirements. Codethink's experience in software development and design in safety-critical industries enables manufacturers to bring digital products to market and satisfy regulatory requirements.

Get in touch!

Interested in how Codethink can help your business with the Cyber Resilience Act?

→ Get in touch

Codethink

What is Deadline Scheduling in Linux?

Real-Time Performance Optimisations for Multi-Axis Robotic Systems using copper-rs

Real-time Considerations in Robotics

Operating System

Middleware Frameworks

copper-rs: A Case Study into Rust-first Robotics

Where ROS 2 Struggles

copper-rs

Modularity and Observability

Thread Pools and Real-time Scheduling

copper-rs on a Hexapod Robot

Specification

How It Works

Logging

Serial Bridge and Zenoh Controller

Rust-centric Workflow

Hexapod Motion Planner

Plans for Future Development

Conclusion

References

Instantaneous CI: Reducing Development Lag Through Cicada with Local-ﬁrst Design and Caching

Understanding Continuous Integration

Available Tooling for CI/CD

REAPI and CI

Developing Cicada

Assessing the reproducibility workflow in freedesktop-sdk

Introduction

Summarising reproducibility

The current solution inside of freedesktop-sdk

A target level approach

Workflow overview

The pros and cons of buildstream-reprotest-script

Metrics

Practical Impact

Conclusion

Future work

Deep Dive into Upstream RISC-V Boot Chain

RISC-V Toolchain

Privilege Levels

QEMU

Boot Flow Overview

Boot ROM aka ZSBL (Zero Stage Bootloader)

Debugging

OpenSBI

Debugging

U-Boot

Debugging

UEFI Boot Manager

Debugging

Linux Kernel

Conclusion

Porting an Automotive Operating System to RISC-V

RISC-V in Automotive

Porting Software

CTRL to RISC-V

Challenges Encountered

Conclusions

Understanding Codethink's IEC 61508 Mapping for the Eclipse Trustable Software Framework

What IEC 61508 Requires for Pre-Existing Software

What the Eclipse Trustable Software Framework Is

What the IEC 61508 Mapping to the TSF Covers

Create Your Own Mapping Between the TSF and a Compliance Standard

Resisting Hyrum's Law with Private Constructors in Python

Why Might You Want to Make the Constructor Private?

How Do I Make My Constructor Uncallable Then?

So How Do We Prevent the Object Being Created?

Private Constructors With Private Tokens

How Can I Create Objects Without Using __init__?

Getting Tooling to Warn

What Are My Alternatives if This Doesn't Work for Me?

Conclusion

FOSDEM 2026

Pre-FOSDEM Events

Talk: Your CRA Plan Is Already Broken: Only Trustable Software Can Save It

Description:

Panel: Deep Dive: Competitiveness & Open Source in the Automotive Industry

Description:

FOSDEM Events

Talk: KDE at 30: Still looking ahead

How Can I Create Objects Without Using `init`?