Introducing Sparrow-IPC: A modern C++ implementation of Arrow IPC
If you work in the world of data engineering, you are probably familiar with Apache Arrow — the universal columnar format and multi-language toolbox that has become the gold standard for fast in-memory analytics and data interchange.
Its IPC (Inter-Process Communication) component is what makes the magic happen across process boundaries. It allows different languages and systems to share data with zero-copy overhead, bundling schema definitions and record batches into a single binary stream that allows for efficient sequential reading.
But while Apache Arrow is powerful, its C++ implementation can sometimes feel heavy for projects that need to stay lean or for developers who prefer a more idiomatic, modern C++ experience.
This is where sparrow and its new companion, sparrow-ipc, come in.
The Foundation: Sparrow
Before diving into IPC, let’s talk about sparrow.
Sparrow is a modern C++ library that provides focused, C++20 idiomatic APIs for structured data in the Apache Arrow format. It emphasizes simplicity, mutability, and high performance. By concentrating on essential functionality and leveraging C++20 features, Sparrow makes handling complex columnar data straightforward without the overhead of a massive framework.
You can explore sparrow on its GitHub repository or read the blog post introducing it.
Sparrow-IPC
Built on top of the lightweight sparrow core, sparrow-ipc brings the same philosophy to Arrow Serialization and InterProcess Communication. It’s designed for developers who want the performance and compatibility of Arrow with the ergonomics of C++20 and minimal binary bloat.
sparrow-ipc fully supports the complete Arrow memory layout specification, ensuring that every data structure defined by Arrow is handled natively and efficiently. As a result, it is entirely compatible with the Apache Arrow ecosystem and interoperates seamlessly with any implementation that adheres to the Arrow specification.
The Anatomy of Arrow IPC
To understand the inner workings of sparrow-ipc, we first need to look at the Apache Arrow Specification — the standard that defines how data should be organized for high-speed interchange.
Columnar vs. Record-Oriented
Traditional formats like CSV are row-based, storing each record together, while JSON is not inherently row-based but is most often used in a record-oriented way.
In contrast, columnar formats like Arrow store each field (e.g., all “Ages” across records) contiguously, making them far more efficient for analytical workloads that access only a subset of columns.
Record Batches: The unit of work
In the Arrow world, a RecordBatch is the core unit of data processing — a collection of rows stored column by column, all following the same schema.
When streaming data via IPC, you don’t send one large dataset at once; instead, you send a sequence of messages that include the schema and multiple RecordBatches, with each RecordBatch carrying a chunk of the data.
Metadata vs. Data
The brilliant part of Arrow’s design is the separation of metadata (the description of the data) from the data itself (the raw bytes).
- The metadata tells you: “This is a column of 64-bit integers called ‘Age’, and its values start at an offset of 128 bytes into the data buffer.”
- The data is just: A raw buffer of memory containing those integers. Because the metadata provides the exact offsets, the receiver doesn’t need to “search” for the data; they can just jump to the correct byte and start reading.
Under the hood: the power of FlatBuffers
This separation is where sparrow-ipc leverages Google FlatBuffers.
While the actual columnar data stays in its raw, “ready-to-use” format (written directly to the stream as raw bytes), the metadata needs to be serialized into a structured format so that the receiving process knows how to interpret those bytes.
sparrow-ipc uses FlatBuffers — as defined in the Arrow IPC specification — to handle this “Message” header. When you serialize a Record Batch, it follows a well-defined sequence of steps:
1. Metadata Construction: It uses FlatBuffers to build a “Message” header. This header describes the schema, the number of rows, and — crucially — the exact offsets and lengths of the data buffers.
2. Zero-Copy Design: Because FlatBuffers itself is designed for zero-copy access, sparrow-ipc can parse these headers during deserialization without expensive memory allocations.
3. The Body: Following the FlatBuffers metadata, the actual columnar data (the “body”) is written. This is where the real “meat” is, and because the FlatBuffers header told us exactly where everything is, we can map these buffers directly into sparrow arrays and start working immediately.
Getting started
sparrow-ipc is already available on conda-forge and is coming soon to Conan Center and VCPKG, making it easy to drop into your existing C++ projects.
Get QuantStack’s stories in your inbox
Join Medium for free to get updates from this writer.
Let’s look at how this works in practice.
Seamless Record Batch serialization and deserialization
This snippet demonstrates the core “Hello World” of sparrow-ipc: turning a collection of sparrow::record_batch objects into a high-performance binary stream and vice-versa.
#include <vector>
// Note: The headers and namespaces shown here are included for context, but they will be omitted in the following snippets to keep them simpler and less verbose.
#include <sparrow.hpp>
#include <sparrow_ipc.hpp>
namespace sp = sparrow;
namespace sp_ipc = sparrow_ipc;
// Serialize
std::vector<uint8_t> serialize_data(const std::vector<sp::record_batch>& batches)
{
std::vector<uint8_t> buffer;
sp_ipc::memory_output_stream stream(buffer);
sp_ipc::serializer serializer(stream);
// Serialize all batches using the streaming operator
// IMPORTANT: Always call end_stream to write the end-of-stream marker
// This is required by the Arrow IPC specification
serializer << batches << sp_ipc::end_stream;
return buffer;
}
// Deserialize (function API)
std::vector<sp::record_batch> deserialize(const std::vector<uint8_t>& data)
{
// Deserialize the entire stream at once
return sp_ipc::deserialize_stream(data);
}For reference, sparrow record batches can be created as follows:
// Primitive types
auto int_array = sp::primitive_array<int32_t>({1, 2, 3});
auto double_array = sp::primitive_array<double>({1.5, 2.5, 3.5});
auto string_array = sp::string_array(std::vector<std::string>{"sparrow-ipc", "is", "great"});
sp::record_batch batch({
{"id", sp::array(std::move(int_array))},
{"score", sp::array(std::move(double_array))},
{"name", sp::array(std::move(string_array))}
});Boosting Performance with Compression
sparrow-ipc provides built-in support for LZ4 and ZSTD compression. This is essential for reducing network I/O or disk footprint.
#include <iostream>
#include <vector>
void serialize_with_compression(const std::vector<sp::record_batch>& batches)
{
std::vector<uint8_t> buffer;
sp_ipc::memory_output_stream stream(buffer);
// Enable ZSTD compression (or LZ4 using sp_ipc::CompressionType::LZ4_FRAME)
sp_ipc::serializer serializer(stream, sp_ipc::CompressionType::ZSTD);
serializer << batches << sp_ipc::end_stream;
std::cout << "Data compressed with zstd and stored in a " << buffer.size() << " byte buffer." << std::endl;
}
// Deserialization auto-detects compression
std::vector<sp::record_batch> deserialize(const std::vector<uint8_t>& data)
{
return sp_ipc::deserialize_stream(data);
}Real-time IPC: Incremental Deserialization
For many IPC use cases, you don’t want to wait for the entire payload to arrive. sparrow-ipc allows you to process “chunks” of data as they come off the wire.
#include <iostream>
#include <span>
#include <vector>
void process_incoming_data()
{
// Create a container to hold the results
std::vector<sp::record_batch> batches;
// Create the deserializer and point it to your container
sp_ipc::deserializer deser(batches);
// Feed the chunks as they arrive
while (auto chunk = read_next_network_chunk())
{
deser << std::span<const uint8_t>(chunk);
// As soon as a full batch is finished, it's added to 'batches'
std::cout << "After chunk: " << batches.size() << " batches accumulated" << std::endl;
}
// All batches are now available in the container
std::cout << "Total batches deserialized: " << batches.size() << std::endl;
}File I/O
Arrow IPC streams can be persisted to disk and loaded using standard C++ I/O streams. The serialized data is portable across platforms and Arrow implementations.
#include <iostream>
#include <fstream>
#include <string>
#include <vector>
// Write Arrow IPC stream to file
void write_to_file(const std::string& filename, const std::vector<uint8_t>& data)
{
std::ofstream out(filename, std::ios::binary);
out.write(reinterpret_cast<const char*>(data.data()),
static_cast<std::streamsize>(data.size()));
}
// Read Arrow IPC stream from file
std::vector<uint8_t> read_from_file(const std::string& filename)
{
std::ifstream in(filename, std::ios::binary);
std::vector<uint8_t> data(
(std::istreambuf_iterator<char>(in)),
std::istreambuf_iterator<char>()
);
return data;
}Using the helper functions above, you can save serialized batches to disk and load them back for deserialization:
// Complete example
void save_and_load_batches()
{
// Create sample batches
auto batch = sp::record_batch({
{"id", sp::array(sp::primitive_array<int32_t>({1, 2, 3}))},
{"name", sp::array(sp::string_array(std::vector<std::string>{"a", "b", "c"}))}
});
// Serialize
std::vector<uint8_t> buffer;
sp_ipc::memory_output_stream stream(buffer);
sp_ipc::serializer ser(stream);
ser << batch << sp_ipc::end_stream;
std::cout << "Serialized " << batch.nb_rows() << " rows into "
<< buffer.size() << " bytes" << std::endl;
// Write to file
write_to_file("data.stream", buffer);
std::cout << "Wrote " << buffer.size() << " bytes to file" << std::endl;
// Read from file
auto file_data = read_from_file("data.stream");
// Deserialize
auto loaded_batches = sp_ipc::deserialize_stream(file_data);
std::cout << "Deserialized " << loaded_batches.size() << " batches" << std::endl;
// Access the data
for (size_t i = 0; i < loaded_batches.size(); ++i)
{
std::cout << "Batch " << i << ": " << loaded_batches[i].nb_rows() << " rows, " << loaded_batches[i].nb_columns() << " columns" << std::endl;
}
}Note: For the complete Arrow file format (with magic bytes and footer validation), use deserialize_file() instead of deserialize_stream().
For more details, examples, and documentation, check out the sparrow-ipc repository. More examples and comprehensive docs coming soon!
What’s up next?
sparrow-ipc delivers full support for the Apache Arrow IPC specification, including all layouts and compression, with a strong emphasis on performance and interoperability.
Looking ahead, several enhancements are planned to further strengthen the project. In particular, we plan to extend support for endianness as described in the Arrow specification, enabling smoother operations across heterogeneous systems.
In parallel, we are expanding documentation and practical examples to improve usability, and evolving the API with higher-level abstractions to provide a more ergonomic developer experience while preserving performance.
Additionally, we aim to explore WebAssembly (WASM) integration to enable running sparrow-ipc directly in browser environments, opening the door to new client-side and edge use cases.
Together, these improvements aim to make sparrow-ipc an even more robust, portable, and accessible component of the Arrow ecosystem.
Go further with the Sparrow ecosystem
Sparrow is growing to match the Apache Arrow ecosystem. Here are other related libraries that extend its capabilities:
sparrow-extensions: Provides additional array types and extension implementations for sparrow, following the Apache Arrow Canonical Extensions specification.
sparrow-rockfinch (WIP): A C++/Python library for exchanging Apache Arrow data between C++ and Python using the Arrow C Data Interface via PyCapsules.
Acknowledgements
The work on sparrow-ipc was funded as part of a collaboration between Man Group, Bloomberg, and QuantStack.
We thank Antoine Pitrou, a core contributor to Apache Arrow, for his valuable guidance and thoughtful reviews throughout this work.
About the Authors
Hind Montassif is a C++ scientific software developer at QuantStack.
Prior to working on the Sparrow-IPC project, Hind has worked on several C++ projects, including Mamba, ArcticDB, and Xeus stack.
Alexis Placet is a C++ scientific software developer at QuantStack.
In addition to Sparrow-IPC, Alexis has mostly been active in the development of the Sparrow ecosystem including Sparrow, Sparrow-extensions and Sparrow Rockfinch.
Johan Mabille is a Technical Director specialized in high-performance computing in C++. As an open source developer, Johan co-authored xtensor, xeus, and xsimd.
He leads the C++ team at QuantStack, where he oversees the development and maintenance of mamba and the Jupyter Xeus project.
Johan has also made significant contributions to JupyterLab.
