Reading/Writing IPC formats¶
Arrow defines two types of binary formats for serializing record batches:
Streaming format: for sending an arbitrary number of record batches. The format must be processed from start to end, and does not support random access
File or Random Access format: for serializing a fixed number of record batches. It supports random access, and thus is very useful when used with memory maps
Writing and Reading Streaming Format¶
First, let’s populate a VectorSchemaRoot with a small batch of records
BitVector bitVector = new BitVector("boolean", allocator);
VarCharVector varCharVector = new VarCharVector("varchar", allocator);
for (int i = 0; i < 10; i++) {
bitVector.setSafe(i, i % 2 == 0 ? 0 : 1);
varCharVector.setSafe(i, ("test" + i).getBytes(StandardCharsets.UTF_8));
}
bitVector.setValueCount(10);
varCharVector.setValueCount(10);
List<Field> fields = Arrays.asList(bitVector.getField(), varCharVector.getField());
List<FieldVector> vectors = Arrays.asList(bitVector, varCharVector);
VectorSchemaRoot root = new VectorSchemaRoot(fields, vectors);
Now, we can begin writing a stream containing some number of these batches. For this we use ArrowStreamWriter
(DictionaryProvider used for any vectors that are dictionary encoded is optional and can be null))
try (
ByteArrayOutputStream out = new ByteArrayOutputStream();
ArrowStreamWriter writer = new ArrowStreamWriter(root, /*DictionaryProvider=*/null, Channels.newChannel(out));
) {
// ... do write into the ArrowStreamWriter
}
Here we used an in-memory stream, but this could have been a socket or some other IO stream. Then we can do
writer.start();
// write the first batch
writer.writeBatch();
// write another four batches.
for (int i = 0; i < 4; i++) {
// populate VectorSchemaRoot data and write the second batch
BitVector childVector1 = (BitVector)root.getVector(0);
VarCharVector childVector2 = (VarCharVector)root.getVector(1);
childVector1.reset();
childVector2.reset();
// ... do some populate work here, could be different for each batch
writer.writeBatch();
}
writer.end();
Note that, since the VectorSchemaRoot in the writer is a container that can hold batches, batches flow through
VectorSchemaRoot as part of a pipeline, so we need to populate data before writeBatch, so that later batches
could overwrite previous ones.
Now the ByteArrayOutputStream contains the complete stream which contains 5 record batches.
We can read such a stream with ArrowStreamReader. Note that the VectorSchemaRoot within the reader
will be loaded with new values on every call to loadNextBatch()
try (ArrowStreamReader reader = new ArrowStreamReader(new ByteArrayInputStream(out.toByteArray()), allocator)) {
// This will be loaded with new values on every call to loadNextBatch
VectorSchemaRoot readRoot = reader.getVectorSchemaRoot();
Schema schema = readRoot.getSchema();
for (int i = 0; i < 5; i++) {
reader.loadNextBatch();
// ... do something with readRoot
}
}
Here we also give a simple example with dictionary encoded vectors
// create provider
DictionaryProvider.MapDictionaryProvider provider = new DictionaryProvider.MapDictionaryProvider();
try (
final VarCharVector