Dataset¶
Warning
Experimental: The Java module dataset is currently under early
development. API might be changed in each release of Apache Arrow until it
gets mature.
Dataset is an universal layer in Apache Arrow for querying data in different formats or in different partitioning strategies. Usually the data to be queried is supposed to be located from a traditional file system, however Arrow Dataset is not designed only for querying files but can be extended to serve all possible data sources such as from inter-process communication or from other network locations, etc.
Getting Started¶
Currently supported file formats are:
Apache Arrow (
.arrow)Apache ORC (
.orc)Apache Parquet (
.parquet)Comma-Separated Values (
.csv)Line-delimited JSON Values (
.json)
Below shows a simplest example of using Dataset to query a Parquet file in Java:
// read data from file /opt/example.parquet
String uri = "file:/opt/example.parquet";
ScanOptions options = new ScanOptions(/*batchSize*/ 32768);
try (
BufferAllocator allocator = new RootAllocator();
DatasetFactory datasetFactory = new FileSystemDatasetFactory(
allocator, NativeMemoryPool.getDefault(),
FileFormat.PARQUET, uri);
Dataset dataset = datasetFactory.finish();
Scanner scanner = dataset.newScan(options);
ArrowReader reader = scanner.scanBatches()
) {
List<ArrowRecordBatch> batches = new ArrayList<>();
while (reader.loadNextBatch()) {
try (VectorSchemaRoot root = reader.getVectorSchemaRoot()) {
final VectorUnloader unloader = new VectorUnloader(root);
batches.add(unloader.getRecordBatch());
}
}
// do something with read record batches, for example:
analyzeArrowData(batches);
// finished the analysis of the data, close all resources:
AutoCloseables.close(batches);
} catch (Exception e) {
e.printStackTrace();
}
Note
ArrowRecordBatch is a low-level composite Arrow data exchange format
that doesn’t provide API to read typed data from it directly.
It’s recommended to use utilities VectorLoader to load it into a schema
aware container VectorSchemaRoot by which user could be able to access
decoded data conveniently in Java.
The ScanOptions batchSize argument takes effect only if it is set to a value
smaller than the number of rows in the recordbatch.
See also
Load record batches with VectorSchemaRoot.
Schema¶
Schema of the data to be queried can be inspected via method
DatasetFactory#inspect() before actually reading it. For example:
// read data from local file /opt/example.parquet
String uri = "file:/opt/example.parquet";
BufferAllocator allocator = new RootAllocator(Long.MAX_VALUE);
DatasetFactory factory = new FileSystemDatasetFactory(allocator,
NativeMemoryPool.getDefault(), FileFormat.PARQUET, uri);
// inspect schema
Schema schema = factory.inspect();
For some of the data format that is compatible with a user-defined schema, user
can use method DatasetFactory#inspect(Schema schema) to create the dataset:
Schema schema = createUserSchema()
Dataset dataset = factory.finish(schema);
Otherwise when the non-parameter method DatasetFactory#inspect() is called,
schema will be inferred automatically from data source. The same as the result
of DatasetFactory#inspect().
Also, if projector is specified during scanning (see next section
Projection (Subset of Columns)), the actual schema of output data can be got
within method Scanner::schema():
Scanner scanner = dataset.newScan(
new ScanOptions(32768, Optional.of(new String[] {"id", "name"})));
Schema projectedSchema = scanner.schema();
Projection (Subset of Columns)¶
User can specify projections in ScanOptions. For example:
String[] projection = new String[] {"id", "name"};
ScanOptions options = new ScanOptions(32768, Optional.of(projection));
If no projection is needed, leave the optional projection argument absent in ScanOptions:
ScanOptions options = new ScanOptions(