Array Data Engine is a Python package for turning heterogeneous sensor and array data into a consistent NumPy-first stream. It can read image sequences, ROS bag files, ROS 2 .db3 recordings or split rosbag2 directories, and DEM tiles, then keep recent context in memory or persist complete streams to MCAP (the default), Apache Arrow/Parquet, or TileDB.
The project is aimed at robotics and perception workflows where algorithms need synchronized windows of image, point cloud, navigation, odometry, IMU, and terrain-like array data.
- NumPy-oriented message dictionaries with
data,timestamp,topic, andnamefields. - Source adapters for image folders,
.bag,.db3,.mcap, split rosbag2 directories, and DEM data — including bags recorded through rosbridge/foxglove with JSON payloads. adecommand-line interface: inspect, export, ingest, and visualize bags without writing code.- Synthetic data source with a geometrically consistent trajectory for demos and tests — no data files needed.
- Topic statistics (
describe_topic/describe_dataset) and portable.npztopic round-trips. - TUM and KITTI trajectory export (compatible with
evoand SLAM evaluation tooling). - Cached ROS topic metadata, so repeated topic and count lookups do not reopen the same bag.
- Rolling in-memory buffers for recent context windows.
- Persistent storage for larger-than-memory datasets: MCAP segments (default), Apache Arrow/Parquet analysis fragments, or TileDB dense arrays. Parquet and TileDB support metadata query pushdown.
- Lazy dataset-level selection by topic, time, message index, frame id, geographic bounds, and spatial bounds.
- Lazy optional imports so lightweight workflows do not need the full ROS, DEM, or visualization stack.
- Notebook examples for demos, iterators, terrain, INS, MOT, and TileDB workflows.
python -m pip install arraydataengineThe distribution and canonical import name are both arraydataengine:
from arraydataengine import DataBuffer, DataSourcesThe legacy-cased ArrayDataEngine package remains available as a convenience facade:
import ArrayDataEngine as ADEFor editable local development:
python -m pip install -e .The base install includes NumPy and MCAP. Install feature extras for the source adapters and storage backends you use:
| Extra | Adds |
|---|---|
arrow |
Parquet analysis caches and explicit Arrow storage |
mcap |
Compatibility extra; MCAP is included in the base install |
tiledb |
Alternative TileDB persistent storage |
image |
Image sequence loading and image-processing dependencies |
ros |
rosbags reader for ROS 1 bags, MCAP, and split rosbag2 directories |
dem |
DEM downloads and raster processing |
visualization |
Matplotlib, Open3D, OpenCV, Pillow, and IPython visualization support |
ml |
PyTorch and model/dataset integrations |
notebook |
JupyterLab |
dev |
Test tooling |
Extras can be combined for either a registry or editable install:
python -m pip install "arraydataengine[ros,arrow]"
python -m pip install -e ".[dev,image,ros,arrow]"The full Docker/notebook environment still uses requirements.txt, which includes the heavier ROS, ML, notebook, and visualization dependencies.
Installing the package also installs the ade command:
ade demo -o demo/ # end-to-end synthetic showcase, no data needed:
# stitched point-cloud map (HTML), TUM trajectory, NPZ export
ade info my_bag.db3 --messages 500 # topics, counts, duration, rates, jitter
ade topics my_bag.db3
ade export my_bag.db3 -t /points -o points.npz # or -o exports/ for exports/points.npz
ade viewer my_bag.db3 -t /points -o viewer.html --stride 5
ade ingest my_bag.db3 -o /data/stores/my_bag/ # persist every topic (mcap by default, --backend arrow for analysis)ade viewer and ade demo write self-contained HTML files that render in any
browser with no extra dependencies. Exported .npz files load back with
arraydataengine.ops.load_topic_npz and plug straight into the ops pipelines.
Standalone SQLite recordings (.db3, or a single SQLite .bag) are read with
the built-in decoder and need no extras; ROS 1 bags, MCAP files, and split
rosbag2 directories require the ros extra. ade ingest also requires the
extra for its storage backend: new stores default to mcap, while an existing
output store is reopened with the backend it was written with. A --backend
that doesn't match an existing store, and any non-empty directory that isn't
a store, are refused.
Point clouds of any size are read by default. --max-points N (on info,
topics, export, viewer, and ingest) skips clouds with more than N
points, and the command says so when every message of a topic was skipped.
ade export zero-pads variable-size clouds to the largest exported cloud and
stores the real row counts in the archive's lengths array. ade ingest
stores up to --max-points rows per cloud (30000 when omitted). It records the
recording it read in the store's ade_source.json, and refuses to resume the
store from a different recording or with a different --max-points
(--force appends anyway). ade info also lists topics it cannot decode and
the extra they need. Missing, empty, corrupt, or Git LFS pointer recordings,
unsupported message types, unwritable outputs, and missing extras are reported
as one-line errors (exit status 2; Ctrl-C exits with 130). Pass --debug
(before or after the subcommand) or set ADE_DEBUG=1 for a full traceback.
| Input | Adapter | Topic(s) | Notes |
|---|---|---|---|
*.png, *.jpg, *.jpeg, *.tif, *.tiff (any case) |
ImgSource |
images |
Reads naturally-sorted image paths from a glob at their native bit depth (16-bit thermal frames stay uint16); each page of a multi-page TIFF stack is one message (name is <file>#<page>) (a literal path containing [ or * also works). With flir="raw" or flir="celsius" (or "kelvin"/"fahrenheit"), FLIR radiometric JPEGs yield their sensor counts or temperatures instead of the palette image. Colour images are RGB; pass color_order="bgr" for OpenCV's order. |
.bag |
BagSource |
Bag topics | ROS1 bags with image, compressed image, point cloud, IMU, odometry, navsat, pose, and DepthAnything calibration messages. SQLite rosbag2 files named .bag are routed to DB3Source. |
.db3/.mcap file or rosbag2 directory |
DB3Source |
Bag topics | Single and split ROS 2 bags (sqlite or MCAP storage) with image, compressed image, point cloud, IMU, odometry, navsat, pose, and DepthAnything calibration messages. Bags without embedded message definitions (Humble and older) are supported. rosbridge/foxglove JSON payloads are decoded for PoseStamped, NavSatFix, PointCloud2, CompressedImage, and DepthAnything calibration messages; the serialization format recorded in the .db3 takes precedence over metadata.yaml. compressed_depth_image_transport (compressedDepth) images are decoded too. |
"DEM" |
DEMSource |
images |
Downloads SRTM HGT tiles over HTTPS using Earthdata credentials. |
SyntheticSource |
(import directly) | configurable | Deterministic synthetic IMU/odometry/point-cloud/navsat/image streams on a circular trajectory. |
Each yielded message has this shape:
{
"data": np.ndarray,
"timestamp": 123.456,
"topic": "images",
"name": "frame_000.png",
}ROS messages also carry frame_id (from the message header). The name of a ROS message is its short type name, such as PointCloud2. Point clouds drop non-finite points and are yielded at their real size as an (N, 3) array in the field dtype, with point_count (= N) on the message. Pass DataSources(path, max_points=...) to skip scans larger than a limit (there is none by default). Messages that fail to decode are skipped with a logged warning. Every source yields colour images as RGB (or RGBA), and image messages carry an encoding key such as rgb8, mono16, or 32FC1. NavSatFix messages also carry status (-1 means no fix) and a 3×3 position_covariance. get_topics() and get_count() only report topics whose message type can be decoded, and DataSources(path, topics=[...]) decodes only the topics you name.
Read an image sequence directly:
from arraydataengine.source import DataSources
source = DataSources("/data/frames/*.png", period=0.1)
print(source.get_topics()) # ["images"]
print(source.get_count("images")) # number of matched frames
for message in source.get_message():
image = message["data"]
timestamp = message["timestamp"]
print(timestamp, image.shape)Read a ROS bag, ROS 2 database, or split rosbag2 directory:
from arraydataengine.source import DataSources
bag_source = DataSources("/data/recording.bag")
ros2_source = DataSources("/data/rosbag2/split_recording/")
for topic in ros2_source.get_topics():
print(topic, ros2_source.get_count(topic))For split ROS 2 bags, pass the directory that contains metadata.yaml and the chunk files. Passing one .db3 chunk that metadata.yaml lists still reads the whole recording; a standalone SQLite file is read on its own. A directory of .db3 files without metadata.yaml is read chunk by chunk in natural filename order. Reading ROS .mcap files uses the ros extra; ADE recordings use the mcap extra described below.
Install pip install "arraydataengine[mcap]" to record ADE message dictionaries
as they arrive. Supply a generator from your device, ROS subscription queue, or
other capture code; ADE does not open live devices or subscribe to ROS itself.
Here is an example using the synthetic source:
from arraydataengine import DataSources, McapRecorder
from arraydataengine.sources.synthetic_source import SyntheticSource
with McapRecorder("session.mcap") as recorder:
recorder.record(SyntheticSource(duration=2.0).get_message())
# For live capture, call recorder.write(message) for each incoming message:
# {"topic": "/imu", "timestamp": seconds, "name": "Imu", "data": numpy_array}
# recorder.flush() # write the current chunk; keep recording
# recorder.flush(durable=True) # also fsync the file
source = DataSources("session.mcap")
for message in source.get_message():
process(message) # yields one message at a time in recorded file orderrecord(iterable) consumes the iterable synchronously and returns the number
of messages written when it ends. An unbounded generator records until it is
stopped; a context manager finalizes the file on normal exit or a Python
exception (including Ctrl-C). write(message) calls are serialized, allowing
callbacks to share one recorder. Capture queues and backpressure belong to the
caller. Existing files are refused unless overwrite=True; appending to an
existing MCAP is not supported.
MCAP stores topic channels, timestamps, indexes, and compressed chunks. The
recorder defaults to Zstd and a 1 MiB chunk target; compression="lz4" or
"none" and chunk_size=... are also supported. A message can exceed the chunk
target. Pending data is flushed on close or explicit flush(); each write()
is not immediately durable. The writer also retains chunk indexes until close,
so index memory grows with recording length. Abrupt process termination can
leave an unfinished file; automatic crash recovery is not provided.
The ADE arraydataengine/numpy/v1 profile supports heterogeneous channels.
write(message) stores its payload as a pickle-free NumPy array, preserving
dtype and shape without padding variable-size scans.
Other fields (such as name, frame_id, encoding, and point_count) use JSON;
NumPy metadata arrays/scalars become lists/scalars. Object arrays and metadata
that cannot be encoded as JSON are rejected. timestamp is non-negative
seconds; optional integer timestamp_ns and publish_time_ns preserve exact
MCAP nanosecond timestamps, and sequence is an optional uint32 counter.
Use write_encoded to keep original serialized messages alongside NumPy
channels. For example, in a ROS callback with serialized CDR bytes and the
message's full ROS definition (including dependencies):
recorder.write_encoded(
"/imu", cdr_bytes,
timestamp_ns=log_time_ns,
message_encoding="cdr",
schema_name="sensor_msgs/msg/Imu",
schema_encoding="ros2msg",
schema_data=full_ros_definition.encode("utf-8"),
)CDR means Common Data Representation, the binary serialization used for ROS 2
messages. The generator selects a decoder per channel and schema: ADE NumPy
and the project's supported ROS 2/CDR sensor types work automatically, even in
one file. Install arraydataengine[mcap,ros] for schema-based ROS decoding,
plus image for image types that need OpenCV. Arbitrary custom ROS types still
need an ADE array conversion. ROS/Foxglove tools need a decoder for the
ade-numpy-v1 channels; mixed ADE recordings are not standard ROS-only bags.
For additional encodings or custom types, pass
DataSources(path, mcap_decoders={encoding: decoder}) or use
{(encoding, schema_name): decoder} for a schema-specific override. Each
decoder receives MCAP (schema, channel, record) objects and returns a
dictionary containing data and optional metadata. The reader fills missing
topic, name, timestamps, and sequence information. Unknown encodings raise
an error; select supported topics with topics= or supply a decoder. Passing
mcap_decoders={} selects the generic reader for an external MCAP; ordinary
ROS MCAP files retain the existing ros reader by default.
No Parquet conversion is required. Pass a completed recording to DataSources,
DataBuffer, or source_pipeline for playback and processing. Playback uses a
lazy generator without sorting or loading the whole recording, runs at the
consumer's pace, and does not tail a growing file or reproduce wall-clock timing.
MCAP is also the default persistent DataBuffer backend. Existing Arrow and
TileDB stores are detected automatically; use_db=False still selects memory.
New persistent DataBuffer(..., use_db=True) stores and ade ingest default
to MCAP. A store is a directory of finalized MCAP segments and an atomic
mcap_store.json manifest. Interrupted, uncommitted segments are ignored on
resume; keep the same source ordering when replaying. One process may append
at a time. Reads flush the current segment before streaming. Eager reads such
as get_buffer() materialize their selection; use buffer.iter_topic_chunks(topic, chunk_size=64)
for payload batches bounded by the MCAP memory budget. Topic pipelines may
regroup those rows up to their requested chunk size. Sensor capture and backpressure remain the
caller's responsibility. Native ingestion records the decoded ADE messages;
use McapRecorder.write_encoded() to retain original ROS wire payloads.
MCAP backend options include chunk_size (1 MiB), segment_bytes (64 MiB),
compression (zstd, lz4, none), and memory_budget_bytes (64 MiB).
The budget limits read batches and rejects oversized individual messages; it
is not a total process RSS limit. Arrow's memory_budget_bytes (64 MiB) limits
staging across all topics, flushing the largest topic when needed.
Use a derived Parquet cache for selective analysis. It separates changes in array shape, dtype, topic, and source schema into partitions, without padding ragged scans. Exact log/publish nanoseconds, sequence, original record ordinal, and per-topic ordinal are stored alongside a pickle-free NumPy message envelope. Trajectory topics additionally expose XYZ and quaternion scalar columns for filtering; other array payloads remain binary envelopes, not flattened SQL columns.
from arraydataengine import mcap_to_parquet, parquet_to_mcap
cache = mcap_to_parquet(
"session.mcap", "analysis/",
topics=["/pose", "/points"],
trajectory_topics={"/pose": "pose"}, # "odometry" for Odometry arrays
memory_budget_bytes=64 * 1024**2,
row_group_bytes=8 * 1024**2,
fidelity="copy",
)
parquet_to_mcap(cache, "restored.mcap")Install arraydataengine[arrow] for these operations. fidelity="reference"
(the default) retains SHA-256-checked references to the original files;
fidelity="copy" also archives the originals inside the cache. Restoration
copies the originals byte-for-byte, including raw CDR, schemas, attachments,
metadata, and message order—even channels omitted from analysis. It does not
apply edits made to derived Parquet columns. Reference restoration requires
unchanged source files. Cache reuse verifies hashes and conversion options;
changed sources/options require a fresh destination. New caches are published
only after successful conversion. For very large recordings, hash verification
adds a sequential I/O pass. Finalize recording before conversion.
Loop closure on a recording automatically uses this Parquet path:
from arraydataengine.ops import iter_recording_loop_closures
for closure in iter_recording_loop_closures(
"session.mcap",
trajectory_topic="/pose",
points_topic="/points",
trajectory_kind="pose",
radius=3.0,
min_separation=60,
time_tolerance=0.05,
voxel_size=0.25,
cache_dir="loop-analysis/",
):
process(closure)Candidate queries read only pose metadata; ICP loads the current scan and one
candidate scan. Clouds are matched by nearest timestamp within the explicit
tolerance (time_tolerance above); missing matches raise an error. No entire
cloud sequence or trajectory spatial index is held in memory. Predicate scans
can still be expensive on long, poorly clustered trajectories. The budget
controls conversion staging, not ICP working memory. Omitting cache_dir uses
a temporary cache cleaned up when the iterator is exhausted or closed.
verify_recording_loop_closures() collects the results into arrays; use the
iterator when results themselves may exceed memory. Existing NumPy-based
verify_loop_closures() remains available for in-memory inputs.
DataBuffer keeps a recent window for each topic and exposes NumPy-like indexing on the selected axis topic.
from arraydataengine.buffer import DataBuffer
from arraydataengine.source import DataSources
axis = "images"
source = DataSources("/data/frames/*.png", period=0.1)
buffer = DataBuffer(
data_source=source,
buffer_depth=10,
axis=axis,
use_db=False,
)
# The constructor preloads one axis message by default.
buffer.roll_buffer(axis)
latest_image = buffer[-1]
window = buffer.get_buffer()
image_window = window[axis]["data"]
timestamps = window[axis]["ts"]Use copy=False when you want to avoid extra in-memory copies during read-only inspection:
window = buffer.get_buffer(copy=False)Use preload=True to fill the entire window during construction, or preload=0 to create the buffer without reading from the source.
buffer = DataBuffer(source, buffer_depth=10, axis="images", preload=True)Slice buffered data by timestamp:
time_range = buffer.get_time_range(axis, start=12.0, end=12.5)
index_range = buffer.get_index_range(axis, start=100, stop=200, step=2)
recent = buffer.get_last_seconds(axis, seconds=0.5)
images = time_range["data"]
timestamps = time_range["ts"]Time ranges are inclusive and return the same {"id", "ts", "data"} shape for in-memory, Arrow-backed, and TileDB-backed buffers.
The arraydataengine.ops package provides NumPy-first helpers for common robotics data operations. They work on topic arrays returned by DataBuffer.get_buffer(), get_time_range(), and get_last_seconds().
from arraydataengine.ops import map_topic, random_downsample, select_time_range, topic_view, voxel_downsample
window = buffer.get_buffer()
images = select_time_range(window["images"], start=12.0, end=12.5)
normalized = map_topic(images, lambda frame: frame.astype("float32") / 255.0)
points = topic_view(window["/points"]).rows()[-1] # latest scan, without storage padding
reduced_points = voxel_downsample(points, voxel_size=0.25)
preview_points = random_downsample(points, count=min(10_000, len(points)), seed=42)Image and depth sequences can be resized, cropped, padded, normalized, color-converted, dtype-converted, backprojected, converted to normals, motion-aligned, and fused as NumPy stacks.
import numpy as np
from arraydataengine.ops import align_images, calibrate_depth_metric_scale, convert_color, convert_image_dtype, crop_images, depth_to_normals, frame_to_frame_optical_flow, fuse_rgbd_frames, image_gradients, image_mask, image_pyramid, iter_rgbd_frame_points, local_statistics, motion_compensated_rolling_windows, normalize_images, open_mask, resize_images
frames = window["images"]["data"]
depth_frames = window["depth"]["data"]
small = resize_images(frames, shape=(240, 320))
roi = crop_images(small, row_start=40, row_stop=200, col_start=80, col_stop=260)
float_roi = normalize_images(convert_image_dtype(roi, np.float32), per_image=True)
gray_roi = convert_color(roi, "rgb_to_gray")
foreground = open_mask(image_mask(gray_roi, min_value=32), size=3)
edges = image_gradients(gray_roi, method="sobel")
pyramid = image_pyramid(gray_roi[0], levels=4)
local = local_statistics(gray_roi, size=5, statistics=("mean", "std"))
flow = frame_to_frame_optical_flow(gray_roi, max_shift=8)
aligned_frames = align_images(gray_roi, max_shift=8)
for aligned_window in motion_compensated_rolling_windows(gray_roi, window_size=5, max_shift=8):
process(aligned_window)
normals = depth_to_normals(depth_frames[0], fx=525.0, fy=525.0, cx=319.5, cy=239.5)
for rgbd_chunk in iter_rgbd_frame_points(depth_frames, frames, fx=525.0, fy=525.0, cx=319.5, cy=239.5):
process(rgbd_chunk)
rgbd_cloud = fuse_rgbd_frames(depth_frames[:10], frames[:10], fx=525.0, fy=525.0, cx=319.5, cy=239.5)
depth_calibration, metric_depth = calibrate_depth_metric_scale(relative_depth, lidar_points, fx=525.0, fy=525.0, cx=319.5, cy=239.5, return_adjusted=True)Image registration uses phase correlation. On sensors with fixed-pattern noise, such as thermal cameras with column/row/pixel offsets, the pattern stays put while the scene moves, and plain align_images then reports no motion. register_with_fixed_pattern(frames, max_shift=..., frozen=...) estimates and removes the pattern while it registers. It returns integer shifts, the corrected registered frames, and the pattern. It needs a few tens of frames in which the scene moves across the sensor.
Thermal (LWIR) cameras deliver 14/16-bit counts, and those stay uint16 through every source (image files, multi-page TIFF stacks, and ROS mono16/16UC1/16SC1 images or 16-bit PNG CompressedImages, including compressedDepth) and every storage backend. A scene usually spans a narrow band of that range, so full-range conversions such as convert_image_dtype(frame, np.uint8) crush it to a few grey levels. Use automatic gain control for display instead:
from arraydataengine.ops import colorize_thermal, radiometric_to_temperature, thermal_agc
display = thermal_agc(frames) # robust 1-99 percentile stretch -> uint8
display = thermal_agc(frames, method="plateau") # plateau histogram equalization, as thermal cameras use
false_colour = colorize_thermal(frames, cmap="inferno") # (N, H, W, 3) uint8
celsius = radiometric_to_temperature(frames, scale=0.01) # FLIR TLinear: kelvin = counts * 0.01One mapping is shared across a sequence (per_frame=True stretches each frame alone), so playback does not flicker. radiometric_to_temperature and temperature_to_radiometric convert radiometric ("TLinear") counts; FLIR's radiometric Boson and Lepton cameras use scale=0.01 (or 0.1 at low resolution). The image VisTool (and Visualizer("image")) scales 16-bit frames between their 1st and 99th percentiles, so a few hot or dead pixels do not flatten the view, and takes a cmap such as "gray" or "inferno".
FLIR radiometric JPEGs (R-JPEG, from FLIR handheld and fixed cameras, the Vue Pro R and Duo Pro R, and the FLIR-built DJI Zenmuse XT/XT2) and .fff files are different: their raw counts follow the radiance reaching the sensor, not the temperature, so the TLinear model above would be wrong for them. Converting them to temperature takes the camera's Planck constants (R1, R2, B, F, O), the emissivity of the object, the reflected apparent temperature, and the atmosphere (distance, air temperature, humidity) and IR window between the object and the camera. The file stores all of these, and read_flir_radiometric reads them together with the raw image:
from arraydataengine import DataSources
from arraydataengine.ops import FlirCalibration, flir_raw_to_temperature, read_flir_radiometric
image = read_flir_radiometric("IR_0042.jpg") # also .fff files, bytes or file objects
image.raw # (H, W) uint16 sensor counts
celsius = image.temperature() # Planck + emissivity + atmosphere, from the file's parameters
celsius = image.temperature(emissivity=0.95, reflected_temperature_c=18.0, object_distance=3.0) # re-analyse
calibration = image.calibration.replace(relative_humidity=0.8) # FlirCalibration (temperatures in °C)
kelvin = flir_raw_to_temperature(image.raw, calibration, unit="kelvin")
frames = DataSources("flight/*.jpg", flir="celsius") # float32 temperatures per file; "raw" for the countsThe conversion follows the model that FLIR's software uses, as documented by ExifTool and the Thermimage R package. It removes the radiance reflected by the object, emitted by the air on both sides of the window, and emitted by the window, then inverts the Planck curve. flir_temperature_to_raw is the inverse, for simulation. With flir=, each image message also carries "flir": the file's calibration and camera information as JSON-compatible metadata, which FlirCalibration.from_dict reads back and recordings keep. scripts/datasets/make_flir_rjpeg.py writes synthetic R-JPEGs with ground-truth temperature maps. DJI's own radiometric JPEGs (H20T, M30T, Mavic Enterprise) use another format that needs DJI's Thermal SDK, and are rejected with a message that says so.
Point-cloud downsampling includes voxel-grid averaging, every-k uniform sampling, seeded random sampling by count or ratio, and farthest-point sampling.
from arraydataengine.ops import calibrate_point_cloud_metric_scale, connected_components, curvature_descriptors, farthest_point_downsample, hybrid_search, multi_scale_icp, nearest_neighbor_distance_stats, segment_ground, to_open3d_point_cloud, uniform_downsample, verify_loop_closures
uniform_points = uniform_downsample(points, every_k=4)
keypoints = farthest_point_downsample(points, count=2_048)
shape_features = curvature_descriptors(points, k=16)
spacing = nearest_neighbor_distance_stats(points, k=4)
distances, indices, counts = hybrid_search(points, query_points, radius=0.5, max_neighbors=32)
components = connected_components(points, radius=0.5, min_component_size=20)
ground, obstacles, ground_mask = segment_ground(points, distance_threshold=0.15)
registration = multi_scale_icp(scan_a, scan_b, voxel_sizes=(1.0, 0.5, 0.25))
open3d_cloud = to_open3d_point_cloud(colored_points, color_columns=(3, 4, 5)) # x, y, z, r, g, b columns
closures = verify_loop_closures(point_cloud_sequence, trajectory, radius=3.0, min_separation=60, voxel_size=0.25)
scale_calibration, metric_points = calibrate_point_cloud_metric_scale(relative_points, lidar_points, return_adjusted=True)Registration metrics are always gated: fitness is the fraction of finite source points with a target correspondence within the gating distance, and inlier_rmse is the point-to-point RMSE of those inliers for every ICP method. verify_loop_closures uses max_correspondence_distance when given. Otherwise it scores fitness at 2×voxel_size, or at 3× the target cloud's median point spacing, so unrelated scans are rejected. It first registers coarse to fine (5×, √5× and 1× that gate) so ordinary odometry drift still converges; pass an explicit gate when the seed may be off by much more than about 15× the point spacing. Nearest-neighbour searches use scipy.spatial.cKDTree when SciPy is installed and a memory-bounded chunked search otherwise. Non-finite points are ignored, and per-point outputs such as normals return NaN for them.
SE(3) coordinate-frame helpers work across common robotics arrays:
from arraydataengine.ops import apply_transform, transform_navsat, transform_odometry
points_in_map = apply_transform(points_in_lidar, lidar_to_map)
odom_in_map = transform_odometry(odom_message_array, odom_to_map) # twist stays in the body (child) frame
gps_in_map_frame = transform_navsat(gps_samples, enu_transform, ref_lat=37.0, ref_lon=-122.0, ref_alt=10.0)nav_msgs/Odometry twists are expressed in the child (body) frame, so transform_odometry re-expresses only the pose by default. Pass twist_frame="parent" if your twist is already in the parent frame and should be rotated along with the pose.
Use FrameGraph when transforms need to be composed by frame name. Static transforms are used directly; time-varying transforms are interpolated by timestamp, including rotation SLERP.
from arraydataengine.ops import FrameGraph
frames = FrameGraph()
frames.add_static_transform("base_link", "odom", base_to_odom)
frames.add_time_varying_transform(
"lidar",
"base_link",
timestamps=lidar_tf_timestamps,
transforms=lidar_to_base_samples,
)
points_in_odom = frames.transform_points(points_in_lidar, "lidar", "odom", timestamp=12.5)Projection helpers connect point clouds, depth images, RGB images, DEM grids, and camera frames with pinhole intrinsics.
from arraydataengine.ops import backproject_pixels, camera_model, colorize_points, depth_to_point_grid, distort_pixels, points_to_depth_image, project_camera_points, project_dem_to_image, rectify_image, rgbd_to_points, scale_camera_matrix
camera = camera_model(fx=525.0, fy=525.0, cx=319.5, cy=239.5, image_shape=rgb_image.shape[:2], distortion=distortion_coeffs)
small_camera_matrix = scale_camera_matrix(camera.camera_matrix, scale_x=0.5) # half_pixel_centers=True to match cv2.resize
rectified_rgb = rectify_image(rgb_image, camera.camera_matrix, distortion=camera.distortion)
distorted_pixels = distort_pixels(raw_pixels, camera.camera_matrix, camera.distortion)
camera_points = backproject_pixels(distorted_pixels, depth_values, camera.camera_matrix, distortion=camera.distortion)
projected_pixels, visible = project_camera_points(points_in_camera, camera)
organized_depth_points = depth_to_point_grid(depth_image, fx=525.0, fy=525.0, cx=319.5, cy=239.5)
rgbd_cloud = rgbd_to_points(depth_image, rgb_image, fx=525.0, fy=525.0, cx=319.5, cy=239.5)
colored_lidar = colorize_points(points_in_camera, rgb_image, fx=525.0, fy=525.0, cx=319.5, cy=239.5, distortion=camera.distortion)
rendered_depth = points_to_depth_image(points_in_camera, image_shape=rgb_image.shape[:2], fx=525.0, fy=525.0, cx=319.5, cy=239.5)
dem_pixels, dem_mask = project_dem_to_image(elevation, fx=525.0, fy=525.0, cx=319.5, cy=239.5, image_shape=rgb_image.shape[:2])colorize_points, sample_image_at_points, points_to_depth_image, and rgbd_to_points accept distortion= for raw (unrectified) images. Undistortion iterates to a tolerance (iterations=50, tolerance=1e-12 by default) and falls back to a damped Newton solve where OpenCV's fixed-point iteration does not converge. Points with no valid solution, or beyond the lens model's fold radius, come back as NaN. Distorted projection likewise marks points beyond the fold radius as not visible instead of wrapping them back into the image.
Crop/select helpers cover row masks, axis-aligned bounds, oriented 3D bounds, and geographic bounding boxes.
from arraydataengine.ops import crop_bounds, crop_geographic_bounds, crop_oriented_bounds, select_mask
valid_points = select_mask(points, valid_mask)
nearby_points = crop_bounds(points, min_bound=[-10, -10, -2], max_bound=[10, 10, 3])
vehicle_box = crop_oriented_bounds(points, center=pose_xyz, extent=[8.0, 4.0, 3.0], rotation=pose_rotation)
gps_window = crop_geographic_bounds(gps_samples, min_lat=36.9, min_lon=-122.3, max_lat=37.8, max_lon=-121.7)IMU, odometry, and NavSat arrays can be normalized into one trajectory representation with pose as [x, y, z, qx, qy, qz, qw] and trajectory as pose plus linear and angular velocity. Resampling uses SLERP for orientation and linear interpolation for position, velocity, acceleration, and covariance fields. Quaternion/Euler conversion, gravity compensation, bias correction, WGS84-to-local ENU/NED conversions, trajectory smoothing, differentiation, integration, dead reckoning, covariance propagation, and quality/status masks cover common navigation preprocessing. NavSat conversions use exact WGS84 geodetic/ECEF/ENU math and wrap longitudes across ±180°; the default local reference is the first finite sample with a valid fix. odometry_to_trajectory rotates the ROS body-frame twist into the odometry parent frame (pass twist_frame="parent" if yours is already world-frame), so dead_reckon_trajectory integrates it with its default world-frame velocity. IMU and odometry samples without an orientation (all-zero quaternion or orientation_covariance[0] == -1) become NaN rows instead of raising, and the nav helpers accept DataBuffer.get_buffer() topic arrays directly. pose_to_trajectory (or sensor_to_trajectory(kind="pose")) converts PoseStamped [x, y, z, qx, qy, qz, qw] topics.
from arraydataengine.ops import (
compensate_imu_gravity,
correct_imu_bias,
add_trajectory_quality_mask,
dead_reckon_trajectory,
differentiate_trajectory,
euler_to_quaternion,
imu_to_trajectory,
integrate_trajectory,
local_to_navsat,
mask_trajectory,
navsat_to_local,
navsat_to_trajectory,
odometry_to_trajectory,
quaternion_to_euler,
resample_imu,
resample_navsat,
resample_odometry,
resample_trajectory,
smooth_trajectory,
propagate_trajectory_covariance,
)
corrected_imu = correct_imu_bias(compensate_imu_gravity(window["/imu"]), sample_slice=slice(0, 50))
imu_traj = imu_to_trajectory(corrected_imu)
odom_traj = odometry_to_trajectory(window["/odom"])
gps_traj = navsat_to_trajectory(window["/gps"], ref_lat=37.0, ref_lon=-122.0, ref_alt=10.0)
gps_ned, gps_ref = navsat_to_local(window["/gps"], frame="ned", return_reference=True)
gps_roundtrip = local_to_navsat(gps_ned, gps_ref["lat"], gps_ref["lon"], gps_ref["alt"], frame=gps_ref["frame"])
orientation = euler_to_quaternion(roll=0.0, pitch=0.0, yaw=1.57)
roll_pitch_yaw = quaternion_to_euler(odom_traj["orientation"])
odom_50hz = resample_trajectory(odom_traj, period=0.02)
imu_at_image_times = resample_imu(window["/imu"], target_timestamps=image_timestamps)
odom_at_image_times = resample_odometry(window["/odom"], target_timestamps=image_timestamps)
gps_at_image_times = resample_navsat(window["/gps"], target_timestamps=image_timestamps, ref_lat=37.0, ref_lon=-122.0, ref_alt=10.0)
smoothed_odom = smooth_trajectory(odom_50hz, window_size=5)
derived_odom = differentiate_trajectory(smoothed_odom)
integrated_odom = integrate_trajectory(derived_odom, initial_position=odom_traj["position"][0])
dead_reckoned = dead_reckon_trajectory(odom_at_image_times, initial_position=odom_traj["position"][0]) # twist already rotated to world frame
covar_odom = propagate_trajectory_covariance(dead_reckoned, process_noise={"position": [0.02, 0.02, 0.05]})
quality_odom = add_trajectory_quality_mask(covar_odom, covariance_limits={"position": [1.0, 1.0, 2.0]})
trusted_odom = mask_trajectory(quality_odom, quality_odom["quality_mask"], drop=True)For large datasets, use the lazy topic pipeline. It records operations and only executes them when chunks, rows, reductions, windows, or explicit collection are requested.
import numpy as np
pipeline = (
buffer.topic("images")
.time_range(12.0, 20.0)
.index_range(0, 1_000)
.map(lambda frame: frame.astype(np.float32) / 255.0)
.filter(lambda frame, ts, name: frame.mean() > 0.05)
)
for chunk in pipeline.iter_chunks(chunk_size=32):
process(chunk.data, chunk.timestamps)
summary = pipeline.reduce(lambda acc, frame: acc + frame.mean(), initial=0.0)
small_result = pipeline.collect(chunk_size=32, max_rows=1_000)collect() is intentionally guarded. By default it refuses results above 512 MiB; pass max_rows, max_bytes, out=, or allow_large=True when materializing a large bounded result is intentional.
Independent topic chunks can be processed concurrently with max_workers. The parallel path preserves output order and requires stateful index_range() filters to be placed before row maps/filters so they can be pushed down before chunks are scheduled.
for chunk in pipeline.iter_chunks(chunk_size=32, max_workers=4):
process(chunk.data, chunk.timestamps)Use DataBuffer.dataset() when the query spans multiple topics. Topic, timestamp, and message-index constraints are applied before later row filters. On persistent Arrow- and TileDB-backed buffers, leading timestamp and index ranges are pushed down before payload arrays are read.
selection = (
buffer.dataset()
.select_topics("/camera/image", "/points", "/gps")
.time_range(12.0, 20.0)
.index_range(0, 10_000)
.frame_id("map")
)
for topic, chunk in selection.iter_chunks(chunk_size=64):
process(topic, chunk.data, chunk.timestamps)Use topic_workers when collecting independent topics concurrently. max_workers still controls per-topic chunk workers:
selected = selection.collect(chunk_size=64, max_workers=2, topic_workers=3, max_rows=30_000)Geographic bounds expect latitude/longitude values by default in columns (0, 1), matching NavSat arrays shaped like [latitude, longitude, altitude]. Spatial bounds work with XYZ vectors or point arrays and keep a message when any point falls inside the bounds.
gps_samples = (
buffer.dataset(["/gps"])
.geographic_bounds(36.9, -122.3, 37.8, -121.7)
.collect(max_rows=50_000)
)
point_chunks = (
buffer.dataset(["/points"])
.spatial_bounds(min_bound=[-20, -20, -2], max_bound=[20, 20, 5])
.iter_chunks(chunk_size=16)
)Topic alignment helpers cover exact timestamp joins, nearest-neighbor joins, bounded-tolerance joins, fixed-rate resampling, and rolling window joins.
from arraydataengine.ops import align_topic, resample_topic, rolling_window_join
imu_at_image_times = align_topic(images, imu, mode="bounded", tolerance=0.02)
gps_10hz = resample_topic(gps, rate_hz=10.0)
recent_points = rolling_window_join(images, points, seconds=0.25)TopicView is still available when the data is already in memory and you want eager, metadata-aware operations. It keeps message ids, timestamps, data, topic name, frame id, source URI, dtype, shape, and time bounds together while exposing the same operations as methods.
from arraydataengine.ops import topic_view
view = topic_view(window["images"], topic="images")
normalized = view.map(lambda frame: frame.astype(np.float32) / 255.0).as_dict()DataBuffer also keeps eager convenience wrappers for smaller results and compatibility:
normalized = buffer.map_topic("images", lambda frame: frame.astype("float32") / 255.0)
normalized = buffer.map_topic("images", lambda frame: frame.astype("float32") / 255.0, chunk_size=16)
recent_windows = list(buffer.window_topic("images", size=5))Row functions passed to map, filter, and reduce receive the row's data. They also receive its timestamp and message id as extra positional arguments, but only when they ask for them. That means either parameters named like a timestamp (ts, t, time, timestamp, stamp) and an id (id, message_id, name, idx, i), or a plain function or lambda of your own with two or three required positional parameters, such as lambda frame, ts: .... NumPy ufuncs, functools.partial objects, operator.itemgetter, and the library's own operations always receive just the data. Bind extra arguments explicitly: view.map(functools.partial(voxel_downsample, voxel_size=0.1)). Calling map(voxel_downsample) raises a TypeError instead of silently using a timestamp as the voxel size.
Variable-size rows, such as point clouds of different sizes, are stored zero-padded to a per-topic capacity together with each row's real length. Results then carry a lengths array next to the padded data. Row functions always receive trimmed rows, and TopicView.rows() / topic_rows(result) return them as a list.
ML-ready helpers turn lazy topic windows into plain iterator datasets, materialized NumPy datasets, or optional PyTorch datasets. They also provide deterministic split helpers, lightweight augmentations, and padding collation for mixed-rate windows and variable-size point clouds.
from arraydataengine.ops import (
augment_image,
augment_point_cloud,
collate_samples,
iter_ml_windows,
split_topic,
to_numpy_dataset,
to_torch_dataset,
)
image_windows = iter_ml_windows(buffer.dataset(["images"]), size=4, chunk_size=32)
numpy_windows = to_numpy_dataset(image_windows)
batch = numpy_windows.as_arrays(pad=True)
splits = split_topic(window["/gps"], by="time", fractions=(0.8, 0.1, 0.1))
augmented_image = augment_image(batch["data"][0], flip_horizontal=True)
point_window = next(iter(iter_ml_windows(buffer.dataset(["/points"]), size=1)))
augmented_points = augment_point_cloud(point_window["data"][0], jitter_std=0.01, seed=7)
collated = collate_samples([{"points": augmented_points}, {"points": augmented_points[:128]}])
torch_dataset = to_torch_dataset(iter_ml_windows(buffer.dataset(["images"]), size=2), iterable=True)Use source_pipeline() when you want operations to run while messages stream from a DataSources object, before full topics are loaded. The same pipeline can write to an in-memory DataBuffer or persist through the default MCAP backend or explicitly selected Arrow/Parquet and TileDB backends. Long-running source and topic pipelines accept progress callbacks, cancellation tokens, and mutable checkpoint dictionaries that can be saved and reused to resume from the last processed row. Checkpoints only advance past rows that were actually delivered. A cancelled collect() attaches the rows gathered so far as PipelineCancelled.partial, so partial plus a resumed collect() equals a full run. When resuming a persistent store, the store's per-topic counts are authoritative: after a hard crash, rows the checkpoint claimed but never flushed are replayed from the source, and rows already stored are never appended twice.
import json
from functools import partial
from arraydataengine.ops import CancellationToken, PipelineCancelled, source_pipeline, voxel_downsample
from arraydataengine.source import DataSources
source = DataSources("/data/rosbag2/split_recording/")
checkpoint = {}
persistent_checkpoint = {}
cancel_token = CancellationToken()
def report(progress):
print(progress.processed, progress.emitted, progress.topic)
pipeline = (
source_pipeline(source)
.select_topics("/points")
.time_range(12.0, 20.0)
.map(partial(voxel_downsample, voxel_size=0.1)) # scans keep their own sizes
.filter(lambda msg: msg["data"].shape[0] > 0)
)
try:
rolling_buffer = pipeline.to_buffer(
buffer_depth=128,
progress_callback=report,
cancel_token=cancel_token,
checkpoint=checkpoint,
progress_interval=100,
)
except PipelineCancelled:
with open("filtered_points.checkpoint.json", "w") as checkpoint_file:
json.dump(checkpoint, checkpoint_file)
persistent_buffer = pipeline.to_buffer(
data_uri="/tmp/ade/filtered_points/",
use_db=True,
checkpoint=persistent_checkpoint,
)Persistent output uses MCAP by default. Select backend="arrow" for Parquet analysis stores. Pass backend="tiledb" to to_buffer(), or use the persist_to_tiledb() convenience method, when TileDB is preferred. Persistent source pipelines require source.get_count(topic); TileDB uses that count to size its dense destination arrays, while all persistent backends record the actual filtered message count in store metadata.
Initial operation coverage includes source-level streaming pipelines, progress reporting, cancellation, resumable checkpoints, topic selection, map/filter/reduce/window helpers, nearest-time alignment, SE(3) transforms, frame graphs, camera projection helpers, camera intrinsics/distortion/rectification utilities, mask and bounds cropping, point cloud downsampling/sampling/KNN-radius-hybrid search/normals/covariance descriptors/distance stats/outlier filters/clustering/connected components/plane and ground segmentation/ICP registration/Open3D adapters, image/depth sequence transforms, morphology, gradients, pyramids, local image statistics, frame-to-frame optical flow, image alignment, motion-compensated rolling windows, valid-depth masks, depth backprojection, depth normals, RGB-D fusion, navsat ENU/NED conversion, quaternion/Euler conversion, gravity compensation, bias correction, trajectory resampling/smoothing/differentiation/integration/dead reckoning/covariance propagation/quality masks, trajectory speed, ML-ready iterators/NumPy/PyTorch adapters/splits/augmentations/collation, and DEM/raster helpers for terrain gradients, normals, roughness, traversability, local patches, point clouds, and meshes.
Set use_db=True to persist messages to disk. This is intended for full-source ingest and larger-than-memory datasets. Three storage engines are available via the backend parameter:
"mcap"(the default for new stores): append-only MCAP segments with an atomic manifest, resumable ingestion, original message metadata, and variable array shapes."arrow"(explicit analysis storage): Apache Arrow / Parquet fragment files. Fast, compressed, dependency-light (pip install "arraydataengine[arrow]"), and every persisted topic is directly readable by Polars, DuckDB, pandas, and anything else that speaks Parquet."tiledb": TileDB dense arrays (pip install "arraydataengine[tiledb]"). Supports in-place cell updates (buffer[i] = ...), which the immutable Arrow fragments do not.
When backend is omitted and use_db=True, new stores use MCAP; existing Arrow and TileDB stores retain their backend.
from arraydataengine.buffer import DataBuffer
from arraydataengine.source import DataSources
axis = "/camera/image"
source = DataSources("/data/rosbag2/split_recording/")
with DataBuffer(
data_source=source,
data_uri="/data/stores/my_dataset/",
axis=axis,
use_db=True, # backend="mcap" is the default for new stores
) as buffer:
buffer.load_data_db(axis)
print(buffer.get_group_uri())Using DataBuffer as a context manager closes the store cleanly and marks completed topics as closed; close_completed() does the same without leaving the context. Topics that are still incomplete stay open, so a later run resumes them. Memory, Arrow, and TileDB topics each have one payload dtype and trailing shape; MCAP permits changes in both. Safe widening casts (for example int32 into an int64 topic) are applied; anything else raises ValueError rather than being silently broadcast or cast. Point-cloud topics, and any topic listed in DataBuffer(ragged_topics={topic: capacity}), may vary in their first dimension. On memory, Arrow, and TileDB they are stored zero-padded to a capacity, set by DataBuffer(max_points=...) or the source's max_points and defaulting to 30000 rows, with each message's real length. Reads return lengths next to the padded data, buf[i] returns the trimmed row, and topic(...).rows() returns trimmed rows. DataBuffer(topics=[...]) limits which topics are buffered, the in-memory ring grows as messages arrive, and get_buffer() returns only filled rows in time order. get_size() is the number of stored rows on every backend. Arrow and TileDB stage rows in memory and reads include staged rows without flushing them. MCAP finalizes its active segment before reading; it preserves native array shapes and pads only eager multi-message results when necessary. Call close() (or use the context manager) so the last batch is written; after a crash, a resumed ingest replays only the rows that were never flushed.
Arrow and TileDB stores use escaped topic directory names so topics such as /a/b and /a_b remain distinct. Existing stores retain their original directories when reopened. Query results expose each topic's location as source_uri.
Arrow and TileDB keep timestamps, message names, frame ids, and per-message spatial bounds alongside the data. Time, index, frame-id, and spatial-bounds constraints from lazy topic or dataset queries are pushed down before data chunks are read, so pipelines avoid loading unselected message payloads. MCAP reads sequentially and evaluates these filters after decoding each message. Streaming payload batches avoid loading complete topics; eager APIs still materialize their selection.
The Arrow backend exposes its tuning knobs through backend_options (reasonable defaults shown):
buffer = DataBuffer(
data_source=source,
data_uri="/data/stores/my_dataset/",
axis=axis,
use_db=True,
backend="arrow",
backend_options={
"flush_bytes": 32 * 1024 * 1024, # staged bytes per topic before a fragment is written
"memory_budget_bytes": 64 * 1024 * 1024, # shared staging limit across all topics
"row_group_bytes": 16 * 1024 * 1024, # target Parquet row-group size (random-access granularity)
"compression": "zstd", # any Parquet codec: zstd, snappy, lz4, none...
"batch_readahead": 1, # scanner prefetch depth; raise for throughput,
"fragment_readahead": 1, # keep at 1 for minimal scan memory
"use_threads": False, # parallel scans (off preserves order + bounds memory)
"row_overhead_bytes": 256, # per-row bookkeeping counted toward flush_bytes
},
)The TileDB backend accepts flush_bytes, row_overhead_bytes, and consolidate_max_bytes the same way. Closing a TileDB store consolidates only what is safe while other processes read it (this session's index fragments and the metadata). Run buffer.compact() when no reader has the store open to merge payload fragments (arrays up to consolidate_max_bytes) and vacuum superseded files. Unknown backend_options raise ValueError for every backend.
Existing stores can be reopened without the original source:
reopened = DataBuffer(
data_source=None,
data_uri="/data/stores/my_dataset/",
axis=axis,
use_db=True,
)
for chunk in reopened.topic(axis).time_range(12.0, 20.0).iter_chunks(chunk_size=64):
process(chunk.data)Interrupted ingests can be resumed by constructing a new DataBuffer with the same source and data_uri. Stored per-topic counts are loaded from the store's metadata, previously written messages are skipped during source replay, and remaining messages are appended.
DEM sources use NASA Earthdata credentials from environment variables:
export earthdata_username="..."
export earthdata_password="..."Then request north/west tile ranges:
from arraydataengine.source import DataSources
north = [37, 39]
west = [122, 124]
source = DataSources("DEM", bounds=[north, west])
for tile in source.get_message():
print(tile["name"], tile["data"].shape)Pass cache_dir to reuse downloaded HGT payloads. Cached tiles can be read without Earthdata credentials:
source = DataSources("DEM", bounds=[north, west], cache_dir="/tmp/ade-dem-cache")Downloads require HTTPS, and Earthdata credentials are only sent to the Earthdata login host (urs.earthdata.nasa.gov) or to the tile host itself; redirects to any other host never receive them. Tiles come from NASA LP DAAC's cloud archive by default; base_url (or ADE_DEM_BASE_URL) can point at a mirror, either a directory of <name>.SRTMGL1.hgt.zip files or a per-tile template containing {name}. A custom http:// base URL requires an explicit allow_insecure_http=True. Tiles are validated before they are cached, cache writes are atomic, a missing (404) tile such as open ocean is skipped with a warning, and tiles are yielded as native, writable int16 arrays.
DEM helper functions operate on NumPy windows, so they can be used on individual tiles, cropped patches, or lazy pipeline chunks:
from arraydataengine.ops import (
dem_to_mesh,
dem_to_point_cloud,
mosaic_dem_tiles,
read_dem_cache,
reproject_raster,
resample_raster,
roughness_map,
sample_elevation,
terrain_normals,
terrain_patch,
traversability_map,
write_dem_cache,
)
tile = next(source.get_message())
elevation = mosaic_dem_tiles([tile]) # float64, north-up, voids -> NaN, SRTM seams merged
lat0, lon0 = north[0], -west[0] # south-west corner; west longitudes are negative
normals = terrain_normals(elevation, resolution=30.0, north_up=True)
roughness = roughness_map(elevation, window_size=5)
traversability = traversability_map(
elevation,
resolution=30.0,
max_slope_degrees=25.0,
max_roughness=1.0,
)
vehicle_patch = terrain_patch(elevation, center=(150.0, 240.0), size=(64, 64), resolution=30.0, north_up=True)
vehicle_elevation = sample_elevation(elevation, x=[150.0], y=[240.0], resolution=30.0, north_up=True)
terrain_points = dem_to_point_cloud(vehicle_patch, resolution=30.0, north_up=True)
terrain_mesh = dem_to_mesh(vehicle_patch, resolution=30.0, north_up=True)
overview = resample_raster(elevation, shape=(512, 512))
local_grid = reproject_raster(
elevation,
src_bounds=(lon0, lat0, lon0 + 1.0, lat0 + 1.0),
dst_bounds=(lon0 + 0.25, lat0 + 0.25, lon0 + 0.75, lat0 + 0.75),
shape=(256, 256),
north_up=True,
)
write_dem_cache("/tmp/ade-dem-cache", tile["name"], elevation, metadata={"bounds": [lon0, lat0, lon0 + 1.0, lat0 + 1.0]})
cached_elevation, metadata = read_dem_cache("/tmp/ade-dem-cache", tile["name"], return_metadata=True)mosaic_dem_tiles also accepts {name: raster} mappings or (name, raster) pairs and fills sparse regions when adjacent SRTM tiles are missing. Adjacent SRTM tiles share one edge row/column, so tiles named like SRTM tiles (N37W122) with the standard 1201 or 3601 samples are merged with overlap=1 automatically; pass overlap= explicitly for other rasters. SRTM voids (-32768) become fill_value (NaN by default).
DEM rasters decoded from HGT tiles are north-up (row 0 is the northern edge). Pass north_up=True to the helpers that map rows to y: slope_aspect, hillshade, terrain_gradients, terrain_normals, terrain_patch, sample_elevation, sample_elevation_at_navsat, dem_to_point_cloud, dem_to_mesh, reproject_raster, dem_grid_to_points, transform_dem_grid, and project_dem_to_image. For geographic (lat/lon) rasters such as SRTM mosaics, sample with sample_elevation_at_latlon(raster, lat, lon, bounds=(west, south, east, north)) or the index from mosaic_dem_tiles(..., return_index=True), or pass bounds=/index= to sample_elevation_at_navsat. Local ENU coordinates do not map onto a lat/lon grid exactly. The default, north_up=False, treats increasing row index as +y. In both conventions origin is the south-west cell, aspect is a compass bearing in [0, 2π) with NaN for flat cells, and sample_elevation returns NaN for points outside the grid. resolution also accepts (dx, dy), because the east-west cell size of a geographic tile shrinks with latitude.
Run the source adapter, synthetic operation, lazy pipeline, and storage backend benchmarks with:
python -m pytest tests/test_source_benchmarks.py -q -sResults below were measured on 2026-07-28 with Python 3.14 on arm64 (Apple Silicon), NumPy 2.4.6, pyarrow 25.0.0, tiledb 0.36.1. Each result is the best of three runs. Bag, DB3, and DEM use mocked readers/network responses, so those rows measure adapter overhead without requiring ROS bag files or Earthdata access.
| Benchmark | Workload | Items | Elapsed | Throughput | Latency |
|---|---|---|---|---|---|
ImgSource.messages |
temporary 64x64 PNG files read through OpenCV | 200 | 0.006641s | 30,114 items/s | 33.2 us/item |
BagSource.messages |
mocked AnyReader and image sensor conversion |
200 | 0.000168s | 1,191,363 items/s | 0.8 us/item |
DB3Source.messages |
mocked AnyReader and image sensor conversion |
200 | 0.000275s | 727,603 items/s | 1.4 us/item |
DEMSource.messages |
mocked Earthdata zip response with 32x32 HGT tiles | 4 | 0.000074s | 54,237 items/s | 18.4 us/item |
ImageOps.synthetic |
resize, normalize, and grayscale 96x96 RGB image frames | 64 | 0.001132s | 56,543 frames/s | 17.7 us/frame |
PointCloudOps.synthetic |
voxel downsample, KNN search, and normal estimation on XYZ+intensity points | 1,024 | 0.010418s | 98,287 points/s | 10.2 us/point |
IMUOps.synthetic |
resample synthetic IMU samples and dead-reckon the trajectory | 2,000 | 0.026483s | 75,520 samples/s | 13.2 us/sample |
OdometryOps.synthetic |
resample synthetic odometry samples and dead-reckon the trajectory | 2,000 | 0.027236s | 73,433 samples/s | 13.6 us/sample |
NavSatOps.synthetic |
convert and resample synthetic WGS84 NavSat samples into local trajectory arrays | 2,000 | 0.012295s | 162,666 samples/s | 6.1 us/sample |
DEMOps.synthetic |
terrain normals, roughness, traversability, and point-cloud conversion over a DEM grid | 16,384 | 0.005833s | 2,808,726 cells/s | 0.4 us/cell |
TopicPipeline.iter_chunks |
lazy in-memory time/index pushdown and row map over 50k synthetic samples | 10,000 | 0.035091s | 284,972 items/s | 3.5 us/item |
TileDB.TopicPipeline.time_range |
lazy TileDB time-range pushdown and row map over a temp persisted topic | 100 | 0.205731s | 486 items/s | 2057.3 us/item |
Arrow.TopicPipeline.time_range |
same pipeline over the Arrow/Parquet backend | 100 | 0.001433s | 69,796 items/s | 14.3 us/item |
The TileDB rows above predate the batched, compressed TileDB writer; see Storage backends for current backend measurements.
voxel_downsample groups 1M points at 0.5 m resolution in ~70 ms (packed int64 voxel keys; ~7-9x faster than the previous row-wise grouping).
Identical workload — 1,500 lidar-sized point clouds of shape (30000, 3) float64, 1.08 GB of data — through each persistent backend, reads in a fresh process. Measured on 2026-09-27 with Python 3.12.3 on Linux x86_64 (Intel i9-13900H, NVMe), NumPy 2.3.2, pyarrow 22.0.0, tiledb 0.36.1; best of two runs:
| Operation | TileDB | Arrow |
|---|---|---|
| Ingest (excluding data generation) | 6.5 s | 7.1 s |
| On-disk footprint | 1,031 MB | 923 MB |
| Full 1.1 GB chunked scan | 0.88 s | 1.38 s |
| Time-range read (151 messages) | 63 ms | 234 ms |
| Single-message fetch | 6 ms | 33 ms |
| Peak RSS, ingest / reads | 246 MB / 360 MB | 269 MB / 565 MB |
The benchmark compares TileDB and Arrow, which stage writes with a 32 MiB per-topic flush target. Arrow now also enforces a 64 MiB shared staging budget. Arrow scans use the default batch_readahead/fragment_readahead of 1 (raising readahead trades memory for throughput). This workload is nearly incompressible; real sensor streams compress further under both backends' Zstd codecs. MCAP is now the default recording backend; Arrow remains available for Parquet analysis with Polars, DuckDB, and pandas. MCAP was not part of this benchmark.
Run the test suite:
python -m pytest -qRelease packaging and Python registry publishing steps are tracked in TODO.md.
Run the notebooks (each in an isolated copy, so nothing they write lands in the repository; needs pip install nbclient):
python scripts/run_notebooks.py # all notebooks, offline defaults
python scripts/run_notebooks.py --save # refresh the committed outputsTo test them against real recordings, fetch the test datasets (about 700 MB, stored in ~/.cache/arraydataengine/test-data, verified against pinned checksums) and run the real profile:
python scripts/datasets/fetch_test_datasets.py
python scripts/run_notebooks.py --profile realThe datasets are the KITTI raw drive 2011_09_26_drive_0001 (GPS/IMU, Velodyne, and camera; CC BY-NC-SA 3.0), converted to ROS 2 bags in both SQLite and MCAP storage by scripts/datasets/kitti_to_rosbag2.py. They also include MOT17-05 (CC BY-NC-SA 3.0), San Francisco Bay Area SRTM-format elevation tiles from the AWS Open Data terrain-tiles bucket (SRTM and USGS 3DEP with ETOPO1 bathymetry offshore; public domain, attribution requested), CIFAR-10, and synthetic FLIR Boson LWIR sequences with blinking beacons from scripts/datasets/make_lwir_sequence.py, because no open thermal dataset contains pulsed beacons. The ros1 profile replays the example recording's ROS 1 bag. The dem-download profile exercises DEMSource's download branch against a local server holding the real tiles. The earthdata-mock profile runs the notebooks' default Earthdata download path, including the login redirect, session cookie, and pre-signed S3 hand-off, against the synthetic Earthdata of scripts/datasets/mock_earthdata.py (python scripts/datasets/mock_earthdata.py --check runs the same flow through the library directly). scripts/datasets/check_earthdata_download.py checks the live service with your own credentials.
Build and run the notebook container:
./build.sh
docker compose up jupyterJupyter uses JUPYTER_TOKEN from your environment when it is set; otherwise it
generates a token and prints the login URL (docker compose logs jupyter).
Notebook examples live in notebooks/. They run offline out of the box on the example recording (example/mapeverything_0.db3), SyntheticSource, or seeded synthetic data, and each has a configuration cell near the top whose ADE_* environment variables point it at your own recordings, image folders, or DEM tiles. Files they generate go under __-prefixed paths, which git ignores.
| Notebook | Shows |
|---|---|
demo.ipynb |
Tour of the library: lidar ICP and mapping, image sequences, INS trajectories, DEM terrain, IR beacon detection |
circular_buffer.ipynb |
Rolling DataBuffer windows, time and index slices, variable-size scans |
iterator_example.ipynb |
Reading messages, get_data, lazy topic and source pipelines, alignment |
bag_to_TileDB.ipynb |
Persisting a recording to TileDB, reopening it, and training a small PyTorch model from it |
loam_example.ipynb |
Scan-to-scan and scan-to-map ICP against the recorded poses, HTML map viewer |
ins.ipynb |
IMU/odometry/NavSat trajectories, gravity and bias correction, dead reckoning |
mot.ipynb |
Segmentation and IoU tracking on a synthetic video, CLEAR-MOT scores |
dem.ipynb |
SRTM tiles (offline synthetic cache by default), mosaics, terrain analysis, lat/lon sampling |
tiff.ipynb, pulsed_encoded_beacons.ipynb |
Thermal (LWIR) frame registration and blinking-beacon detection by frequency |
mlp.ipynb |
A small PyTorch classifier fed through the ML helpers (CIFAR-10 optional) |