Skip to content

About

Analysis and generator for cyber-physical systems data

Topics

Resources

Stars

1 star

Watchers

1 watching

Forks

Repository files navigation

Array Data Engine logo

Array Data Engine

PyPI tests License: MIT

Array Data Engine is a Python package for turning heterogeneous sensor and array data into a consistent NumPy-first stream. It can read image sequences, ROS bag files, ROS 2 .db3 recordings or split rosbag2 directories, and DEM tiles, then keep recent context in memory or persist complete streams to MCAP (the default), Apache Arrow/Parquet, or TileDB.

The project is aimed at robotics and perception workflows where algorithms need synchronized windows of image, point cloud, navigation, odometry, IMU, and terrain-like array data.

Features

  • NumPy-oriented message dictionaries with data, timestamp, topic, and name fields.
  • Source adapters for image folders, .bag, .db3, .mcap, split rosbag2 directories, and DEM data — including bags recorded through rosbridge/foxglove with JSON payloads.
  • ade command-line interface: inspect, export, ingest, and visualize bags without writing code.
  • Synthetic data source with a geometrically consistent trajectory for demos and tests — no data files needed.
  • Topic statistics (describe_topic/describe_dataset) and portable .npz topic round-trips.
  • TUM and KITTI trajectory export (compatible with evo and SLAM evaluation tooling).
  • Cached ROS topic metadata, so repeated topic and count lookups do not reopen the same bag.
  • Rolling in-memory buffers for recent context windows.
  • Persistent storage for larger-than-memory datasets: MCAP segments (default), Apache Arrow/Parquet analysis fragments, or TileDB dense arrays. Parquet and TileDB support metadata query pushdown.
  • Lazy dataset-level selection by topic, time, message index, frame id, geographic bounds, and spatial bounds.
  • Lazy optional imports so lightweight workflows do not need the full ROS, DEM, or visualization stack.
  • Notebook examples for demos, iterators, terrain, INS, MOT, and TileDB workflows.

Installation

python -m pip install arraydataengine

The distribution and canonical import name are both arraydataengine:

from arraydataengine import DataBuffer, DataSources

The legacy-cased ArrayDataEngine package remains available as a convenience facade:

import ArrayDataEngine as ADE

For editable local development:

python -m pip install -e .

The base install includes NumPy and MCAP. Install feature extras for the source adapters and storage backends you use:

Extra Adds
arrow Parquet analysis caches and explicit Arrow storage
mcap Compatibility extra; MCAP is included in the base install
tiledb Alternative TileDB persistent storage
image Image sequence loading and image-processing dependencies
ros rosbags reader for ROS 1 bags, MCAP, and split rosbag2 directories
dem DEM downloads and raster processing
visualization Matplotlib, Open3D, OpenCV, Pillow, and IPython visualization support
ml PyTorch and model/dataset integrations
notebook JupyterLab
dev Test tooling

Extras can be combined for either a registry or editable install:

python -m pip install "arraydataengine[ros,arrow]"
python -m pip install -e ".[dev,image,ros,arrow]"

The full Docker/notebook environment still uses requirements.txt, which includes the heavier ROS, ML, notebook, and visualization dependencies.

Command line

Installing the package also installs the ade command:

ade demo -o demo/                 # end-to-end synthetic showcase, no data needed:
                                  # stitched point-cloud map (HTML), TUM trajectory, NPZ export
ade info my_bag.db3 --messages 500   # topics, counts, duration, rates, jitter
ade topics my_bag.db3
ade export my_bag.db3 -t /points -o points.npz   # or -o exports/ for exports/points.npz
ade viewer my_bag.db3 -t /points -o viewer.html --stride 5
ade ingest my_bag.db3 -o /data/stores/my_bag/  # persist every topic (mcap by default, --backend arrow for analysis)

ade viewer and ade demo write self-contained HTML files that render in any browser with no extra dependencies. Exported .npz files load back with arraydataengine.ops.load_topic_npz and plug straight into the ops pipelines. Standalone SQLite recordings (.db3, or a single SQLite .bag) are read with the built-in decoder and need no extras; ROS 1 bags, MCAP files, and split rosbag2 directories require the ros extra. ade ingest also requires the extra for its storage backend: new stores default to mcap, while an existing output store is reopened with the backend it was written with. A --backend that doesn't match an existing store, and any non-empty directory that isn't a store, are refused.

Point clouds of any size are read by default. --max-points N (on info, topics, export, viewer, and ingest) skips clouds with more than N points, and the command says so when every message of a topic was skipped. ade export zero-pads variable-size clouds to the largest exported cloud and stores the real row counts in the archive's lengths array. ade ingest stores up to --max-points rows per cloud (30000 when omitted). It records the recording it read in the store's ade_source.json, and refuses to resume the store from a different recording or with a different --max-points (--force appends anyway). ade info also lists topics it cannot decode and the extra they need. Missing, empty, corrupt, or Git LFS pointer recordings, unsupported message types, unwritable outputs, and missing extras are reported as one-line errors (exit status 2; Ctrl-C exits with 130). Pass --debug (before or after the subcommand) or set ADE_DEBUG=1 for a full traceback.

Supported Sources

Input Adapter Topic(s) Notes
*.png, *.jpg, *.jpeg, *.tif, *.tiff (any case) ImgSource images Reads naturally-sorted image paths from a glob at their native bit depth (16-bit thermal frames stay uint16); each page of a multi-page TIFF stack is one message (name is <file>#<page>) (a literal path containing [ or * also works). With flir="raw" or flir="celsius" (or "kelvin"/"fahrenheit"), FLIR radiometric JPEGs yield their sensor counts or temperatures instead of the palette image. Colour images are RGB; pass color_order="bgr" for OpenCV's order.
.bag BagSource Bag topics ROS1 bags with image, compressed image, point cloud, IMU, odometry, navsat, pose, and DepthAnything calibration messages. SQLite rosbag2 files named .bag are routed to DB3Source.
.db3/.mcap file or rosbag2 directory DB3Source Bag topics Single and split ROS 2 bags (sqlite or MCAP storage) with image, compressed image, point cloud, IMU, odometry, navsat, pose, and DepthAnything calibration messages. Bags without embedded message definitions (Humble and older) are supported. rosbridge/foxglove JSON payloads are decoded for PoseStamped, NavSatFix, PointCloud2, CompressedImage, and DepthAnything calibration messages; the serialization format recorded in the .db3 takes precedence over metadata.yaml. compressed_depth_image_transport (compressedDepth) images are decoded too.
"DEM" DEMSource images Downloads SRTM HGT tiles over HTTPS using Earthdata credentials.
SyntheticSource (import directly) configurable Deterministic synthetic IMU/odometry/point-cloud/navsat/image streams on a circular trajectory.

Each yielded message has this shape:

{
    "data": np.ndarray,
    "timestamp": 123.456,
    "topic": "images",
    "name": "frame_000.png",
}

ROS messages also carry frame_id (from the message header). The name of a ROS message is its short type name, such as PointCloud2. Point clouds drop non-finite points and are yielded at their real size as an (N, 3) array in the field dtype, with point_count (= N) on the message. Pass DataSources(path, max_points=...) to skip scans larger than a limit (there is none by default). Messages that fail to decode are skipped with a logged warning. Every source yields colour images as RGB (or RGBA), and image messages carry an encoding key such as rgb8, mono16, or 32FC1. NavSatFix messages also carry status (-1 means no fix) and a 3×3 position_covariance. get_topics() and get_count() only report topics whose message type can be decoded, and DataSources(path, topics=[...]) decodes only the topics you name.

Quick Start

Read an image sequence directly:

from arraydataengine.source import DataSources

source = DataSources("/data/frames/*.png", period=0.1)

print(source.get_topics())        # ["images"]
print(source.get_count("images")) # number of matched frames

for message in source.get_message():
    image = message["data"]
    timestamp = message["timestamp"]
    print(timestamp, image.shape)

Read a ROS bag, ROS 2 database, or split rosbag2 directory:

from arraydataengine.source import DataSources

bag_source = DataSources("/data/recording.bag")
ros2_source = DataSources("/data/rosbag2/split_recording/")

for topic in ros2_source.get_topics():
    print(topic, ros2_source.get_count(topic))

For split ROS 2 bags, pass the directory that contains metadata.yaml and the chunk files. Passing one .db3 chunk that metadata.yaml lists still reads the whole recording; a standalone SQLite file is read on its own. A directory of .db3 files without metadata.yaml is read chunk by chunk in natural filename order. Reading ROS .mcap files uses the ros extra; ADE recordings use the mcap extra described below.

Live recording to MCAP

Install pip install "arraydataengine[mcap]" to record ADE message dictionaries as they arrive. Supply a generator from your device, ROS subscription queue, or other capture code; ADE does not open live devices or subscribe to ROS itself. Here is an example using the synthetic source:

from arraydataengine import DataSources, McapRecorder
from arraydataengine.sources.synthetic_source import SyntheticSource

with McapRecorder("session.mcap") as recorder:
    recorder.record(SyntheticSource(duration=2.0).get_message())

# For live capture, call recorder.write(message) for each incoming message:
# {"topic": "/imu", "timestamp": seconds, "name": "Imu", "data": numpy_array}
# recorder.flush()              # write the current chunk; keep recording
# recorder.flush(durable=True)  # also fsync the file

source = DataSources("session.mcap")
for message in source.get_message():
    process(message)  # yields one message at a time in recorded file order

record(iterable) consumes the iterable synchronously and returns the number of messages written when it ends. An unbounded generator records until it is stopped; a context manager finalizes the file on normal exit or a Python exception (including Ctrl-C). write(message) calls are serialized, allowing callbacks to share one recorder. Capture queues and backpressure belong to the caller. Existing files are refused unless overwrite=True; appending to an existing MCAP is not supported.

MCAP stores topic channels, timestamps, indexes, and compressed chunks. The recorder defaults to Zstd and a 1 MiB chunk target; compression="lz4" or "none" and chunk_size=... are also supported. A message can exceed the chunk target. Pending data is flushed on close or explicit flush(); each write() is not immediately durable. The writer also retains chunk indexes until close, so index memory grows with recording length. Abrupt process termination can leave an unfinished file; automatic crash recovery is not provided.

The ADE arraydataengine/numpy/v1 profile supports heterogeneous channels. write(message) stores its payload as a pickle-free NumPy array, preserving dtype and shape without padding variable-size scans. Other fields (such as name, frame_id, encoding, and point_count) use JSON; NumPy metadata arrays/scalars become lists/scalars. Object arrays and metadata that cannot be encoded as JSON are rejected. timestamp is non-negative seconds; optional integer timestamp_ns and publish_time_ns preserve exact MCAP nanosecond timestamps, and sequence is an optional uint32 counter.

Use write_encoded to keep original serialized messages alongside NumPy channels. For example, in a ROS callback with serialized CDR bytes and the message's full ROS definition (including dependencies):

recorder.write_encoded(
    "/imu", cdr_bytes,
    timestamp_ns=log_time_ns,
    message_encoding="cdr",
    schema_name="sensor_msgs/msg/Imu",
    schema_encoding="ros2msg",
    schema_data=full_ros_definition.encode("utf-8"),
)

CDR means Common Data Representation, the binary serialization used for ROS 2 messages. The generator selects a decoder per channel and schema: ADE NumPy and the project's supported ROS 2/CDR sensor types work automatically, even in one file. Install arraydataengine[mcap,ros] for schema-based ROS decoding, plus image for image types that need OpenCV. Arbitrary custom ROS types still need an ADE array conversion. ROS/Foxglove tools need a decoder for the ade-numpy-v1 channels; mixed ADE recordings are not standard ROS-only bags.

For additional encodings or custom types, pass DataSources(path, mcap_decoders={encoding: decoder}) or use {(encoding, schema_name): decoder} for a schema-specific override. Each decoder receives MCAP (schema, channel, record) objects and returns a dictionary containing data and optional metadata. The reader fills missing topic, name, timestamps, and sequence information. Unknown encodings raise an error; select supported topics with topics= or supply a decoder. Passing mcap_decoders={} selects the generic reader for an external MCAP; ordinary ROS MCAP files retain the existing ros reader by default.

No Parquet conversion is required. Pass a completed recording to DataSources, DataBuffer, or source_pipeline for playback and processing. Playback uses a lazy generator without sorting or loading the whole recording, runs at the consumer's pace, and does not tail a growing file or reproduce wall-clock timing. MCAP is also the default persistent DataBuffer backend. Existing Arrow and TileDB stores are detected automatically; use_db=False still selects memory.

Parquet analysis and trajectory loop closure

New persistent DataBuffer(..., use_db=True) stores and ade ingest default to MCAP. A store is a directory of finalized MCAP segments and an atomic mcap_store.json manifest. Interrupted, uncommitted segments are ignored on resume; keep the same source ordering when replaying. One process may append at a time. Reads flush the current segment before streaming. Eager reads such as get_buffer() materialize their selection; use buffer.iter_topic_chunks(topic, chunk_size=64) for payload batches bounded by the MCAP memory budget. Topic pipelines may regroup those rows up to their requested chunk size. Sensor capture and backpressure remain the caller's responsibility. Native ingestion records the decoded ADE messages; use McapRecorder.write_encoded() to retain original ROS wire payloads.

MCAP backend options include chunk_size (1 MiB), segment_bytes (64 MiB), compression (zstd, lz4, none), and memory_budget_bytes (64 MiB). The budget limits read batches and rejects oversized individual messages; it is not a total process RSS limit. Arrow's memory_budget_bytes (64 MiB) limits staging across all topics, flushing the largest topic when needed.

Use a derived Parquet cache for selective analysis. It separates changes in array shape, dtype, topic, and source schema into partitions, without padding ragged scans. Exact log/publish nanoseconds, sequence, original record ordinal, and per-topic ordinal are stored alongside a pickle-free NumPy message envelope. Trajectory topics additionally expose XYZ and quaternion scalar columns for filtering; other array payloads remain binary envelopes, not flattened SQL columns.

from arraydataengine import mcap_to_parquet, parquet_to_mcap

cache = mcap_to_parquet(
    "session.mcap", "analysis/",
    topics=["/pose", "/points"],
    trajectory_topics={"/pose": "pose"},  # "odometry" for Odometry arrays
    memory_budget_bytes=64 * 1024**2,
    row_group_bytes=8 * 1024**2,
    fidelity="copy",
)
parquet_to_mcap(cache, "restored.mcap")

Install arraydataengine[arrow] for these operations. fidelity="reference" (the default) retains SHA-256-checked references to the original files; fidelity="copy" also archives the originals inside the cache. Restoration copies the originals byte-for-byte, including raw CDR, schemas, attachments, metadata, and message order—even channels omitted from analysis. It does not apply edits made to derived Parquet columns. Reference restoration requires unchanged source files. Cache reuse verifies hashes and conversion options; changed sources/options require a fresh destination. New caches are published only after successful conversion. For very large recordings, hash verification adds a sequential I/O pass. Finalize recording before conversion.

Loop closure on a recording automatically uses this Parquet path:

from arraydataengine.ops import iter_recording_loop_closures

for closure in iter_recording_loop_closures(
    "session.mcap",
    trajectory_topic="/pose",
    points_topic="/points",
    trajectory_kind="pose",
    radius=3.0,
    min_separation=60,
    time_tolerance=0.05,
    voxel_size=0.25,
    cache_dir="loop-analysis/",
):
    process(closure)

Candidate queries read only pose metadata; ICP loads the current scan and one candidate scan. Clouds are matched by nearest timestamp within the explicit tolerance (time_tolerance above); missing matches raise an error. No entire cloud sequence or trajectory spatial index is held in memory. Predicate scans can still be expensive on long, poorly clustered trajectories. The budget controls conversion staging, not ICP working memory. Omitting cache_dir uses a temporary cache cleaned up when the iterator is exhausted or closed. verify_recording_loop_closures() collects the results into arrays; use the iterator when results themselves may exceed memory. Existing NumPy-based verify_loop_closures() remains available for in-memory inputs.

Rolling Buffers

DataBuffer keeps a recent window for each topic and exposes NumPy-like indexing on the selected axis topic.

from arraydataengine.buffer import DataBuffer
from arraydataengine.source import DataSources

axis = "images"
source = DataSources("/data/frames/*.png", period=0.1)

buffer = DataBuffer(
    data_source=source,
    buffer_depth=10,
    axis=axis,
    use_db=False,
)

# The constructor preloads one axis message by default.
buffer.roll_buffer(axis)
latest_image = buffer[-1]

window = buffer.get_buffer()
image_window = window[axis]["data"]
timestamps = window[axis]["ts"]

Use copy=False when you want to avoid extra in-memory copies during read-only inspection:

window = buffer.get_buffer(copy=False)

Use preload=True to fill the entire window during construction, or preload=0 to create the buffer without reading from the source.

buffer = DataBuffer(source, buffer_depth=10, axis="images", preload=True)

Slice buffered data by timestamp:

time_range = buffer.get_time_range(axis, start=12.0, end=12.5)
index_range = buffer.get_index_range(axis, start=100, stop=200, step=2)
recent = buffer.get_last_seconds(axis, seconds=0.5)

images = time_range["data"]
timestamps = time_range["ts"]

Time ranges are inclusive and return the same {"id", "ts", "data"} shape for in-memory, Arrow-backed, and TileDB-backed buffers.

Array Operations

The arraydataengine.ops package provides NumPy-first helpers for common robotics data operations. They work on topic arrays returned by DataBuffer.get_buffer(), get_time_range(), and get_last_seconds().

from arraydataengine.ops import map_topic, random_downsample, select_time_range, topic_view, voxel_downsample

window = buffer.get_buffer()
images = select_time_range(window["images"], start=12.0, end=12.5)
normalized = map_topic(images, lambda frame: frame.astype("float32") / 255.0)

points = topic_view(window["/points"]).rows()[-1]  # latest scan, without storage padding
reduced_points = voxel_downsample(points, voxel_size=0.25)
preview_points = random_downsample(points, count=min(10_000, len(points)), seed=42)

Image and depth sequences can be resized, cropped, padded, normalized, color-converted, dtype-converted, backprojected, converted to normals, motion-aligned, and fused as NumPy stacks.

import numpy as np

from arraydataengine.ops import align_images, calibrate_depth_metric_scale, convert_color, convert_image_dtype, crop_images, depth_to_normals, frame_to_frame_optical_flow, fuse_rgbd_frames, image_gradients, image_mask, image_pyramid, iter_rgbd_frame_points, local_statistics, motion_compensated_rolling_windows, normalize_images, open_mask, resize_images

frames = window["images"]["data"]
depth_frames = window["depth"]["data"]
small = resize_images(frames, shape=(240, 320))
roi = crop_images(small, row_start=40, row_stop=200, col_start=80, col_stop=260)
float_roi = normalize_images(convert_image_dtype(roi, np.float32), per_image=True)
gray_roi = convert_color(roi, "rgb_to_gray")
foreground = open_mask(image_mask(gray_roi, min_value=32), size=3)
edges = image_gradients(gray_roi, method="sobel")
pyramid = image_pyramid(gray_roi[0], levels=4)
local = local_statistics(gray_roi, size=5, statistics=("mean", "std"))
flow = frame_to_frame_optical_flow(gray_roi, max_shift=8)
aligned_frames = align_images(gray_roi, max_shift=8)
for aligned_window in motion_compensated_rolling_windows(gray_roi, window_size=5, max_shift=8):
    process(aligned_window)
normals = depth_to_normals(depth_frames[0], fx=525.0, fy=525.0, cx=319.5, cy=239.5)
for rgbd_chunk in iter_rgbd_frame_points(depth_frames, frames, fx=525.0, fy=525.0, cx=319.5, cy=239.5):
    process(rgbd_chunk)
rgbd_cloud = fuse_rgbd_frames(depth_frames[:10], frames[:10], fx=525.0, fy=525.0, cx=319.5, cy=239.5)
depth_calibration, metric_depth = calibrate_depth_metric_scale(relative_depth, lidar_points, fx=525.0, fy=525.0, cx=319.5, cy=239.5, return_adjusted=True)

Image registration uses phase correlation. On sensors with fixed-pattern noise, such as thermal cameras with column/row/pixel offsets, the pattern stays put while the scene moves, and plain align_images then reports no motion. register_with_fixed_pattern(frames, max_shift=..., frozen=...) estimates and removes the pattern while it registers. It returns integer shifts, the corrected registered frames, and the pattern. It needs a few tens of frames in which the scene moves across the sensor.

Thermal (LWIR) cameras deliver 14/16-bit counts, and those stay uint16 through every source (image files, multi-page TIFF stacks, and ROS mono16/16UC1/16SC1 images or 16-bit PNG CompressedImages, including compressedDepth) and every storage backend. A scene usually spans a narrow band of that range, so full-range conversions such as convert_image_dtype(frame, np.uint8) crush it to a few grey levels. Use automatic gain control for display instead:

from arraydataengine.ops import colorize_thermal, radiometric_to_temperature, thermal_agc

display = thermal_agc(frames)                     # robust 1-99 percentile stretch -> uint8
display = thermal_agc(frames, method="plateau")   # plateau histogram equalization, as thermal cameras use
false_colour = colorize_thermal(frames, cmap="inferno")          # (N, H, W, 3) uint8
celsius = radiometric_to_temperature(frames, scale=0.01)         # FLIR TLinear: kelvin = counts * 0.01

One mapping is shared across a sequence (per_frame=True stretches each frame alone), so playback does not flicker. radiometric_to_temperature and temperature_to_radiometric convert radiometric ("TLinear") counts; FLIR's radiometric Boson and Lepton cameras use scale=0.01 (or 0.1 at low resolution). The image VisTool (and Visualizer("image")) scales 16-bit frames between their 1st and 99th percentiles, so a few hot or dead pixels do not flatten the view, and takes a cmap such as "gray" or "inferno".

FLIR radiometric JPEGs (R-JPEG, from FLIR handheld and fixed cameras, the Vue Pro R and Duo Pro R, and the FLIR-built DJI Zenmuse XT/XT2) and .fff files are different: their raw counts follow the radiance reaching the sensor, not the temperature, so the TLinear model above would be wrong for them. Converting them to temperature takes the camera's Planck constants (R1, R2, B, F, O), the emissivity of the object, the reflected apparent temperature, and the atmosphere (distance, air temperature, humidity) and IR window between the object and the camera. The file stores all of these, and read_flir_radiometric reads them together with the raw image:

from arraydataengine import DataSources
from arraydataengine.ops import FlirCalibration, flir_raw_to_temperature, read_flir_radiometric

image = read_flir_radiometric("IR_0042.jpg")      # also .fff files, bytes or file objects
image.raw                                         # (H, W) uint16 sensor counts
celsius = image.temperature()                     # Planck + emissivity + atmosphere, from the file's parameters
celsius = image.temperature(emissivity=0.95, reflected_temperature_c=18.0, object_distance=3.0)  # re-analyse
calibration = image.calibration.replace(relative_humidity=0.8)  # FlirCalibration (temperatures in °C)
kelvin = flir_raw_to_temperature(image.raw, calibration, unit="kelvin")

frames = DataSources("flight/*.jpg", flir="celsius")  # float32 temperatures per file; "raw" for the counts

The conversion follows the model that FLIR's software uses, as documented by ExifTool and the Thermimage R package. It removes the radiance reflected by the object, emitted by the air on both sides of the window, and emitted by the window, then inverts the Planck curve. flir_temperature_to_raw is the inverse, for simulation. With flir=, each image message also carries "flir": the file's calibration and camera information as JSON-compatible metadata, which FlirCalibration.from_dict reads back and recordings keep. scripts/datasets/make_flir_rjpeg.py writes synthetic R-JPEGs with ground-truth temperature maps. DJI's own radiometric JPEGs (H20T, M30T, Mavic Enterprise) use another format that needs DJI's Thermal SDK, and are rejected with a message that says so.

Point-cloud downsampling includes voxel-grid averaging, every-k uniform sampling, seeded random sampling by count or ratio, and farthest-point sampling.

from arraydataengine.ops import calibrate_point_cloud_metric_scale, connected_components, curvature_descriptors, farthest_point_downsample, hybrid_search, multi_scale_icp, nearest_neighbor_distance_stats, segment_ground, to_open3d_point_cloud, uniform_downsample, verify_loop_closures

uniform_points = uniform_downsample(points, every_k=4)
keypoints = farthest_point_downsample(points, count=2_048)
shape_features = curvature_descriptors(points, k=16)
spacing = nearest_neighbor_distance_stats(points, k=4)
distances, indices, counts = hybrid_search(points, query_points, radius=0.5, max_neighbors=32)
components = connected_components(points, radius=0.5, min_component_size=20)
ground, obstacles, ground_mask = segment_ground(points, distance_threshold=0.15)
registration = multi_scale_icp(scan_a, scan_b, voxel_sizes=(1.0, 0.5, 0.25))
open3d_cloud = to_open3d_point_cloud(colored_points, color_columns=(3, 4, 5))  # x, y, z, r, g, b columns

closures = verify_loop_closures(point_cloud_sequence, trajectory, radius=3.0, min_separation=60, voxel_size=0.25)
scale_calibration, metric_points = calibrate_point_cloud_metric_scale(relative_points, lidar_points, return_adjusted=True)

Registration metrics are always gated: fitness is the fraction of finite source points with a target correspondence within the gating distance, and inlier_rmse is the point-to-point RMSE of those inliers for every ICP method. verify_loop_closures uses max_correspondence_distance when given. Otherwise it scores fitness at 2×voxel_size, or at 3× the target cloud's median point spacing, so unrelated scans are rejected. It first registers coarse to fine (5×, √5× and 1× that gate) so ordinary odometry drift still converges; pass an explicit gate when the seed may be off by much more than about 15× the point spacing. Nearest-neighbour searches use scipy.spatial.cKDTree when SciPy is installed and a memory-bounded chunked search otherwise. Non-finite points are ignored, and per-point outputs such as normals return NaN for them.

SE(3) coordinate-frame helpers work across common robotics arrays:

from arraydataengine.ops import apply_transform, transform_navsat, transform_odometry

points_in_map = apply_transform(points_in_lidar, lidar_to_map)
odom_in_map = transform_odometry(odom_message_array, odom_to_map)  # twist stays in the body (child) frame
gps_in_map_frame = transform_navsat(gps_samples, enu_transform, ref_lat=37.0, ref_lon=-122.0, ref_alt=10.0)

nav_msgs/Odometry twists are expressed in the child (body) frame, so transform_odometry re-expresses only the pose by default. Pass twist_frame="parent" if your twist is already in the parent frame and should be rotated along with the pose.

Use FrameGraph when transforms need to be composed by frame name. Static transforms are used directly; time-varying transforms are interpolated by timestamp, including rotation SLERP.

from arraydataengine.ops import FrameGraph

frames = FrameGraph()
frames.add_static_transform("base_link", "odom", base_to_odom)
frames.add_time_varying_transform(
    "lidar",
    "base_link",
    timestamps=lidar_tf_timestamps,
    transforms=lidar_to_base_samples,
)

points_in_odom = frames.transform_points(points_in_lidar, "lidar", "odom", timestamp=12.5)

Projection helpers connect point clouds, depth images, RGB images, DEM grids, and camera frames with pinhole intrinsics.

from arraydataengine.ops import backproject_pixels, camera_model, colorize_points, depth_to_point_grid, distort_pixels, points_to_depth_image, project_camera_points, project_dem_to_image, rectify_image, rgbd_to_points, scale_camera_matrix

camera = camera_model(fx=525.0, fy=525.0, cx=319.5, cy=239.5, image_shape=rgb_image.shape[:2], distortion=distortion_coeffs)
small_camera_matrix = scale_camera_matrix(camera.camera_matrix, scale_x=0.5)  # half_pixel_centers=True to match cv2.resize
rectified_rgb = rectify_image(rgb_image, camera.camera_matrix, distortion=camera.distortion)
distorted_pixels = distort_pixels(raw_pixels, camera.camera_matrix, camera.distortion)
camera_points = backproject_pixels(distorted_pixels, depth_values, camera.camera_matrix, distortion=camera.distortion)
projected_pixels, visible = project_camera_points(points_in_camera, camera)
organized_depth_points = depth_to_point_grid(depth_image, fx=525.0, fy=525.0, cx=319.5, cy=239.5)
rgbd_cloud = rgbd_to_points(depth_image, rgb_image, fx=525.0, fy=525.0, cx=319.5, cy=239.5)
colored_lidar = colorize_points(points_in_camera, rgb_image, fx=525.0, fy=525.0, cx=319.5, cy=239.5, distortion=camera.distortion)
rendered_depth = points_to_depth_image(points_in_camera, image_shape=rgb_image.shape[:2], fx=525.0, fy=525.0, cx=319.5, cy=239.5)
dem_pixels, dem_mask = project_dem_to_image(elevation, fx=525.0, fy=525.0, cx=319.5, cy=239.5, image_shape=rgb_image.shape[:2])

colorize_points, sample_image_at_points, points_to_depth_image, and rgbd_to_points accept distortion= for raw (unrectified) images. Undistortion iterates to a tolerance (iterations=50, tolerance=1e-12 by default) and falls back to a damped Newton solve where OpenCV's fixed-point iteration does not converge. Points with no valid solution, or beyond the lens model's fold radius, come back as NaN. Distorted projection likewise marks points beyond the fold radius as not visible instead of wrapping them back into the image.

Crop/select helpers cover row masks, axis-aligned bounds, oriented 3D bounds, and geographic bounding boxes.

from arraydataengine.ops import crop_bounds, crop_geographic_bounds, crop_oriented_bounds, select_mask

valid_points = select_mask(points, valid_mask)
nearby_points = crop_bounds(points, min_bound=[-10, -10, -2], max_bound=[10, 10, 3])
vehicle_box = crop_oriented_bounds(points, center=pose_xyz, extent=[8.0, 4.0, 3.0], rotation=pose_rotation)
gps_window = crop_geographic_bounds(gps_samples, min_lat=36.9, min_lon=-122.3, max_lat=37.8, max_lon=-121.7)

IMU, odometry, and NavSat arrays can be normalized into one trajectory representation with pose as [x, y, z, qx, qy, qz, qw] and trajectory as pose plus linear and angular velocity. Resampling uses SLERP for orientation and linear interpolation for position, velocity, acceleration, and covariance fields. Quaternion/Euler conversion, gravity compensation, bias correction, WGS84-to-local ENU/NED conversions, trajectory smoothing, differentiation, integration, dead reckoning, covariance propagation, and quality/status masks cover common navigation preprocessing. NavSat conversions use exact WGS84 geodetic/ECEF/ENU math and wrap longitudes across ±180°; the default local reference is the first finite sample with a valid fix. odometry_to_trajectory rotates the ROS body-frame twist into the odometry parent frame (pass twist_frame="parent" if yours is already world-frame), so dead_reckon_trajectory integrates it with its default world-frame velocity. IMU and odometry samples without an orientation (all-zero quaternion or orientation_covariance[0] == -1) become NaN rows instead of raising, and the nav helpers accept DataBuffer.get_buffer() topic arrays directly. pose_to_trajectory (or sensor_to_trajectory(kind="pose")) converts PoseStamped [x, y, z, qx, qy, qz, qw] topics.

from arraydataengine.ops import (
    compensate_imu_gravity,
    correct_imu_bias,
    add_trajectory_quality_mask,
    dead_reckon_trajectory,
    differentiate_trajectory,
    euler_to_quaternion,
    imu_to_trajectory,
    integrate_trajectory,
    local_to_navsat,
    mask_trajectory,
    navsat_to_local,
    navsat_to_trajectory,
    odometry_to_trajectory,
    quaternion_to_euler,
    resample_imu,
    resample_navsat,
    resample_odometry,
    resample_trajectory,
    smooth_trajectory,
    propagate_trajectory_covariance,
)

corrected_imu = correct_imu_bias(compensate_imu_gravity(window["/imu"]), sample_slice=slice(0, 50))
imu_traj = imu_to_trajectory(corrected_imu)
odom_traj = odometry_to_trajectory(window["/odom"])
gps_traj = navsat_to_trajectory(window["/gps"], ref_lat=37.0, ref_lon=-122.0, ref_alt=10.0)
gps_ned, gps_ref = navsat_to_local(window["/gps"], frame="ned", return_reference=True)
gps_roundtrip = local_to_navsat(gps_ned, gps_ref["lat"], gps_ref["lon"], gps_ref["alt"], frame=gps_ref["frame"])
orientation = euler_to_quaternion(roll=0.0, pitch=0.0, yaw=1.57)
roll_pitch_yaw = quaternion_to_euler(odom_traj["orientation"])

odom_50hz = resample_trajectory(odom_traj, period=0.02)
imu_at_image_times = resample_imu(window["/imu"], target_timestamps=image_timestamps)
odom_at_image_times = resample_odometry(window["/odom"], target_timestamps=image_timestamps)
gps_at_image_times = resample_navsat(window["/gps"], target_timestamps=image_timestamps, ref_lat=37.0, ref_lon=-122.0, ref_alt=10.0)

smoothed_odom = smooth_trajectory(odom_50hz, window_size=5)
derived_odom = differentiate_trajectory(smoothed_odom)
integrated_odom = integrate_trajectory(derived_odom, initial_position=odom_traj["position"][0])
dead_reckoned = dead_reckon_trajectory(odom_at_image_times, initial_position=odom_traj["position"][0])  # twist already rotated to world frame
covar_odom = propagate_trajectory_covariance(dead_reckoned, process_noise={"position": [0.02, 0.02, 0.05]})
quality_odom = add_trajectory_quality_mask(covar_odom, covariance_limits={"position": [1.0, 1.0, 2.0]})
trusted_odom = mask_trajectory(quality_odom, quality_odom["quality_mask"], drop=True)

For large datasets, use the lazy topic pipeline. It records operations and only executes them when chunks, rows, reductions, windows, or explicit collection are requested.

import numpy as np

pipeline = (
    buffer.topic("images")
    .time_range(12.0, 20.0)
    .index_range(0, 1_000)
    .map(lambda frame: frame.astype(np.float32) / 255.0)
    .filter(lambda frame, ts, name: frame.mean() > 0.05)
)

for chunk in pipeline.iter_chunks(chunk_size=32):
    process(chunk.data, chunk.timestamps)

summary = pipeline.reduce(lambda acc, frame: acc + frame.mean(), initial=0.0)
small_result = pipeline.collect(chunk_size=32, max_rows=1_000)

collect() is intentionally guarded. By default it refuses results above 512 MiB; pass max_rows, max_bytes, out=, or allow_large=True when materializing a large bounded result is intentional.

Independent topic chunks can be processed concurrently with max_workers. The parallel path preserves output order and requires stateful index_range() filters to be placed before row maps/filters so they can be pushed down before chunks are scheduled.

for chunk in pipeline.iter_chunks(chunk_size=32, max_workers=4):
    process(chunk.data, chunk.timestamps)

Use DataBuffer.dataset() when the query spans multiple topics. Topic, timestamp, and message-index constraints are applied before later row filters. On persistent Arrow- and TileDB-backed buffers, leading timestamp and index ranges are pushed down before payload arrays are read.

selection = (
    buffer.dataset()
    .select_topics("/camera/image", "/points", "/gps")
    .time_range(12.0, 20.0)
    .index_range(0, 10_000)
    .frame_id("map")
)

for topic, chunk in selection.iter_chunks(chunk_size=64):
    process(topic, chunk.data, chunk.timestamps)

Use topic_workers when collecting independent topics concurrently. max_workers still controls per-topic chunk workers:

selected = selection.collect(chunk_size=64, max_workers=2, topic_workers=3, max_rows=30_000)

Geographic bounds expect latitude/longitude values by default in columns (0, 1), matching NavSat arrays shaped like [latitude, longitude, altitude]. Spatial bounds work with XYZ vectors or point arrays and keep a message when any point falls inside the bounds.

gps_samples = (
    buffer.dataset(["/gps"])
    .geographic_bounds(36.9, -122.3, 37.8, -121.7)
    .collect(max_rows=50_000)
)

point_chunks = (
    buffer.dataset(["/points"])
    .spatial_bounds(min_bound=[-20, -20, -2], max_bound=[20, 20, 5])
    .iter_chunks(chunk_size=16)
)

Topic alignment helpers cover exact timestamp joins, nearest-neighbor joins, bounded-tolerance joins, fixed-rate resampling, and rolling window joins.

from arraydataengine.ops import align_topic, resample_topic, rolling_window_join

imu_at_image_times = align_topic(images, imu, mode="bounded", tolerance=0.02)
gps_10hz = resample_topic(gps, rate_hz=10.0)
recent_points = rolling_window_join(images, points, seconds=0.25)

TopicView is still available when the data is already in memory and you want eager, metadata-aware operations. It keeps message ids, timestamps, data, topic name, frame id, source URI, dtype, shape, and time bounds together while exposing the same operations as methods.

from arraydataengine.ops import topic_view

view = topic_view(window["images"], topic="images")
normalized = view.map(lambda frame: frame.astype(np.float32) / 255.0).as_dict()

DataBuffer also keeps eager convenience wrappers for smaller results and compatibility:

normalized = buffer.map_topic("images", lambda frame: frame.astype("float32") / 255.0)
normalized = buffer.map_topic("images", lambda frame: frame.astype("float32") / 255.0, chunk_size=16)
recent_windows = list(buffer.window_topic("images", size=5))

Row functions passed to map, filter, and reduce receive the row's data. They also receive its timestamp and message id as extra positional arguments, but only when they ask for them. That means either parameters named like a timestamp (ts, t, time, timestamp, stamp) and an id (id, message_id, name, idx, i), or a plain function or lambda of your own with two or three required positional parameters, such as lambda frame, ts: .... NumPy ufuncs, functools.partial objects, operator.itemgetter, and the library's own operations always receive just the data. Bind extra arguments explicitly: view.map(functools.partial(voxel_downsample, voxel_size=0.1)). Calling map(voxel_downsample) raises a TypeError instead of silently using a timestamp as the voxel size.

Variable-size rows, such as point clouds of different sizes, are stored zero-padded to a per-topic capacity together with each row's real length. Results then carry a lengths array next to the padded data. Row functions always receive trimmed rows, and TopicView.rows() / topic_rows(result) return them as a list.

ML-ready helpers turn lazy topic windows into plain iterator datasets, materialized NumPy datasets, or optional PyTorch datasets. They also provide deterministic split helpers, lightweight augmentations, and padding collation for mixed-rate windows and variable-size point clouds.

from arraydataengine.ops import (
    augment_image,
    augment_point_cloud,
    collate_samples,
    iter_ml_windows,
    split_topic,
    to_numpy_dataset,
    to_torch_dataset,
)

image_windows = iter_ml_windows(buffer.dataset(["images"]), size=4, chunk_size=32)
numpy_windows = to_numpy_dataset(image_windows)
batch = numpy_windows.as_arrays(pad=True)

splits = split_topic(window["/gps"], by="time", fractions=(0.8, 0.1, 0.1))
augmented_image = augment_image(batch["data"][0], flip_horizontal=True)
point_window = next(iter(iter_ml_windows(buffer.dataset(["/points"]), size=1)))
augmented_points = augment_point_cloud(point_window["data"][0], jitter_std=0.01, seed=7)
collated = collate_samples([{"points": augmented_points}, {"points": augmented_points[:128]}])

torch_dataset = to_torch_dataset(iter_ml_windows(buffer.dataset(["images"]), size=2), iterable=True)

Use source_pipeline() when you want operations to run while messages stream from a DataSources object, before full topics are loaded. The same pipeline can write to an in-memory DataBuffer or persist through the default MCAP backend or explicitly selected Arrow/Parquet and TileDB backends. Long-running source and topic pipelines accept progress callbacks, cancellation tokens, and mutable checkpoint dictionaries that can be saved and reused to resume from the last processed row. Checkpoints only advance past rows that were actually delivered. A cancelled collect() attaches the rows gathered so far as PipelineCancelled.partial, so partial plus a resumed collect() equals a full run. When resuming a persistent store, the store's per-topic counts are authoritative: after a hard crash, rows the checkpoint claimed but never flushed are replayed from the source, and rows already stored are never appended twice.

import json

from functools import partial

from arraydataengine.ops import CancellationToken, PipelineCancelled, source_pipeline, voxel_downsample
from arraydataengine.source import DataSources

source = DataSources("/data/rosbag2/split_recording/")
checkpoint = {}
persistent_checkpoint = {}
cancel_token = CancellationToken()

def report(progress):
    print(progress.processed, progress.emitted, progress.topic)

pipeline = (
    source_pipeline(source)
    .select_topics("/points")
    .time_range(12.0, 20.0)
    .map(partial(voxel_downsample, voxel_size=0.1))  # scans keep their own sizes
    .filter(lambda msg: msg["data"].shape[0] > 0)
)

try:
    rolling_buffer = pipeline.to_buffer(
        buffer_depth=128,
        progress_callback=report,
        cancel_token=cancel_token,
        checkpoint=checkpoint,
        progress_interval=100,
    )
except PipelineCancelled:
    with open("filtered_points.checkpoint.json", "w") as checkpoint_file:
        json.dump(checkpoint, checkpoint_file)

persistent_buffer = pipeline.to_buffer(
    data_uri="/tmp/ade/filtered_points/",
    use_db=True,
    checkpoint=persistent_checkpoint,
)

Persistent output uses MCAP by default. Select backend="arrow" for Parquet analysis stores. Pass backend="tiledb" to to_buffer(), or use the persist_to_tiledb() convenience method, when TileDB is preferred. Persistent source pipelines require source.get_count(topic); TileDB uses that count to size its dense destination arrays, while all persistent backends record the actual filtered message count in store metadata.

Initial operation coverage includes source-level streaming pipelines, progress reporting, cancellation, resumable checkpoints, topic selection, map/filter/reduce/window helpers, nearest-time alignment, SE(3) transforms, frame graphs, camera projection helpers, camera intrinsics/distortion/rectification utilities, mask and bounds cropping, point cloud downsampling/sampling/KNN-radius-hybrid search/normals/covariance descriptors/distance stats/outlier filters/clustering/connected components/plane and ground segmentation/ICP registration/Open3D adapters, image/depth sequence transforms, morphology, gradients, pyramids, local image statistics, frame-to-frame optical flow, image alignment, motion-compensated rolling windows, valid-depth masks, depth backprojection, depth normals, RGB-D fusion, navsat ENU/NED conversion, quaternion/Euler conversion, gravity compensation, bias correction, trajectory resampling/smoothing/differentiation/integration/dead reckoning/covariance propagation/quality masks, trajectory speed, ML-ready iterators/NumPy/PyTorch adapters/splits/augmentations/collation, and DEM/raster helpers for terrain gradients, normals, roughness, traversability, local patches, point clouds, and meshes.

Persistent Storage

Set use_db=True to persist messages to disk. This is intended for full-source ingest and larger-than-memory datasets. Three storage engines are available via the backend parameter:

  • "mcap" (the default for new stores): append-only MCAP segments with an atomic manifest, resumable ingestion, original message metadata, and variable array shapes.
  • "arrow" (explicit analysis storage): Apache Arrow / Parquet fragment files. Fast, compressed, dependency-light (pip install "arraydataengine[arrow]"), and every persisted topic is directly readable by Polars, DuckDB, pandas, and anything else that speaks Parquet.
  • "tiledb": TileDB dense arrays (pip install "arraydataengine[tiledb]"). Supports in-place cell updates (buffer[i] = ...), which the immutable Arrow fragments do not.

When backend is omitted and use_db=True, new stores use MCAP; existing Arrow and TileDB stores retain their backend.

from arraydataengine.buffer import DataBuffer
from arraydataengine.source import DataSources

axis = "/camera/image"
source = DataSources("/data/rosbag2/split_recording/")

with DataBuffer(
    data_source=source,
    data_uri="/data/stores/my_dataset/",
    axis=axis,
    use_db=True,          # backend="mcap" is the default for new stores
) as buffer:
    buffer.load_data_db(axis)
    print(buffer.get_group_uri())

Using DataBuffer as a context manager closes the store cleanly and marks completed topics as closed; close_completed() does the same without leaving the context. Topics that are still incomplete stay open, so a later run resumes them. Memory, Arrow, and TileDB topics each have one payload dtype and trailing shape; MCAP permits changes in both. Safe widening casts (for example int32 into an int64 topic) are applied; anything else raises ValueError rather than being silently broadcast or cast. Point-cloud topics, and any topic listed in DataBuffer(ragged_topics={topic: capacity}), may vary in their first dimension. On memory, Arrow, and TileDB they are stored zero-padded to a capacity, set by DataBuffer(max_points=...) or the source's max_points and defaulting to 30000 rows, with each message's real length. Reads return lengths next to the padded data, buf[i] returns the trimmed row, and topic(...).rows() returns trimmed rows. DataBuffer(topics=[...]) limits which topics are buffered, the in-memory ring grows as messages arrive, and get_buffer() returns only filled rows in time order. get_size() is the number of stored rows on every backend. Arrow and TileDB stage rows in memory and reads include staged rows without flushing them. MCAP finalizes its active segment before reading; it preserves native array shapes and pads only eager multi-message results when necessary. Call close() (or use the context manager) so the last batch is written; after a crash, a resumed ingest replays only the rows that were never flushed.

Arrow and TileDB stores use escaped topic directory names so topics such as /a/b and /a_b remain distinct. Existing stores retain their original directories when reopened. Query results expose each topic's location as source_uri.

Arrow and TileDB keep timestamps, message names, frame ids, and per-message spatial bounds alongside the data. Time, index, frame-id, and spatial-bounds constraints from lazy topic or dataset queries are pushed down before data chunks are read, so pipelines avoid loading unselected message payloads. MCAP reads sequentially and evaluates these filters after decoding each message. Streaming payload batches avoid loading complete topics; eager APIs still materialize their selection.

The Arrow backend exposes its tuning knobs through backend_options (reasonable defaults shown):

buffer = DataBuffer(
    data_source=source,
    data_uri="/data/stores/my_dataset/",
    axis=axis,
    use_db=True,
    backend="arrow",
    backend_options={
        "flush_bytes": 32 * 1024 * 1024,      # staged bytes per topic before a fragment is written
        "memory_budget_bytes": 64 * 1024 * 1024,  # shared staging limit across all topics
        "row_group_bytes": 16 * 1024 * 1024,  # target Parquet row-group size (random-access granularity)
        "compression": "zstd",                # any Parquet codec: zstd, snappy, lz4, none...
        "batch_readahead": 1,                 # scanner prefetch depth; raise for throughput,
        "fragment_readahead": 1,              #   keep at 1 for minimal scan memory
        "use_threads": False,                 # parallel scans (off preserves order + bounds memory)
        "row_overhead_bytes": 256,            # per-row bookkeeping counted toward flush_bytes
    },
)

The TileDB backend accepts flush_bytes, row_overhead_bytes, and consolidate_max_bytes the same way. Closing a TileDB store consolidates only what is safe while other processes read it (this session's index fragments and the metadata). Run buffer.compact() when no reader has the store open to merge payload fragments (arrays up to consolidate_max_bytes) and vacuum superseded files. Unknown backend_options raise ValueError for every backend.

Existing stores can be reopened without the original source:

reopened = DataBuffer(
    data_source=None,
    data_uri="/data/stores/my_dataset/",
    axis=axis,
    use_db=True,
)

for chunk in reopened.topic(axis).time_range(12.0, 20.0).iter_chunks(chunk_size=64):
    process(chunk.data)

Interrupted ingests can be resumed by constructing a new DataBuffer with the same source and data_uri. Stored per-topic counts are loaded from the store's metadata, previously written messages are skipped during source replay, and remaining messages are appended.

DEM Tiles

DEM sources use NASA Earthdata credentials from environment variables:

export earthdata_username="..."
export earthdata_password="..."

Then request north/west tile ranges:

from arraydataengine.source import DataSources

north = [37, 39]
west = [122, 124]

source = DataSources("DEM", bounds=[north, west])

for tile in source.get_message():
    print(tile["name"], tile["data"].shape)

Pass cache_dir to reuse downloaded HGT payloads. Cached tiles can be read without Earthdata credentials:

source = DataSources("DEM", bounds=[north, west], cache_dir="/tmp/ade-dem-cache")

Downloads require HTTPS, and Earthdata credentials are only sent to the Earthdata login host (urs.earthdata.nasa.gov) or to the tile host itself; redirects to any other host never receive them. Tiles come from NASA LP DAAC's cloud archive by default; base_url (or ADE_DEM_BASE_URL) can point at a mirror, either a directory of <name>.SRTMGL1.hgt.zip files or a per-tile template containing {name}. A custom http:// base URL requires an explicit allow_insecure_http=True. Tiles are validated before they are cached, cache writes are atomic, a missing (404) tile such as open ocean is skipped with a warning, and tiles are yielded as native, writable int16 arrays.

DEM helper functions operate on NumPy windows, so they can be used on individual tiles, cropped patches, or lazy pipeline chunks:

from arraydataengine.ops import (
    dem_to_mesh,
    dem_to_point_cloud,
    mosaic_dem_tiles,
    read_dem_cache,
    reproject_raster,
    resample_raster,
    roughness_map,
    sample_elevation,
    terrain_normals,
    terrain_patch,
    traversability_map,
    write_dem_cache,
)

tile = next(source.get_message())
elevation = mosaic_dem_tiles([tile])  # float64, north-up, voids -> NaN, SRTM seams merged
lat0, lon0 = north[0], -west[0]       # south-west corner; west longitudes are negative

normals = terrain_normals(elevation, resolution=30.0, north_up=True)
roughness = roughness_map(elevation, window_size=5)
traversability = traversability_map(
    elevation,
    resolution=30.0,
    max_slope_degrees=25.0,
    max_roughness=1.0,
)

vehicle_patch = terrain_patch(elevation, center=(150.0, 240.0), size=(64, 64), resolution=30.0, north_up=True)
vehicle_elevation = sample_elevation(elevation, x=[150.0], y=[240.0], resolution=30.0, north_up=True)
terrain_points = dem_to_point_cloud(vehicle_patch, resolution=30.0, north_up=True)
terrain_mesh = dem_to_mesh(vehicle_patch, resolution=30.0, north_up=True)

overview = resample_raster(elevation, shape=(512, 512))
local_grid = reproject_raster(
    elevation,
    src_bounds=(lon0, lat0, lon0 + 1.0, lat0 + 1.0),
    dst_bounds=(lon0 + 0.25, lat0 + 0.25, lon0 + 0.75, lat0 + 0.75),
    shape=(256, 256),
    north_up=True,
)

write_dem_cache("/tmp/ade-dem-cache", tile["name"], elevation, metadata={"bounds": [lon0, lat0, lon0 + 1.0, lat0 + 1.0]})
cached_elevation, metadata = read_dem_cache("/tmp/ade-dem-cache", tile["name"], return_metadata=True)

mosaic_dem_tiles also accepts {name: raster} mappings or (name, raster) pairs and fills sparse regions when adjacent SRTM tiles are missing. Adjacent SRTM tiles share one edge row/column, so tiles named like SRTM tiles (N37W122) with the standard 1201 or 3601 samples are merged with overlap=1 automatically; pass overlap= explicitly for other rasters. SRTM voids (-32768) become fill_value (NaN by default).

DEM rasters decoded from HGT tiles are north-up (row 0 is the northern edge). Pass north_up=True to the helpers that map rows to y: slope_aspect, hillshade, terrain_gradients, terrain_normals, terrain_patch, sample_elevation, sample_elevation_at_navsat, dem_to_point_cloud, dem_to_mesh, reproject_raster, dem_grid_to_points, transform_dem_grid, and project_dem_to_image. For geographic (lat/lon) rasters such as SRTM mosaics, sample with sample_elevation_at_latlon(raster, lat, lon, bounds=(west, south, east, north)) or the index from mosaic_dem_tiles(..., return_index=True), or pass bounds=/index= to sample_elevation_at_navsat. Local ENU coordinates do not map onto a lat/lon grid exactly. The default, north_up=False, treats increasing row index as +y. In both conventions origin is the south-west cell, aspect is a compass bearing in [0, 2π) with NaN for flat cells, and sample_elevation returns NaN for points outside the grid. resolution also accepts (dx, dy), because the east-west cell size of a geographic tile shrinks with latitude.

Benchmarks

Run the source adapter, synthetic operation, lazy pipeline, and storage backend benchmarks with:

python -m pytest tests/test_source_benchmarks.py -q -s

Results below were measured on 2026-07-28 with Python 3.14 on arm64 (Apple Silicon), NumPy 2.4.6, pyarrow 25.0.0, tiledb 0.36.1. Each result is the best of three runs. Bag, DB3, and DEM use mocked readers/network responses, so those rows measure adapter overhead without requiring ROS bag files or Earthdata access.

Benchmark Workload Items Elapsed Throughput Latency
ImgSource.messages temporary 64x64 PNG files read through OpenCV 200 0.006641s 30,114 items/s 33.2 us/item
BagSource.messages mocked AnyReader and image sensor conversion 200 0.000168s 1,191,363 items/s 0.8 us/item
DB3Source.messages mocked AnyReader and image sensor conversion 200 0.000275s 727,603 items/s 1.4 us/item
DEMSource.messages mocked Earthdata zip response with 32x32 HGT tiles 4 0.000074s 54,237 items/s 18.4 us/item
ImageOps.synthetic resize, normalize, and grayscale 96x96 RGB image frames 64 0.001132s 56,543 frames/s 17.7 us/frame
PointCloudOps.synthetic voxel downsample, KNN search, and normal estimation on XYZ+intensity points 1,024 0.010418s 98,287 points/s 10.2 us/point
IMUOps.synthetic resample synthetic IMU samples and dead-reckon the trajectory 2,000 0.026483s 75,520 samples/s 13.2 us/sample
OdometryOps.synthetic resample synthetic odometry samples and dead-reckon the trajectory 2,000 0.027236s 73,433 samples/s 13.6 us/sample
NavSatOps.synthetic convert and resample synthetic WGS84 NavSat samples into local trajectory arrays 2,000 0.012295s 162,666 samples/s 6.1 us/sample
DEMOps.synthetic terrain normals, roughness, traversability, and point-cloud conversion over a DEM grid 16,384 0.005833s 2,808,726 cells/s 0.4 us/cell
TopicPipeline.iter_chunks lazy in-memory time/index pushdown and row map over 50k synthetic samples 10,000 0.035091s 284,972 items/s 3.5 us/item
TileDB.TopicPipeline.time_range lazy TileDB time-range pushdown and row map over a temp persisted topic 100 0.205731s 486 items/s 2057.3 us/item
Arrow.TopicPipeline.time_range same pipeline over the Arrow/Parquet backend 100 0.001433s 69,796 items/s 14.3 us/item

The TileDB rows above predate the batched, compressed TileDB writer; see Storage backends for current backend measurements.

voxel_downsample groups 1M points at 0.5 m resolution in ~70 ms (packed int64 voxel keys; ~7-9x faster than the previous row-wise grouping).

Storage backends

Identical workload — 1,500 lidar-sized point clouds of shape (30000, 3) float64, 1.08 GB of data — through each persistent backend, reads in a fresh process. Measured on 2026-09-27 with Python 3.12.3 on Linux x86_64 (Intel i9-13900H, NVMe), NumPy 2.3.2, pyarrow 22.0.0, tiledb 0.36.1; best of two runs:

Operation TileDB Arrow
Ingest (excluding data generation) 6.5 s 7.1 s
On-disk footprint 1,031 MB 923 MB
Full 1.1 GB chunked scan 0.88 s 1.38 s
Time-range read (151 messages) 63 ms 234 ms
Single-message fetch 6 ms 33 ms
Peak RSS, ingest / reads 246 MB / 360 MB 269 MB / 565 MB

The benchmark compares TileDB and Arrow, which stage writes with a 32 MiB per-topic flush target. Arrow now also enforces a 64 MiB shared staging budget. Arrow scans use the default batch_readahead/fragment_readahead of 1 (raising readahead trades memory for throughput). This workload is nearly incompressible; real sensor streams compress further under both backends' Zstd codecs. MCAP is now the default recording backend; Arrow remains available for Parquet analysis with Polars, DuckDB, and pandas. MCAP was not part of this benchmark.

Development

Run the test suite:

python -m pytest -q

Release packaging and Python registry publishing steps are tracked in TODO.md.

Run the notebooks (each in an isolated copy, so nothing they write lands in the repository; needs pip install nbclient):

python scripts/run_notebooks.py                    # all notebooks, offline defaults
python scripts/run_notebooks.py --save             # refresh the committed outputs

To test them against real recordings, fetch the test datasets (about 700 MB, stored in ~/.cache/arraydataengine/test-data, verified against pinned checksums) and run the real profile:

python scripts/datasets/fetch_test_datasets.py
python scripts/run_notebooks.py --profile real

The datasets are the KITTI raw drive 2011_09_26_drive_0001 (GPS/IMU, Velodyne, and camera; CC BY-NC-SA 3.0), converted to ROS 2 bags in both SQLite and MCAP storage by scripts/datasets/kitti_to_rosbag2.py. They also include MOT17-05 (CC BY-NC-SA 3.0), San Francisco Bay Area SRTM-format elevation tiles from the AWS Open Data terrain-tiles bucket (SRTM and USGS 3DEP with ETOPO1 bathymetry offshore; public domain, attribution requested), CIFAR-10, and synthetic FLIR Boson LWIR sequences with blinking beacons from scripts/datasets/make_lwir_sequence.py, because no open thermal dataset contains pulsed beacons. The ros1 profile replays the example recording's ROS 1 bag. The dem-download profile exercises DEMSource's download branch against a local server holding the real tiles. The earthdata-mock profile runs the notebooks' default Earthdata download path, including the login redirect, session cookie, and pre-signed S3 hand-off, against the synthetic Earthdata of scripts/datasets/mock_earthdata.py (python scripts/datasets/mock_earthdata.py --check runs the same flow through the library directly). scripts/datasets/check_earthdata_download.py checks the live service with your own credentials.

Build and run the notebook container:

./build.sh
docker compose up jupyter

Jupyter uses JUPYTER_TOKEN from your environment when it is set; otherwise it generates a token and prints the login URL (docker compose logs jupyter).

Notebook examples live in notebooks/. They run offline out of the box on the example recording (example/mapeverything_0.db3), SyntheticSource, or seeded synthetic data, and each has a configuration cell near the top whose ADE_* environment variables point it at your own recordings, image folders, or DEM tiles. Files they generate go under __-prefixed paths, which git ignores.

Notebook Shows
demo.ipynb Tour of the library: lidar ICP and mapping, image sequences, INS trajectories, DEM terrain, IR beacon detection
circular_buffer.ipynb Rolling DataBuffer windows, time and index slices, variable-size scans
iterator_example.ipynb Reading messages, get_data, lazy topic and source pipelines, alignment
bag_to_TileDB.ipynb Persisting a recording to TileDB, reopening it, and training a small PyTorch model from it
loam_example.ipynb Scan-to-scan and scan-to-map ICP against the recorded poses, HTML map viewer
ins.ipynb IMU/odometry/NavSat trajectories, gravity and bias correction, dead reckoning
mot.ipynb Segmentation and IoU tracking on a synthetic video, CLEAR-MOT scores
dem.ipynb SRTM tiles (offline synthetic cache by default), mosaics, terrain analysis, lat/lon sampling
tiff.ipynb, pulsed_encoded_beacons.ipynb Thermal (LWIR) frame registration and blinking-beacon detection by frequency
mlp.ipynb A small PyTorch classifier fed through the ML helpers (CIFAR-10 optional)

About

Analysis and generator for cyber-physical systems data

Topics

Resources

Stars

1 star

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages