Skip to main content
Version: 1.11

Dataset

Dataset is a storage for managing structured tabular data.

The dataset class provides a high-level interface for storing and retrieving structured data with consistent schema, similar to database tables or spreadsheets. It abstracts the underlying storage implementation details, offering a consistent API regardless of where the data is physically stored.

Dataset operates in an append-only mode, allowing new records to be added but not modified or deleted after creation. This makes it particularly suitable for storing crawling results and other data that should be immutable once collected.

The class provides methods for adding data, retrieving data with various filtering options, and exporting data to different formats. You can create a dataset using the open class method, specifying either a name or ID. The underlying storage implementation is determined by the configured storage client.

Usage

from crawlee.storages import Dataset

# Open a dataset
dataset = await Dataset.open(name='my-dataset')

# Add data
await dataset.push_data({'title': 'Example Product', 'price': 99.99})

# Retrieve filtered data
results = await dataset.get_data(limit=10, desc=True)

# Export data
await dataset.export_to('results.json', content_type='json')

Hierarchy

Index