Skip to main content
Version: Next

FileSystemDatasetClient

File system implementation of the dataset client.

This client persists dataset items to the file system as individual JSON files within a structured directory hierarchy following the pattern:

{STORAGE_DIR}/datasets/{DATASET_ID}/{ITEM_ID}.json

Each item is stored as a separate file, which allows for durability and the ability to recover after process termination. Dataset operations like filtering, sorting, and pagination are implemented by processing the stored files according to the requested parameters.

This implementation is ideal for long-running crawlers where data persistence is important, and for development environments where you want to easily inspect the collected data between runs.

Hierarchy

Index

Methods

__init__

  • __init__(*, metadata, path_to_dataset, lock): None
  • Initialize a new instance.

    Preferably use the FileSystemDatasetClient.open class method to create a new instance.


    Parameters

    • keyword-onlymetadata: DatasetMetadata
    • keyword-onlypath_to_dataset: Path
    • keyword-onlylock: asyncio.Lock

    Returns None

drop

  • async drop(): None
  • Drop the whole dataset and remove all its items.

    The backend method for the Dataset.drop call.


    Returns None

get_data

  • async get_data(*, offset, limit, clean, desc, fields, omit, unwind, skip_empty, skip_hidden, flatten, view): DatasetItemsListPage
  • Get data from the dataset with various filtering options.

    The backend method for the Dataset.get_data call.