FileSystemDatasetClient
Hierarchy
- DatasetClient
- FileSystemDatasetClient
Index
Methods
__init__
Initialize a new instance.
Preferably use the
FileSystemDatasetClient.openclass method to create a new instance.Parameters
keyword-onlymetadata: DatasetMetadata
keyword-onlypath_to_dataset: Path
keyword-onlylock: asyncio.Lock
Returns None
drop
Drop the whole dataset and remove all its items.
The backend method for the
Dataset.dropcall.Returns None
get_data
Get data from the dataset with various filtering options.
The backend method for the
Dataset.get_datacall.
File system implementation of the dataset client.
This client persists dataset items to the file system as individual JSON files within a structured directory hierarchy following the pattern:
Each item is stored as a separate file, which allows for durability and the ability to recover after process termination. Dataset operations like filtering, sorting, and pagination are implemented by processing the stored files according to the requested parameters.
This implementation is ideal for long-running crawlers where data persistence is important, and for development environments where you want to easily inspect the collected data between runs.