Skip to main content
Version: Next

FileSystemKeyValueStoreClient

File system implementation of the key-value store client.

This client persists data to the file system, making it suitable for scenarios where data needs to survive process restarts. Keys are mapped to file paths in a directory structure following the pattern:

{STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}

Binary data is stored as-is, while JSON and text data are stored in human-readable format. The implementation automatically handles serialization based on the content type and maintains metadata about each record.

This implementation is ideal for long-running crawlers where persistence is important and for development environments where you want to easily inspect the stored data between runs.

Hierarchy

Index

Methods

__init__

  • __init__(*, metadata, path_to_kvs, lock): None
  • Initialize a new instance.

    Preferably use the FileSystemKeyValueStoreClient.open class method to create a new instance.


    Parameters

    Returns None

delete_value

  • async delete_value(*, key): None
  • Delete a value from the key-value store by its key.

    The backend method for the KeyValueStore.delete_value call.


    Parameters

    • keyword-onlykey: str

    Returns None

drop

  • async drop(): None
  • Drop the whole key-value store and remove all its values.

    The backend method for the KeyValueStore.drop call.


    Returns None

get_metadata

get_public_url

  • async get_public_url(*, key): str
  • Return a file:// URL for the given key.


    Parameters

    • keyword-onlykey: str

      The key to get the public URL for.

    Returns str

    A file:// URL pointing to the file on the local filesystem.

get_value

iterate_entries

  • async iterate_entries(*, exclusive_start_key, limit): AsyncIterator[KeyValueStoreRecord]