Skip to content

Hotfix/arff - #1388

Merged
PGijsbers merged 3 commits into
developfrom
hotfix/arff
Jan 25, 2025
Merged

PGijsbers merged 3 commits into
developfrom
hotfix/arff

Conversation

@PGijsbers

@PGijsbers PGijsbers commented Jan 25, 2025 •

Copy link
Copy Markdown
Member

This introduces a backdoor for disabling attempting to download parquet files through the OPENML_SKIP_PQ variable.
If OPENML_SKIP_PQ is set to true (case insensitive), then parquet files will not be downloaded in the get_dataset and OpenMLDataset.get_data calls.

The PR also fixes a bug where an error would be raised if the parquet file failed to download. The str(_get_dataset_parquet_file(self)) would return None if it failed to download, but by converting it to a string the next check is always false and thus it would not fall back to arff.

The reason for using an environment variable:

  • It's a bit quicker to implement than a configuration option, and it makes it easier to turn it on for a single call.
  • Compared to adding function arguments, the environment variable (or configuration option) don't need changes to existing scripts. Making it easier for people to start using it now, and stop using it later. We can issue a warning if the environment variable remains set in a later release.