Data format#
Each dataset is stored in pyml_datasets/data/<name>.npz, a compressed NumPy archive.
The loader reads it with allow_pickle=False, so an archive can never execute code
when it is opened. The current format is version 2.
Contents of a file#
Key |
Content |
|---|---|
|
2D |
|
Targets: |
|
One string per column of |
|
Description text with the source, citation, license and changes. Required. It is generated from the information of the dataset. |
|
The integer |
|
The |
|
Class or target names. Optional. |
|
Informational JSON written by the recipe, for example the version of the library the data came from or the SHA-256 digest of the downloaded file. The loader ignores it. |
If a required key is missing, the loader raises a ValueError that names the dataset and
the missing keys. If the file itself is missing from the installed package, it raises a
FileNotFoundError.
Missing values and categorical features#
Missing values are stored as NaN in X. DatasetInfo.has_missing says whether a
dataset has any, and the build script refuses a file whose NaN do not match the flag.
Categorical features are stored as integer codes in X (still float64, so a missing
category is NaN). The names of the categories are in DatasetInfo.categorical: the code
is the index of the name. Dataset.categorical_features gives the column indices.
The pyml-datasets package does not impute or encode anything, that is left to you.
Rebuilding the files#
The files are generated from the original sources by the scripts in scripts/. Each
dataset has a recipe in scripts/recipes/ that fetches the data and returns it as a
Built object, and the provenance in pyml_datasets/datasets/<name>.py. The dev
dependency group, which uv sync installs, contains the packages the recipes need, and
recipes that download data need an internet connection. Run the commands from the repository
root:
$ uv run -m scripts build
$ uv run -m scripts build iris wine --force
$ uv run -m scripts gen
Existing files are skipped unless --force is given. .npz files are not
byte-for-byte reproducible, so every rebuild would add a new binary file to the git history.
That is why pyml_datasets/data/MANIFEST.json stores a digest of the arrays (not of the
file), and why a test compares every file with it. Before saving, the script checks that
X is a finite 2D matrix without unexpected NaN, that y and feature_names
match it, that class labels are 0 .. n_classes - 1 and that categorical codes are valid.
Adding a dataset#
The steps are in CONTRIBUTING.md, section Adding a dataset. In short:
uv run -m scripts new creates the files, you fill in the facts, build the file and run
uv run -m scripts gen (or uv run -m scripts refresh), then the Nox sessions.