Bug: Querying metadataset missing metafeatures

[EWitting]

Thank you for publishing the raw result datasets, this is useful for analysis since I do not have the computational resources to run the benchmarks fully myself. However, when I follow the instructions to download the dataset and run the example notebook, it shows that the metafeatures are missing.

Steps to reproduce:

1. Clone the repository, create python environment
2. Create a fresh /data folder, download and unzip micro, mini, extended datasets according to readme
3. Run the first cell in example_query_metadataset.ipynb

Result:

Notebook fails with error: No such file or directory: 'data/qt_metadataset/dataset-meta-features/meta-album/meta-features.json'

Possible Explanation

It appears that compute_embeddings.py must be used to generate this .json file. However, if I understand correctly this requires applying a vision encoder to images in a large number of image datasets. This is undocumented and prevents the accessible use of the metadataset as demonstrated in the example.

I assume that if the dataset-meta-features folder is also included in the micro/mini/extended download, or a as separate link, it would be possible to fully analyze the results and simulate CASH without requiring access to the original dataset, as seems the intention.

Are the metafeatures accessible somewhere that I missed? And if not, would it be possible to publish these online? Thanks in advance

Perhaps @sebastianpinedaar has access to the data? As author of the notebook?

[sebastianpinedaar]

Hi Emiel!

thank you for the heads up, and for the interest in our project.

Indeed, we missed to the metafeatures, which can be accessed via this link.

Best,

Sebastian