Can a private/gated dataset be added to a leaderboard?

[KennethEnevoldsen]

Hi guys,

We are currently working on putting together a new benchmark, but some of the underlying datasets can't be made fully public. I would like to know if all datasets need to be made public in order to see the results of corresponding tasks on the leaderboard. Could it work if a gated access is enabled for the dataset?

Thanks a lot for help

Originally asked here

[KennethEnevoldsen]

Thanks for the question. As a part of RTEB, we have actually just started using closed datasets to estimate generalisation.

So this is indeed possible, but it requires:

1. We can make a sample public (~5 samples), which can be anonymised
2. We can publish descriptive statistics of the task
3. The MTEB maintainers will have access to the dataset to ensure that benchmarks remain up to date and to allow for validation of the data samples. To control who has access, we use the "mteb-private" org on HuggingFace, which reduced the number of people with access compared to the full organization. We will not make any of these datasets public or share them in any way.

gating is also an option, but then you need to describe the gating policy

[andrejridzik]

Thanks a lot for the answer!

[KennethEnevoldsen]

No problem, hoping to see more private datasets to prevent overfitting