README.md

SUPERB Submission Template

Welcome to the SUPERB benchmark! SUPERB is a collection of benchmarking resources to evaluate the capability of a universal shared representation for speech processing. SUPERB consists of the following:

1. A benchmark of ten speech processing tasks built on established public datasets
2. A benchmark toolkit designed to evaluate and analyze pretrained model performance on various downstream tasks following the conventional evaluation protocols from speech communities
3. A public leaderboard for submission and performance tracking on the benchmark

SUPERB aims to offer the community a standard and comprehensive framework to train, evaluate, and compare the generalizability of universal speech representations on speech processing tasks. A universal speech representation can be leveraged to quickly adapt to diverse downstream tasks with minimum architectural change and downstream fine-tuning, so as to reduce the model development cycle time for new tasks. To emphasize on evaluating the quality of the learned universal representation, SUPERB puts an explicit constraint on the downstream model and limits its parameter size.

The ultimate goal of SUPERB is to democratize the advancement in speech processing with powerful, generalizable, and reusable speech representations. SUPERB is a long-term maintained and continuously developing project. As we are gradually releasing new tasks and opening new tracks, we invite researchers to participate in the challenge and advance the research frontier together.

This repository can be used to generate a template so you can submit your predictions for evaluation on the leaderboard.

Quickstart

1. Create an account and organisation on the Hugging Face Hub

First create an account on the Hugging Face Hub and you can sign up here if you haven't already! Next, create a a new organisation for your submissions and invite the following benchmark organisers join it:

• Shu-wen (Leo) Yang: leo19941227

2. Create a template repository on your machine

The next step is to create a template repository on your local machine that contains various files and a CLI to help you validate and submit your pretrained models. The Hugging Face Hub uses Git Large File Storage (LFS) to manage large files, so first install it if you don't have it already. For example, on macOS you can run:

brew install git-lfs
git lfs install

Next, run the following commands to create the repository. We recommend creating a Python virtual environment for the project, e.g. with Anaconda:

# Create and activate a virtual environment
conda create -n superb python=3.8 && conda activate superb
# Install the following libraries
pip install cookiecutter huggingface-hub==0.0.16
# Create the template repository
cookiecutter git+https://huggingface.co/superb/superb-submission

This will ask you to specify your Hugging Face Hub username, password, organisation, and the name of the repository:

hf_hub_username [<huggingface>]:
hf_hub_password [<password>]:
hf_hub_organisation [superb-submissions]:
repo_name [<my-superb-submissions>]:

This will trigger the following steps:

1. Create a private dataset repository on the Hugging Face Hub under {hf_hub_organisation}/{repo_name}
2. Clone the repository to your local machine
3. Add various template files, commit them locally to the repository, and push them to the Hub

The resulting repository should have the following structure:

my-superb-submission
├── LICENSE
├── README.md               <- The README with submission instructions
├── cli.py                  <- The CLI for validating predictions etc
└── requirements.txt        <- The requirements file for the submissions

3. Install the dependencies

The final step is to install the project's dependencies:

# Navigate to the template repository
cd my-superb-submission
# Install dependencies
python -m pip install -r requirements.txt

That's it! You're now all set to start pretraining your speech models - see the instructions below on how to submit them to the Hub.

Submitting to the leaderboard

To make a submission to the leaderboard), there are three main steps:

1. Generate predictions on the unlabeled test set of each task
2. Validate the predictions are compatible with the evaluation framework
3. Push the predictions to the Hub!

See the instructions below for more details.

Rules

1. To prevent overfitting to the public leaderboard, we only evaluate one submission per week. You can push predictions to the Hub as many times as you wish, but we will only evaluate the most recent commit in a given week.
2. Transfer or meta-learning using other datasets, including further pre-training on other corpora, is allowed.
3. Use of unlabeled test data is allowed, as is it always available in the applied setting. For example, further pre-training using the unlabeled data for a task would be permitted.
4. Systems may be augmented with information retrieved from the internet, e.g. via automated web searches.

Submission file format

Validate your submission

To ensure that your submission files are correctly formatted, run the following command from the root of the repository:

python cli.py validate

If everything is correct, you should see the following message:

All submission files validated! ✨ 🚀 ✨
Now you can make a submission 🤗

Push your submission to the Hugging Face Hub!

The final step is to commit your files and push them to the Hub:

python cli.py submit

If there are no errors, you should see the following message:

Submission successful! 🎉 🥳 🎉
Your submission will be fine-tuned and evaulated on Sunday 05 September 2021 ⏳

where the evaluation is run every Sunday and your results will be visible on the leaderboard.