leo19941227 commited on
Commit
450ba27
β€’
1 Parent(s): b589517

add README in template repo

Browse files
Files changed (1) hide show
  1. {{cookiecutter.repo_name}}/README.md +106 -0
{{cookiecutter.repo_name}}/README.md ADDED
@@ -0,0 +1,106 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # SUPERB Submission Template
2
+
3
+ This repository can be used to generate a template so you can submit your pretrained model for evaluation on [the leaderboard](https://huggingface.co/spaces/superb/superb-leaderboard) in the [SUPERB Challenge](https://superbbenchmark.org/challenge).
4
+
5
+ ## Quickstart
6
+
7
+ ### 1. Create an account and organisation on the Hugging Face Hub
8
+
9
+ First create an account on the Hugging Face Hub and you can sign up [here](https://huggingface.co/join) if you haven't already! Next, create a new organization and invite the SUPERB Hidden Set Committee to join. You will upload your model to a repository under this organization so that members inside it can access the model.
10
+
11
+ * [superb-hidden-set](https://huggingface.co/superb-hidden-set)
12
+
13
+ ### 2. Create a template repository on your machine
14
+
15
+ The next step is to create a template repository on your local machine that contains various files and a CLI to help you validate and submit your pretrained models. The Hugging Face Hub uses [Git Large File Storage (LFS)](https://git-lfs.github.com) to manage large files, so first install it if you don't have it already. For example, on macOS you can run:
16
+
17
+ ```bash
18
+ brew install git-lfs
19
+ git lfs install
20
+ ```
21
+
22
+ Next, run the following commands to create the repository. We recommend creating a Python virtual environment for the project, e.g. with Anaconda:
23
+
24
+ ```bash
25
+ # Create and activate a virtual environment
26
+ conda create -n superb-submit python=3.8 && conda activate superb-submit
27
+ # Install the following libraries
28
+ pip install cookiecutter huggingface-hub==0.0.16
29
+ # Create the template repository
30
+ cookiecutter git+https://huggingface.co/superb/superb-submission
31
+ ```
32
+
33
+ This will ask you to specify your Hugging Face Hub username, password, organisation, and the name of the repository:
34
+
35
+ ```
36
+ hf_hub_username [<huggingface>]:
37
+ hf_hub_password [<password>]:
38
+ hf_hub_organisation [superb-submissions]:
39
+ repo_name [<my-superb-submissions>]:
40
+ ```
41
+
42
+ This will trigger the following steps:
43
+
44
+ 1. Create a private dataset repository on the Hugging Face Hub under `{hf_hub_organisation}/{repo_name}`
45
+ 2. Clone the repository to your local machine
46
+ 3. Add various template files, commit them locally to the repository, and push them to the Hub
47
+
48
+ The resulting repository should have the following structure:
49
+
50
+ ```
51
+ my-superb-submission
52
+ β”œβ”€β”€ LICENSE
53
+ β”œβ”€β”€ README.md <- The README with submission instructions
54
+ β”œβ”€β”€ cli.py <- The CLI for validating predictions etc
55
+ └── requirements.txt <- The requirements packages for the submissions
56
+ β”œβ”€β”€ expert.py <- Your model definition
57
+ └── model.pt <- Your model weights
58
+ ```
59
+
60
+ ### 3. Install the dependencies
61
+
62
+ The final step is to install the project's dependencies:
63
+
64
+ ```bash
65
+ # Navigate to the template repository
66
+ cd my-superb-submission
67
+ # Install dependencies
68
+ python -m pip install -r requirements.txt
69
+ ```
70
+
71
+ That's it! You're now all set to start pretraining your speech models - see the instructions below on how to submit them to the Hub.
72
+
73
+
74
+ ## Submitting to the leaderboard
75
+
76
+ To make a submission to the [leaderboard](https://superbbenchmark.org/leaderboard), there are 4 main steps:
77
+
78
+ 1. Modify `expert.py` and `model.py` so we can initialize an upstream model following the [policy](https://superbbenchmark.org/challenge) by:
79
+
80
+ ```python
81
+ upstream = UpstreamExpert(ckpt="./model.pt")
82
+ ```
83
+
84
+ 2. Validate the upstream model meets the requirements in the [policy](https://superbbenchmark.org/challenge). If everything is correct, you should see the following message: "All submission files validated! Now you can make a submission."
85
+
86
+ ```
87
+ python cli.py validate
88
+ ```
89
+
90
+ 3. Push the predictions to the Hub! If there are no errors, you should see the following message: "Upload successful!"
91
+
92
+ ```
93
+ python cli.py upload "commit message: my best model"
94
+ ```
95
+
96
+ 4. [Make a submission at SUPERB website](https://superbbenchmark.org/submit) by uniquely indentifying this submission/model with the following information, which will be shown by:
97
+
98
+ ```
99
+ python cli.py info
100
+ ```
101
+
102
+ - Organization Name
103
+ - Repository Name
104
+ - Commit Hash (full 40 characters)
105
+
106
+ After you finish the above 4 steps. Please stay tuned and wait for us to get the finetuned results on the hidden set!