w11wo commited on
Commit
f6c67b2
1 Parent(s): e2a1094

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +142 -105
README.md CHANGED
@@ -1,132 +1,169 @@
1
  ---
 
2
  license: apache-2.0
3
  tags:
4
- - automatic-speech-recognition
5
- - kresnik/zeroth_korean
6
- - generated_from_trainer
7
  datasets:
8
- - zeroth_korean_asr
9
  model-index:
10
- - name: ''
11
- results: []
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
  ---
13
 
14
- <!-- This model card has been generated automatically according to the information the Trainer had access to. You
15
- should probably proofread and complete it, then remove this comment. -->
16
 
17
- #
18
 
19
- This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the KRESNIK/ZEROTH_KOREAN - CLEAN dataset.
20
- It achieves the following results on the evaluation set:
21
- - Loss: 0.2089
22
- - Wer: 0.2954
23
- - Cer: 0.0953
24
 
25
- ## Model description
26
 
27
- More information needed
28
 
29
- ## Intended uses & limitations
 
 
30
 
31
- More information needed
32
 
33
- ## Training and evaluation data
34
 
35
- More information needed
 
 
 
36
 
37
  ## Training procedure
38
 
39
  ### Training hyperparameters
40
 
41
  The following hyperparameters were used during training:
42
- - learning_rate: 7.5e-05
43
- - train_batch_size: 8
44
- - eval_batch_size: 8
45
- - seed: 42
46
- - gradient_accumulation_steps: 4
47
- - total_train_batch_size: 32
48
- - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
49
- - lr_scheduler_type: linear
50
- - lr_scheduler_warmup_steps: 2000
51
- - num_epochs: 50.0
52
- - mixed_precision_training: Native AMP
 
53
 
54
  ### Training results
55
 
56
- | Training Loss | Epoch | Step | Validation Loss | Wer | Cer |
57
- |:-------------:|:-----:|:-----:|:---------------:|:------:|:------:|
58
- | 19.7138 | 0.72 | 500 | 19.6427 | 1.0 | 1.0 |
59
- | 4.8039 | 1.44 | 1000 | 4.7842 | 1.0 | 1.0 |
60
- | 4.5619 | 2.16 | 1500 | 4.5608 | 0.9992 | 0.9598 |
61
- | 4.254 | 2.88 | 2000 | 4.2729 | 0.9955 | 0.9063 |
62
- | 4.1905 | 3.6 | 2500 | 4.2257 | 0.9903 | 0.8758 |
63
- | 4.0683 | 4.32 | 3000 | 3.9294 | 0.9937 | 0.7911 |
64
- | 3.486 | 5.04 | 3500 | 2.7045 | 1.0012 | 0.5934 |
65
- | 2.946 | 5.75 | 4000 | 1.9691 | 0.9425 | 0.4634 |
66
- | 2.634 | 6.47 | 4500 | 1.5212 | 0.8807 | 0.3850 |
67
- | 2.4066 | 7.19 | 5000 | 1.2551 | 0.8177 | 0.3601 |
68
- | 2.2651 | 7.91 | 5500 | 1.0423 | 0.7650 | 0.3039 |
69
- | 2.1828 | 8.63 | 6000 | 0.9599 | 0.7273 | 0.3106 |
70
- | 2.1023 | 9.35 | 6500 | 0.9482 | 0.7161 | 0.3063 |
71
- | 2.0536 | 10.07 | 7000 | 0.8242 | 0.6767 | 0.2860 |
72
- | 1.9803 | 10.79 | 7500 | 0.7643 | 0.6563 | 0.2637 |
73
- | 1.9468 | 11.51 | 8000 | 0.7319 | 0.6441 | 0.2505 |
74
- | 1.9178 | 12.23 | 8500 | 0.6937 | 0.6320 | 0.2489 |
75
- | 1.8515 | 12.95 | 9000 | 0.6443 | 0.6053 | 0.2196 |
76
- | 1.8083 | 13.67 | 9500 | 0.6286 | 0.6122 | 0.2148 |
77
- | 1.819 | 14.39 | 10000 | 0.6015 | 0.5986 | 0.2074 |
78
- | 1.7684 | 15.11 | 10500 | 0.5682 | 0.5741 | 0.1982 |
79
- | 1.7195 | 15.83 | 11000 | 0.5385 | 0.5592 | 0.2007 |
80
- | 1.7044 | 16.55 | 11500 | 0.5362 | 0.5524 | 0.2097 |
81
- | 1.6879 | 17.27 | 12000 | 0.5119 | 0.5489 | 0.2083 |
82
- | 1.656 | 17.98 | 12500 | 0.4990 | 0.5362 | 0.1968 |
83
- | 1.6122 | 18.7 | 13000 | 0.4561 | 0.5092 | 0.1900 |
84
- | 1.5919 | 19.42 | 13500 | 0.4778 | 0.5225 | 0.1975 |
85
- | 1.5896 | 20.14 | 14000 | 0.4563 | 0.5098 | 0.1859 |
86
- | 1.5589 | 20.86 | 14500 | 0.4362 | 0.4940 | 0.1725 |
87
- | 1.5353 | 21.58 | 15000 | 0.4140 | 0.4826 | 0.1580 |
88
- | 1.5441 | 22.3 | 15500 | 0.4031 | 0.4742 | 0.1550 |
89
- | 1.5116 | 23.02 | 16000 | 0.3916 | 0.4748 | 0.1545 |
90
- | 1.4731 | 23.74 | 16500 | 0.3841 | 0.4810 | 0.1542 |
91
- | 1.4647 | 24.46 | 17000 | 0.3752 | 0.4524 | 0.1475 |
92
- | 1.4328 | 25.18 | 17500 | 0.3587 | 0.4476 | 0.1461 |
93
- | 1.4129 | 25.9 | 18000 | 0.3429 | 0.4242 | 0.1366 |
94
- | 1.4062 | 26.62 | 18500 | 0.3450 | 0.4251 | 0.1355 |
95
- | 1.3928 | 27.34 | 19000 | 0.3297 | 0.4145 | 0.1322 |
96
- | 1.3906 | 28.06 | 19500 | 0.3210 | 0.4185 | 0.1336 |
97
- | 1.358 | 28.78 | 20000 | 0.3131 | 0.3970 | 0.1275 |
98
- | 1.3445 | 29.5 | 20500 | 0.3069 | 0.3920 | 0.1276 |
99
- | 1.3159 | 30.22 | 21000 | 0.3035 | 0.3961 | 0.1255 |
100
- | 1.3044 | 30.93 | 21500 | 0.2952 | 0.3854 | 0.1242 |
101
- | 1.3034 | 31.65 | 22000 | 0.2966 | 0.3772 | 0.1227 |
102
- | 1.2963 | 32.37 | 22500 | 0.2844 | 0.3706 | 0.1208 |
103
- | 1.2765 | 33.09 | 23000 | 0.2841 | 0.3567 | 0.1173 |
104
- | 1.2438 | 33.81 | 23500 | 0.2734 | 0.3552 | 0.1137 |
105
- | 1.2487 | 34.53 | 24000 | 0.2703 | 0.3502 | 0.1118 |
106
- | 1.2249 | 35.25 | 24500 | 0.2650 | 0.3484 | 0.1142 |
107
- | 1.2229 | 35.97 | 25000 | 0.2584 | 0.3374 | 0.1097 |
108
- | 1.2374 | 36.69 | 25500 | 0.2568 | 0.3337 | 0.1095 |
109
- | 1.2153 | 37.41 | 26000 | 0.2494 | 0.3327 | 0.1071 |
110
- | 1.1925 | 38.13 | 26500 | 0.2518 | 0.3366 | 0.1077 |
111
- | 1.1908 | 38.85 | 27000 | 0.2437 | 0.3272 | 0.1057 |
112
- | 1.1858 | 39.57 | 27500 | 0.2396 | 0.3265 | 0.1044 |
113
- | 1.1808 | 40.29 | 28000 | 0.2373 | 0.3156 | 0.1028 |
114
- | 1.1842 | 41.01 | 28500 | 0.2356 | 0.3152 | 0.1026 |
115
- | 1.1668 | 41.73 | 29000 | 0.2319 | 0.3188 | 0.1025 |
116
- | 1.1448 | 42.45 | 29500 | 0.2293 | 0.3099 | 0.0995 |
117
- | 1.1327 | 43.17 | 30000 | 0.2265 | 0.3047 | 0.0979 |
118
- | 1.1307 | 43.88 | 30500 | 0.2222 | 0.3078 | 0.0989 |
119
- | 1.1419 | 44.6 | 31000 | 0.2215 | 0.3038 | 0.0981 |
120
- | 1.1231 | 45.32 | 31500 | 0.2193 | 0.3013 | 0.0972 |
121
- | 1.139 | 46.04 | 32000 | 0.2162 | 0.3007 | 0.0968 |
122
- | 1.1114 | 46.76 | 32500 | 0.2122 | 0.2982 | 0.0960 |
123
- | 1.111 | 47.48 | 33000 | 0.2125 | 0.2946 | 0.0948 |
124
- | 1.0982 | 48.2 | 33500 | 0.2099 | 0.2957 | 0.0953 |
125
- | 1.109 | 48.92 | 34000 | 0.2092 | 0.2955 | 0.0955 |
126
- | 1.0905 | 49.64 | 34500 | 0.2088 | 0.2954 | 0.0953 |
127
-
128
-
129
- ### Framework versions
 
 
 
 
 
 
 
130
 
131
  - Transformers 4.17.0.dev0
132
  - Pytorch 1.10.2+cu102
 
1
  ---
2
+ language: ko
3
  license: apache-2.0
4
  tags:
5
+ - automatic-speech-recognition
6
+ - generated_from_trainer
7
+ - robust-speech-event
8
  datasets:
9
+ - kresnik/zeroth_korean
10
  model-index:
11
+ - name: Wav2Vec2 XLS-R 300M Korean
12
+ results:
13
+ - task:
14
+ name: Automatic Speech Recognition
15
+ type: automatic-speech-recognition
16
+ dataset:
17
+ name: Zeroth Korean
18
+ type: kresnik/zeroth_korean
19
+ args: clean
20
+ metrics:
21
+ - name: Test WER
22
+ type: wer
23
+ value: 29.54
24
+ - name: Test CER
25
+ type: cer
26
+ value: 9.53
27
+ - task:
28
+ name: Automatic Speech Recognition
29
+ type: automatic-speech-recognition
30
+ dataset:
31
+ name: Robust Speech Event - Dev Data
32
+ type: speech-recognition-community-v2/dev_data
33
+ args: ko
34
+ metrics:
35
+ - name: Test WER
36
+ type: wer
37
+ value: 76.26
38
+ - name: Test CER
39
+ type: cer
40
+ value: 38.67
41
  ---
42
 
43
+ # Wav2Vec2 XLS-R 300M Korean
 
44
 
45
+ Wav2Vec2 XLS-R 300M Korean is an automatic speech recognition model based on the [XLS-R](https://arxiv.org/abs/2111.09296) architecture. This model is a fine-tuned version of [Wav2Vec2-XLS-R-300M](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the [Zeroth Korean](https://huggingface.co/datasets/kresnik/zeroth_korean) dataset.
46
 
47
+ This model was trained using HuggingFace's PyTorch framework and is part of the [Robust Speech Challenge Event](https://discuss.huggingface.co/t/open-to-the-community-robust-speech-recognition-challenge/13614) organized by HuggingFace. All training was done on a Tesla V100, sponsored by OVH.
 
 
 
 
48
 
49
+ All necessary scripts used for training could be found in the [Files and versions](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-korean/tree/main) tab, as well as the [Training metrics](https://huggingface.co/w11wo/wav2vec2-xls-r-300m-korean/tensorboard) logged via Tensorboard.
50
 
51
+ ## Model
52
 
53
+ | Model | #params | Arch. | Training/Validation data (text) |
54
+ | ---------------------------- | ------- | ----- | ------------------------------- |
55
+ | `wav2vec2-xls-r-300m-korean` | 300M | XLS-R | `Zeroth Korean` Dataset |
56
 
57
+ ## Evaluation Results
58
 
59
+ The model achieves the following results on evaluation:
60
 
61
+ | Dataset | Loss | WER | CER |
62
+ | -------------------------------- | ------ | ------ | ------ |
63
+ | `Zeroth Korean` | 0.2089 | 29.54% | 9.53% |
64
+ | `Robust Speech Event - Dev Data` | N/A | 76.26% | 38.67% |
65
 
66
  ## Training procedure
67
 
68
  ### Training hyperparameters
69
 
70
  The following hyperparameters were used during training:
71
+
72
+ - `learning_rate`: 7.5e-05
73
+ - `train_batch_size`: 8
74
+ - `eval_batch_size`: 8
75
+ - `seed`: 42
76
+ - `gradient_accumulation_steps`: 4
77
+ - `total_train_batch_size`: 32
78
+ - `optimizer`: Adam with `betas=(0.9, 0.999)` and `epsilon=1e-08`
79
+ - `lr_scheduler_type`: linear
80
+ - `lr_scheduler_warmup_steps`: 2000
81
+ - `num_epochs`: 50.0
82
+ - `mixed_precision_training`: Native AMP
83
 
84
  ### Training results
85
 
86
+ | Training Loss | Epoch | Step | Validation Loss | Wer | Cer |
87
+ | :-----------: | :---: | :---: | :-------------: | :----: | :----: |
88
+ | 19.7138 | 0.72 | 500 | 19.6427 | 1.0 | 1.0 |
89
+ | 4.8039 | 1.44 | 1000 | 4.7842 | 1.0 | 1.0 |
90
+ | 4.5619 | 2.16 | 1500 | 4.5608 | 0.9992 | 0.9598 |
91
+ | 4.254 | 2.88 | 2000 | 4.2729 | 0.9955 | 0.9063 |
92
+ | 4.1905 | 3.6 | 2500 | 4.2257 | 0.9903 | 0.8758 |
93
+ | 4.0683 | 4.32 | 3000 | 3.9294 | 0.9937 | 0.7911 |
94
+ | 3.486 | 5.04 | 3500 | 2.7045 | 1.0012 | 0.5934 |
95
+ | 2.946 | 5.75 | 4000 | 1.9691 | 0.9425 | 0.4634 |
96
+ | 2.634 | 6.47 | 4500 | 1.5212 | 0.8807 | 0.3850 |
97
+ | 2.4066 | 7.19 | 5000 | 1.2551 | 0.8177 | 0.3601 |
98
+ | 2.2651 | 7.91 | 5500 | 1.0423 | 0.7650 | 0.3039 |
99
+ | 2.1828 | 8.63 | 6000 | 0.9599 | 0.7273 | 0.3106 |
100
+ | 2.1023 | 9.35 | 6500 | 0.9482 | 0.7161 | 0.3063 |
101
+ | 2.0536 | 10.07 | 7000 | 0.8242 | 0.6767 | 0.2860 |
102
+ | 1.9803 | 10.79 | 7500 | 0.7643 | 0.6563 | 0.2637 |
103
+ | 1.9468 | 11.51 | 8000 | 0.7319 | 0.6441 | 0.2505 |
104
+ | 1.9178 | 12.23 | 8500 | 0.6937 | 0.6320 | 0.2489 |
105
+ | 1.8515 | 12.95 | 9000 | 0.6443 | 0.6053 | 0.2196 |
106
+ | 1.8083 | 13.67 | 9500 | 0.6286 | 0.6122 | 0.2148 |
107
+ | 1.819 | 14.39 | 10000 | 0.6015 | 0.5986 | 0.2074 |
108
+ | 1.7684 | 15.11 | 10500 | 0.5682 | 0.5741 | 0.1982 |
109
+ | 1.7195 | 15.83 | 11000 | 0.5385 | 0.5592 | 0.2007 |
110
+ | 1.7044 | 16.55 | 11500 | 0.5362 | 0.5524 | 0.2097 |
111
+ | 1.6879 | 17.27 | 12000 | 0.5119 | 0.5489 | 0.2083 |
112
+ | 1.656 | 17.98 | 12500 | 0.4990 | 0.5362 | 0.1968 |
113
+ | 1.6122 | 18.7 | 13000 | 0.4561 | 0.5092 | 0.1900 |
114
+ | 1.5919 | 19.42 | 13500 | 0.4778 | 0.5225 | 0.1975 |
115
+ | 1.5896 | 20.14 | 14000 | 0.4563 | 0.5098 | 0.1859 |
116
+ | 1.5589 | 20.86 | 14500 | 0.4362 | 0.4940 | 0.1725 |
117
+ | 1.5353 | 21.58 | 15000 | 0.4140 | 0.4826 | 0.1580 |
118
+ | 1.5441 | 22.3 | 15500 | 0.4031 | 0.4742 | 0.1550 |
119
+ | 1.5116 | 23.02 | 16000 | 0.3916 | 0.4748 | 0.1545 |
120
+ | 1.4731 | 23.74 | 16500 | 0.3841 | 0.4810 | 0.1542 |
121
+ | 1.4647 | 24.46 | 17000 | 0.3752 | 0.4524 | 0.1475 |
122
+ | 1.4328 | 25.18 | 17500 | 0.3587 | 0.4476 | 0.1461 |
123
+ | 1.4129 | 25.9 | 18000 | 0.3429 | 0.4242 | 0.1366 |
124
+ | 1.4062 | 26.62 | 18500 | 0.3450 | 0.4251 | 0.1355 |
125
+ | 1.3928 | 27.34 | 19000 | 0.3297 | 0.4145 | 0.1322 |
126
+ | 1.3906 | 28.06 | 19500 | 0.3210 | 0.4185 | 0.1336 |
127
+ | 1.358 | 28.78 | 20000 | 0.3131 | 0.3970 | 0.1275 |
128
+ | 1.3445 | 29.5 | 20500 | 0.3069 | 0.3920 | 0.1276 |
129
+ | 1.3159 | 30.22 | 21000 | 0.3035 | 0.3961 | 0.1255 |
130
+ | 1.3044 | 30.93 | 21500 | 0.2952 | 0.3854 | 0.1242 |
131
+ | 1.3034 | 31.65 | 22000 | 0.2966 | 0.3772 | 0.1227 |
132
+ | 1.2963 | 32.37 | 22500 | 0.2844 | 0.3706 | 0.1208 |
133
+ | 1.2765 | 33.09 | 23000 | 0.2841 | 0.3567 | 0.1173 |
134
+ | 1.2438 | 33.81 | 23500 | 0.2734 | 0.3552 | 0.1137 |
135
+ | 1.2487 | 34.53 | 24000 | 0.2703 | 0.3502 | 0.1118 |
136
+ | 1.2249 | 35.25 | 24500 | 0.2650 | 0.3484 | 0.1142 |
137
+ | 1.2229 | 35.97 | 25000 | 0.2584 | 0.3374 | 0.1097 |
138
+ | 1.2374 | 36.69 | 25500 | 0.2568 | 0.3337 | 0.1095 |
139
+ | 1.2153 | 37.41 | 26000 | 0.2494 | 0.3327 | 0.1071 |
140
+ | 1.1925 | 38.13 | 26500 | 0.2518 | 0.3366 | 0.1077 |
141
+ | 1.1908 | 38.85 | 27000 | 0.2437 | 0.3272 | 0.1057 |
142
+ | 1.1858 | 39.57 | 27500 | 0.2396 | 0.3265 | 0.1044 |
143
+ | 1.1808 | 40.29 | 28000 | 0.2373 | 0.3156 | 0.1028 |
144
+ | 1.1842 | 41.01 | 28500 | 0.2356 | 0.3152 | 0.1026 |
145
+ | 1.1668 | 41.73 | 29000 | 0.2319 | 0.3188 | 0.1025 |
146
+ | 1.1448 | 42.45 | 29500 | 0.2293 | 0.3099 | 0.0995 |
147
+ | 1.1327 | 43.17 | 30000 | 0.2265 | 0.3047 | 0.0979 |
148
+ | 1.1307 | 43.88 | 30500 | 0.2222 | 0.3078 | 0.0989 |
149
+ | 1.1419 | 44.6 | 31000 | 0.2215 | 0.3038 | 0.0981 |
150
+ | 1.1231 | 45.32 | 31500 | 0.2193 | 0.3013 | 0.0972 |
151
+ | 1.139 | 46.04 | 32000 | 0.2162 | 0.3007 | 0.0968 |
152
+ | 1.1114 | 46.76 | 32500 | 0.2122 | 0.2982 | 0.0960 |
153
+ | 1.111 | 47.48 | 33000 | 0.2125 | 0.2946 | 0.0948 |
154
+ | 1.0982 | 48.2 | 33500 | 0.2099 | 0.2957 | 0.0953 |
155
+ | 1.109 | 48.92 | 34000 | 0.2092 | 0.2955 | 0.0955 |
156
+ | 1.0905 | 49.64 | 34500 | 0.2088 | 0.2954 | 0.0953 |
157
+
158
+ ## Disclaimer
159
+
160
+ Do consider the biases which came from pre-training datasets that may be carried over into the results of this model.
161
+
162
+ ## Authors
163
+
164
+ Wav2Vec2 XLS-R 300M Korean was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on OVH Cloud.
165
+
166
+ ## Framework versions
167
 
168
  - Transformers 4.17.0.dev0
169
  - Pytorch 1.10.2+cu102