ivanlau commited on
Commit
a9bf195
1 Parent(s): 030b2da

added evaluation metric on cv8 test dataset

Browse files
.gitignore CHANGED
@@ -1,2 +1 @@
1
  checkpoint-*/
2
- log_*
 
1
  checkpoint-*/
 
.ipynb_checkpoints/README-checkpoint.md CHANGED
@@ -6,13 +6,24 @@ tags:
6
  - automatic-speech-recognition
7
  - mozilla-foundation/common_voice_8_0
8
  - generated_from_trainer
9
- - zh-HK
10
- - robust-speech-event
11
  datasets:
12
- - common_voice
13
- model-index:
14
- - name: ''
15
- results: []
 
 
 
 
 
 
 
 
 
 
 
 
 
16
  ---
17
 
18
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
@@ -22,8 +33,8 @@ should probably proofread and complete it, then remove this comment. -->
22
 
23
  This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the MOZILLA-FOUNDATION/COMMON_VOICE_8_0 - ZH-HK dataset.
24
  It achieves the following results on the evaluation set:
25
- - Loss: 2.6726
26
- - Wer: 0.9815
27
 
28
  ## Model description
29
 
@@ -51,23 +62,113 @@ The following hyperparameters were used during training:
51
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
52
  - lr_scheduler_type: linear
53
  - lr_scheduler_warmup_steps: 500
54
- - num_epochs: 10.0
55
  - mixed_precision_training: Native AMP
56
 
57
  ### Training results
58
 
59
- | Training Loss | Epoch | Step | Validation Loss | Wer |
60
- |:-------------:|:-----:|:----:|:---------------:|:------:|
61
- | No log | 1.0 | 183 | 47.8442 | 1.0 |
62
- | No log | 2.0 | 366 | 6.3109 | 1.0 |
63
- | 41.8902 | 3.0 | 549 | 6.2392 | 1.0 |
64
- | 41.8902 | 4.0 | 732 | 5.9739 | 1.1123 |
65
- | 41.8902 | 5.0 | 915 | 4.9014 | 1.9474 |
66
- | 5.5817 | 6.0 | 1098 | 3.9892 | 1.0188 |
67
- | 5.5817 | 7.0 | 1281 | 3.5080 | 1.0104 |
68
- | 5.5817 | 8.0 | 1464 | 3.0797 | 0.9905 |
69
- | 3.5579 | 9.0 | 1647 | 2.8111 | 0.9836 |
70
- | 3.5579 | 10.0 | 1830 | 2.6726 | 0.9815 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
71
 
72
 
73
  ### Framework versions
 
6
  - automatic-speech-recognition
7
  - mozilla-foundation/common_voice_8_0
8
  - generated_from_trainer
 
 
9
  datasets:
10
+ - mozilla-foundation/common_voice_8_0
11
+ - name: XLS-R-300M - Chinese_HongKong (Cantonese)
12
+ results:
13
+ - task:
14
+ name: Automatic Speech Recognition
15
+ type: automatic-speech-recognition
16
+ dataset:
17
+ name: Common Voice 8
18
+ type: mozilla-foundation/common_voice_7_0
19
+ args: zh-HK
20
+ metrics:
21
+ - name: Test WER
22
+ type: wer
23
+ value: 0.8111349803079126
24
+ - name: Test CER
25
+ type: cer
26
+ value: 0.21962250882996914
27
  ---
28
 
29
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 
33
 
34
  This model is a fine-tuned version of [facebook/wav2vec2-xls-r-300m](https://huggingface.co/facebook/wav2vec2-xls-r-300m) on the MOZILLA-FOUNDATION/COMMON_VOICE_8_0 - ZH-HK dataset.
35
  It achieves the following results on the evaluation set:
36
+ - Loss: 1.4848
37
+ - Wer: 0.8004
38
 
39
  ## Model description
40
 
 
62
  - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
63
  - lr_scheduler_type: linear
64
  - lr_scheduler_warmup_steps: 500
65
+ - num_epochs: 100.0
66
  - mixed_precision_training: Native AMP
67
 
68
  ### Training results
69
 
70
+ | Training Loss | Epoch | Step | Validation Loss | Wer |
71
+ |:-------------:|:-----:|:-----:|:---------------:|:------:|
72
+ | No log | 1.0 | 183 | 47.8442 | 1.0 |
73
+ | No log | 2.0 | 366 | 6.3109 | 1.0 |
74
+ | 41.8902 | 3.0 | 549 | 6.2392 | 1.0 |
75
+ | 41.8902 | 4.0 | 732 | 5.9739 | 1.1123 |
76
+ | 41.8902 | 5.0 | 915 | 4.9014 | 1.9474 |
77
+ | 5.5817 | 6.0 | 1098 | 3.9892 | 1.0188 |
78
+ | 5.5817 | 7.0 | 1281 | 3.5080 | 1.0104 |
79
+ | 5.5817 | 8.0 | 1464 | 3.0797 | 0.9905 |
80
+ | 3.5579 | 9.0 | 1647 | 2.8111 | 0.9836 |
81
+ | 3.5579 | 10.0 | 1830 | 2.6726 | 0.9815 |
82
+ | 2.7771 | 11.0 | 2013 | 2.7177 | 0.9809 |
83
+ | 2.7771 | 12.0 | 2196 | 2.3582 | 0.9692 |
84
+ | 2.7771 | 13.0 | 2379 | 2.1708 | 0.9757 |
85
+ | 2.3488 | 14.0 | 2562 | 2.0491 | 0.9526 |
86
+ | 2.3488 | 15.0 | 2745 | 1.8518 | 0.9378 |
87
+ | 2.3488 | 16.0 | 2928 | 1.6845 | 0.9286 |
88
+ | 1.7859 | 17.0 | 3111 | 1.6412 | 0.9280 |
89
+ | 1.7859 | 18.0 | 3294 | 1.5488 | 0.9035 |
90
+ | 1.7859 | 19.0 | 3477 | 1.4546 | 0.9010 |
91
+ | 1.3898 | 20.0 | 3660 | 1.5147 | 0.9201 |
92
+ | 1.3898 | 21.0 | 3843 | 1.4467 | 0.8959 |
93
+ | 1.1291 | 22.0 | 4026 | 1.4743 | 0.9035 |
94
+ | 1.1291 | 23.0 | 4209 | 1.3827 | 0.8762 |
95
+ | 1.1291 | 24.0 | 4392 | 1.3437 | 0.8792 |
96
+ | 0.8993 | 25.0 | 4575 | 1.2895 | 0.8577 |
97
+ | 0.8993 | 26.0 | 4758 | 1.2928 | 0.8558 |
98
+ | 0.8993 | 27.0 | 4941 | 1.2947 | 0.9163 |
99
+ | 0.6298 | 28.0 | 5124 | 1.3151 | 0.8738 |
100
+ | 0.6298 | 29.0 | 5307 | 1.2972 | 0.8514 |
101
+ | 0.6298 | 30.0 | 5490 | 1.3030 | 0.8432 |
102
+ | 0.4757 | 31.0 | 5673 | 1.3264 | 0.8364 |
103
+ | 0.4757 | 32.0 | 5856 | 1.3131 | 0.8421 |
104
+ | 0.3735 | 33.0 | 6039 | 1.3457 | 0.8588 |
105
+ | 0.3735 | 34.0 | 6222 | 1.3450 | 0.8473 |
106
+ | 0.3735 | 35.0 | 6405 | 1.3452 | 0.9218 |
107
+ | 0.3253 | 36.0 | 6588 | 1.3754 | 0.8397 |
108
+ | 0.3253 | 37.0 | 6771 | 1.3554 | 0.8353 |
109
+ | 0.3253 | 38.0 | 6954 | 1.3532 | 0.8312 |
110
+ | 0.2816 | 39.0 | 7137 | 1.3694 | 0.8345 |
111
+ | 0.2816 | 40.0 | 7320 | 1.3953 | 0.8296 |
112
+ | 0.2397 | 41.0 | 7503 | 1.3858 | 0.8293 |
113
+ | 0.2397 | 42.0 | 7686 | 1.3959 | 0.8402 |
114
+ | 0.2397 | 43.0 | 7869 | 1.4350 | 0.9318 |
115
+ | 0.2084 | 44.0 | 8052 | 1.4004 | 0.8806 |
116
+ | 0.2084 | 45.0 | 8235 | 1.3871 | 0.8255 |
117
+ | 0.2084 | 46.0 | 8418 | 1.4060 | 0.8252 |
118
+ | 0.1853 | 47.0 | 8601 | 1.3992 | 0.8501 |
119
+ | 0.1853 | 48.0 | 8784 | 1.4186 | 0.8252 |
120
+ | 0.1853 | 49.0 | 8967 | 1.4120 | 0.8165 |
121
+ | 0.1671 | 50.0 | 9150 | 1.4166 | 0.8214 |
122
+ | 0.1671 | 51.0 | 9333 | 1.4411 | 0.8501 |
123
+ | 0.1513 | 52.0 | 9516 | 1.4692 | 0.8394 |
124
+ | 0.1513 | 53.0 | 9699 | 1.4640 | 0.8391 |
125
+ | 0.1513 | 54.0 | 9882 | 1.4501 | 0.8419 |
126
+ | 0.133 | 55.0 | 10065 | 1.4134 | 0.8351 |
127
+ | 0.133 | 56.0 | 10248 | 1.4593 | 0.8405 |
128
+ | 0.133 | 57.0 | 10431 | 1.4560 | 0.8389 |
129
+ | 0.1198 | 58.0 | 10614 | 1.4734 | 0.8334 |
130
+ | 0.1198 | 59.0 | 10797 | 1.4649 | 0.8318 |
131
+ | 0.1198 | 60.0 | 10980 | 1.4659 | 0.8100 |
132
+ | 0.1109 | 61.0 | 11163 | 1.4784 | 0.8119 |
133
+ | 0.1109 | 62.0 | 11346 | 1.4938 | 0.8149 |
134
+ | 0.1063 | 63.0 | 11529 | 1.5050 | 0.8152 |
135
+ | 0.1063 | 64.0 | 11712 | 1.4773 | 0.8176 |
136
+ | 0.1063 | 65.0 | 11895 | 1.4836 | 0.8261 |
137
+ | 0.0966 | 66.0 | 12078 | 1.4979 | 0.8157 |
138
+ | 0.0966 | 67.0 | 12261 | 1.4603 | 0.8048 |
139
+ | 0.0966 | 68.0 | 12444 | 1.4803 | 0.8127 |
140
+ | 0.0867 | 69.0 | 12627 | 1.4974 | 0.8130 |
141
+ | 0.0867 | 70.0 | 12810 | 1.4721 | 0.8078 |
142
+ | 0.0867 | 71.0 | 12993 | 1.4644 | 0.8192 |
143
+ | 0.0827 | 72.0 | 13176 | 1.4835 | 0.8138 |
144
+ | 0.0827 | 73.0 | 13359 | 1.4934 | 0.8122 |
145
+ | 0.0734 | 74.0 | 13542 | 1.4951 | 0.8062 |
146
+ | 0.0734 | 75.0 | 13725 | 1.4908 | 0.8070 |
147
+ | 0.0734 | 76.0 | 13908 | 1.4876 | 0.8124 |
148
+ | 0.0664 | 77.0 | 14091 | 1.4934 | 0.8053 |
149
+ | 0.0664 | 78.0 | 14274 | 1.4603 | 0.8048 |
150
+ | 0.0664 | 79.0 | 14457 | 1.4732 | 0.8073 |
151
+ | 0.0602 | 80.0 | 14640 | 1.4925 | 0.8078 |
152
+ | 0.0602 | 81.0 | 14823 | 1.4812 | 0.8064 |
153
+ | 0.057 | 82.0 | 15006 | 1.4950 | 0.8013 |
154
+ | 0.057 | 83.0 | 15189 | 1.4785 | 0.8056 |
155
+ | 0.057 | 84.0 | 15372 | 1.4856 | 0.7993 |
156
+ | 0.0517 | 85.0 | 15555 | 1.4755 | 0.8034 |
157
+ | 0.0517 | 86.0 | 15738 | 1.4813 | 0.8034 |
158
+ | 0.0517 | 87.0 | 15921 | 1.4966 | 0.8048 |
159
+ | 0.0468 | 88.0 | 16104 | 1.4883 | 0.8002 |
160
+ | 0.0468 | 89.0 | 16287 | 1.4746 | 0.8023 |
161
+ | 0.0468 | 90.0 | 16470 | 1.4697 | 0.7974 |
162
+ | 0.0426 | 91.0 | 16653 | 1.4775 | 0.8004 |
163
+ | 0.0426 | 92.0 | 16836 | 1.4852 | 0.8023 |
164
+ | 0.0387 | 93.0 | 17019 | 1.4868 | 0.8004 |
165
+ | 0.0387 | 94.0 | 17202 | 1.4785 | 0.8021 |
166
+ | 0.0387 | 95.0 | 17385 | 1.4892 | 0.8015 |
167
+ | 0.0359 | 96.0 | 17568 | 1.4862 | 0.8018 |
168
+ | 0.0359 | 97.0 | 17751 | 1.4851 | 0.8007 |
169
+ | 0.0359 | 98.0 | 17934 | 1.4846 | 0.7999 |
170
+ | 0.0347 | 99.0 | 18117 | 1.4852 | 0.7993 |
171
+ | 0.0347 | 100.0 | 18300 | 1.4848 | 0.8004 |
172
 
173
 
174
  ### Framework versions
.ipynb_checkpoints/run-checkpoint.sh CHANGED
@@ -24,5 +24,4 @@ python run_speech_recognition_ctc.py \
24
  --do_eval \
25
  --max_duration_in_seconds="6" \
26
  --evaluation_strategy='epoch' \
27
- --save_strategy='epoch' \
28
- --ignore_data_skip \
 
24
  --do_eval \
25
  --max_duration_in_seconds="6" \
26
  --evaluation_strategy='epoch' \
27
+ --save_strategy='epoch'
 
.ipynb_checkpoints/run_eval-checkpoint.sh CHANGED
@@ -1,8 +1,6 @@
1
  python eval.py \
2
  --model_id="ivanlau/wav2vec2-large-xls-r-300m-cantonese" \
3
- --dataset="speech-recognition-community-v2/dev_data" \
4
  --config="zh-HK" \
5
- --split="validation" \
6
- --chunk_length_s="5.0" \
7
- --stride_length_s="1.0" \
8
  --log_outputs \
 
1
  python eval.py \
2
  --model_id="ivanlau/wav2vec2-large-xls-r-300m-cantonese" \
3
+ --dataset="mozilla-foundation/common_voice_8_0" \
4
  --config="zh-HK" \
5
+ --split="test" \
 
 
6
  --log_outputs \
README.md CHANGED
@@ -7,10 +7,23 @@ tags:
7
  - mozilla-foundation/common_voice_8_0
8
  - generated_from_trainer
9
  datasets:
10
- - common_voice
11
- model-index:
12
- - name: ''
13
- results: []
 
 
 
 
 
 
 
 
 
 
 
 
 
14
  ---
15
 
16
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 
7
  - mozilla-foundation/common_voice_8_0
8
  - generated_from_trainer
9
  datasets:
10
+ - mozilla-foundation/common_voice_8_0
11
+ - name: XLS-R-300M - Chinese_HongKong (Cantonese)
12
+ results:
13
+ - task:
14
+ name: Automatic Speech Recognition
15
+ type: automatic-speech-recognition
16
+ dataset:
17
+ name: Common Voice 8
18
+ type: mozilla-foundation/common_voice_7_0
19
+ args: zh-HK
20
+ metrics:
21
+ - name: Test WER
22
+ type: wer
23
+ value: 0.8111349803079126
24
+ - name: Test CER
25
+ type: cer
26
+ value: 0.21962250882996914
27
  ---
28
 
29
  <!-- This model card has been generated automatically according to the information the Trainer had access to. You
evaluation_no_lm/.ipynb_checkpoints/mozilla-foundation_common_voice_8_0_zh-HK_test_eval_results-checkpoint.txt ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ WER: 0.8111349803079126
2
+ CER: 0.21962250882996914
evaluation_no_lm/log_mozilla-foundation_common_voice_8_0_zh-HK_test_predictions.txt ADDED
The diff for this file is too large to render. See raw diff
 
evaluation_no_lm/log_mozilla-foundation_common_voice_8_0_zh-HK_test_targets.txt ADDED
The diff for this file is too large to render. See raw diff
 
evaluation_no_lm/mozilla-foundation_common_voice_8_0_zh-HK_test_eval_results.txt ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ WER: 0.8111349803079126
2
+ CER: 0.21962250882996914
run.sh CHANGED
@@ -24,5 +24,4 @@ python run_speech_recognition_ctc.py \
24
  --do_eval \
25
  --max_duration_in_seconds="6" \
26
  --evaluation_strategy='epoch' \
27
- --save_strategy='epoch' \
28
- --ignore_data_skip \
 
24
  --do_eval \
25
  --max_duration_in_seconds="6" \
26
  --evaluation_strategy='epoch' \
27
+ --save_strategy='epoch'
 
run_eval.sh CHANGED
@@ -1,8 +1,6 @@
1
  python eval.py \
2
  --model_id="ivanlau/wav2vec2-large-xls-r-300m-cantonese" \
3
- --dataset="speech-recognition-community-v2/dev_data" \
4
  --config="zh-HK" \
5
- --split="validation" \
6
- --chunk_length_s="5.0" \
7
- --stride_length_s="1.0" \
8
  --log_outputs \
 
1
  python eval.py \
2
  --model_id="ivanlau/wav2vec2-large-xls-r-300m-cantonese" \
3
+ --dataset="mozilla-foundation/common_voice_8_0" \
4
  --config="zh-HK" \
5
+ --split="test" \
 
 
6
  --log_outputs \