Mirco commited on
Commit
7320fc0
1 Parent(s): 0eb0c05

model upload

Browse files

Files changed (10) hide show
  1. .gitattributes +4 -0
  2. README.md +134 -0
  3. asr.ckpt +3 -0
  4. hyperparams.yaml +152 -0
  5. lm.ckpt +3 -0
  6. normalizer.ckpt +3 -0
  7. record_0_16k.wav +0 -0
  8. record_1_16k.wav +0 -0
  9. record_2_16k.wav +0 -0
  10. tokenizer.ckpt +3 -0
.gitattributes CHANGED
@@ -25,3 +25,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
25
  *.zip filter=lfs diff=lfs merge=lfs -text
26
  *.zstandard filter=lfs diff=lfs merge=lfs -text
27
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
25
  *.zip filter=lfs diff=lfs merge=lfs -text
26
  *.zstandard filter=lfs diff=lfs merge=lfs -text
27
  *tfevents* filter=lfs diff=lfs merge=lfs -text
28
+ asr.ckpt filter=lfs diff=lfs merge=lfs -text
29
+ lm.ckpt filter=lfs diff=lfs merge=lfs -text
30
+ normalizer.ckpt filter=lfs diff=lfs merge=lfs -text
31
+ tokenizer.ckpt filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,134 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: "kr"
3
+ thumbnail:
4
+ tags:
5
+ - ASR
6
+ - CTC
7
+ - Attention
8
+ - Conformer
9
+ - pytorch
10
+ - speechbrain
11
+ license: "apache-2.0"
12
+ datasets:
13
+ - ksponspeech
14
+ metrics:
15
+ - wer
16
+ - cer
17
+ ---
18
+
19
+ <iframe src="https://ghbtns.com/github-btn.html?user=speechbrain&repo=speechbrain&type=star&count=true&size=large&v=2" frameborder="0" scrolling="0" width="170" height="30" title="GitHub"></iframe>
20
+ <br/><br/>
21
+
22
+ # Conformer for KsponSpeech (with Transformer LM)
23
+
24
+ This repository provides all the necessary tools to perform automatic speech
25
+ recognition from an end-to-end system pretrained on KsponSpeech (Kr) within
26
+ SpeechBrain. For a better experience, we encourage you to learn more about
27
+ [SpeechBrain](https://speechbrain.github.io).
28
+ The performance of the model is the following:
29
+
30
+ | Release | eval clean CER | eval other CER | GPUs |
31
+ | :------: | :------------: | :------------: | :---------: |
32
+ | 09-05-21 | 7.48% | 8.38% | 6xA100 80GB |
33
+
34
+ ## Pipeline description
35
+
36
+ This ASR system is composed of 3 different but linked blocks:
37
+ - Tokenizer (unigram) that transforms words into subword units and trained with
38
+ the train transcriptions of KsponSpeech.
39
+ - Neural language model (Transformer LM) trained on the train transcriptions of KsponSpeech
40
+ - Acoustic model made of a conformer encoder and a joint decoder with CTC +
41
+ transformer. Hence, the decoding also incorporates the CTC probabilities.
42
+ ## Install SpeechBrain
43
+ First of all, please install SpeechBrain with the following command:
44
+ ```
45
+ !pip install git+https://github.com/speechbrain/speechbrain.git@develop
46
+ ```
47
+ Please notice that we encourage you to read our tutorials and learn more about
48
+ [SpeechBrain](https://speechbrain.github.io).
49
+ ### Transcribing your own audio files (in Korean)
50
+ ```python
51
+ from speechbrain.pretrained import EncoderDecoderASR
52
+ asr_model = EncoderDecoderASR.from_hparams(source="ddwkim/asr-conformer-transformerlm-ksponspeech", savedir="pretrained_models/asr-conformer-transformerlm-ksponspeech", run_opts={"device":"cuda"})
53
+ asr_model.transcribe_file("ddwkim/asr-conformer-transformerlm-ksponspeech/record_0_16k.wav")
54
+ ```
55
+
56
+ ### Inference on GPU
57
+
58
+ To perform inference on the GPU, add `run_opts={"device":"cuda"}` when calling the `from_hparams` method.
59
+
60
+ ## Parallel Inference on a Batch
61
+
62
+ Please, [see this Colab notebook](https://colab.research.google.com/drive/10N98aGoeLGfh6Hu6xOCH5BbjVTVYgCyB?usp=sharing) on using the pretrained model
63
+
64
+ ### Training
65
+
66
+ The model was trained with SpeechBrain (Commit hash: 'fd9826c').
67
+ To train it from scratch follow these steps:
68
+ 1. Clone SpeechBrain:
69
+ ```bash
70
+ git clone https://github.com/speechbrain/speechbrain/
71
+ ```
72
+ 2. Install it:
73
+ ```bash
74
+ cd speechbrain
75
+ pip install -r requirements.txt
76
+ pip install .
77
+ ```
78
+ 3. Run Training:
79
+ ```bash
80
+ cd recipes/KsponSpeech/ASR/transformer
81
+ python train.py hparams/conformer_medium.yaml --data_folder=your_data_folder
82
+ ```
83
+ You can find our training results (models, logs, etc) at the subdirectories.
84
+
85
+ ### Limitations
86
+
87
+ The SpeechBrain team does not provide any warranty on the performance achieved by this model when used on other datasets.
88
+
89
+ # **About SpeechBrain**
90
+
91
+ - Website: https://speechbrain.github.io/
92
+ - Code: https://github.com/speechbrain/speechbrain/
93
+ - HuggingFace: https://huggingface.co/speechbrain/
94
+
95
+ # **Citing SpeechBrain**
96
+
97
+ Please, cite SpeechBrain if you use it for your research or business.
98
+ ```bibtex
99
+ @misc{speechbrain,
100
+ title={{SpeechBrain}: A General-Purpose Speech Toolkit},
101
+ author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
102
+ year={2021},
103
+ eprint={2106.04624},
104
+ archivePrefix={arXiv},
105
+ primaryClass={eess.AS},
106
+ note={arXiv:2106.04624}
107
+ }
108
+ ```
109
+
110
+ # Citing the model
111
+ ```bibtex
112
+ @misc{returnzero,
113
+ title = {ReturnZero Conformer Korean ASR model},
114
+ author = {Dongwon Kim and Dongwoo Kim and Roh Jeongkyu},
115
+ year = {2021},
116
+ howpublished = {\url{https://huggingface.co/ddwkim/asr-conformer-transformerlm-ksponspeech}},
117
+ }
118
+ ```
119
+
120
+ # Citing KsponSpeech dataset
121
+ ```bibtex
122
+ @Article{app10196936,
123
+ AUTHOR = {Bang, Jeong-Uk and Yun, Seung and Kim, Seung-Hi and Choi, Mu-Yeol and Lee, Min-Kyu and Kim, Yeo-Jeong and Kim, Dong-Hyun and Park, Jun and Lee, Young-Jik and Kim, Sang-Hun},
124
+ TITLE = {KsponSpeech: Korean Spontaneous Speech Corpus for Automatic Speech Recognition},
125
+ JOURNAL = {Applied Sciences},
126
+ VOLUME = {10},
127
+ YEAR = {2020},
128
+ NUMBER = {19},
129
+ ARTICLE-NUMBER = {6936},
130
+ URL = {https://www.mdpi.com/2076-3417/10/19/6936},
131
+ ISSN = {2076-3417},
132
+ DOI = {10.3390/app10196936}
133
+ }
134
+ ```
asr.ckpt ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:893a5fb84a67315a954d7645fd3b5f96cee806531f538e0073f6dcdf17dcf7c3
3
+ size 183510489
hyperparams.yaml ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ############################################################################
2
+ # Model: E2E ASR with Transformer
3
+ # Encoder: Conformer Encoder
4
+ # Decoder: Transformer Decoder + (CTC/ATT joint) beamsearch + TransformerLM
5
+ # Tokens: unigram
6
+ # losses: CTC + KLdiv (Label Smoothing loss)
7
+ # Training: KsponSpeech 965.2h
8
+ # Based on the works of: Jianyuan Zhong, Titouan Parcollet 2021
9
+ # Authors: Dongwon Kim, Dongwoo Kim 2021
10
+ # ############################################################################
11
+ # Seed needs to be set at top of yaml, before objects with parameters are made
12
+
13
+ # Feature parameters
14
+ sample_rate: 16000
15
+ n_fft: 400
16
+ n_mels: 80
17
+
18
+ ####################### Model parameters ###########################
19
+ # Transformer
20
+ d_model: 256
21
+ nhead: 4
22
+ num_encoder_layers: 12
23
+ num_decoder_layers: 6
24
+ d_ffn: 2048
25
+ transformer_dropout: 0.0
26
+ activation: !name:torch.nn.GELU
27
+ output_neurons: 5000
28
+ vocab_size: 5000
29
+
30
+ # Outputs
31
+ blank_index: 0
32
+ label_smoothing: 0.1
33
+ pad_index: 0
34
+ bos_index: 1
35
+ eos_index: 2
36
+ unk_index: 0
37
+
38
+ # Decoding parameters
39
+ min_decode_ratio: 0.0
40
+ max_decode_ratio: 1.0
41
+ valid_search_interval: 10
42
+ valid_beam_size: 10
43
+ test_beam_size: 60
44
+ lm_weight: 0.20
45
+ ctc_weight_decode: 0.40
46
+
47
+ ############################## models ################################
48
+
49
+ normalizer: !new:speechbrain.processing.features.InputNormalization
50
+ norm_type: global
51
+
52
+ CNN: !new:speechbrain.lobes.models.convolution.ConvolutionFrontEnd
53
+ input_shape: (8, 10, 80)
54
+ num_blocks: 2
55
+ num_layers_per_block: 1
56
+ out_channels: (64, 32)
57
+ kernel_sizes: (3, 3)
58
+ strides: (2, 2)
59
+ residuals: (False, False)
60
+
61
+ Transformer: !new:speechbrain.lobes.models.transformer.TransformerASR.TransformerASR # yamllint disable-line rule:line-length
62
+ input_size: 640
63
+ tgt_vocab: !ref <output_neurons>
64
+ d_model: !ref <d_model>
65
+ nhead: !ref <nhead>
66
+ num_encoder_layers: !ref <num_encoder_layers>
67
+ num_decoder_layers: !ref <num_decoder_layers>
68
+ d_ffn: !ref <d_ffn>
69
+ dropout: !ref <transformer_dropout>
70
+ activation: !ref <activation>
71
+ encoder_module: conformer
72
+ attention_type: RelPosMHAXL
73
+ normalize_before: True
74
+ causal: False
75
+
76
+ # NB: It has to match the pre-trained TransformerLM!!
77
+ lm_model: !new:speechbrain.lobes.models.transformer.TransformerLM.TransformerLM # yamllint disable-line rule:line-length
78
+ vocab: !ref <output_neurons>
79
+ d_model: 768
80
+ nhead: 12
81
+ num_encoder_layers: 12
82
+ num_decoder_layers: 0
83
+ d_ffn: 3072
84
+ dropout: 0.0
85
+ activation: !name:torch.nn.GELU
86
+ normalize_before: False
87
+
88
+ tokenizer: !new:sentencepiece.SentencePieceProcessor
89
+
90
+ ctc_lin: !new:speechbrain.nnet.linear.Linear
91
+ input_size: !ref <d_model>
92
+ n_neurons: !ref <output_neurons>
93
+
94
+ seq_lin: !new:speechbrain.nnet.linear.Linear
95
+ input_size: !ref <d_model>
96
+ n_neurons: !ref <output_neurons>
97
+
98
+ decoder: !new:speechbrain.decoders.S2STransformerBeamSearch
99
+ modules: [!ref <Transformer>, !ref <seq_lin>, !ref <ctc_lin>]
100
+ bos_index: !ref <bos_index>
101
+ eos_index: !ref <eos_index>
102
+ blank_index: !ref <blank_index>
103
+ min_decode_ratio: !ref <min_decode_ratio>
104
+ max_decode_ratio: !ref <max_decode_ratio>
105
+ beam_size: !ref <test_beam_size>
106
+ ctc_weight: !ref <ctc_weight_decode>
107
+ lm_weight: !ref <lm_weight>
108
+ lm_modules: !ref <lm_model>
109
+ temperature: 1.25
110
+ temperature_lm: 1.25
111
+ using_eos_threshold: False
112
+ length_normalization: True
113
+
114
+ Tencoder: !new:speechbrain.lobes.models.transformer.TransformerASR.EncoderWrapper
115
+ transformer: !ref <Transformer>
116
+
117
+ encoder: !new:speechbrain.nnet.containers.LengthsCapableSequential
118
+ input_shape: [null, null, !ref <n_mels>]
119
+ compute_features: !ref <compute_features>
120
+ normalize: !ref <normalizer>
121
+ cnn: !ref <CNN>
122
+ transformer_encoder: !ref <Tencoder>
123
+
124
+ asr_model: !new:torch.nn.ModuleList
125
+ - [!ref <normalizer>, !ref <CNN>, !ref <Transformer>, !ref <seq_lin>, !ref <ctc_lin>]
126
+
127
+ log_softmax: !new:torch.nn.LogSoftmax
128
+ dim: -1
129
+
130
+
131
+ compute_features: !new:speechbrain.lobes.features.Fbank
132
+ sample_rate: !ref <sample_rate>
133
+ n_fft: !ref <n_fft>
134
+ n_mels: !ref <n_mels>
135
+
136
+ modules:
137
+ compute_features: !ref <compute_features>
138
+ normalizer: !ref <normalizer>
139
+ pre_transformer: !ref <CNN>
140
+ transformer: !ref <Transformer>
141
+ asr_model: !ref <asr_model>
142
+ lm_model: !ref <lm_model>
143
+ encoder: !ref <encoder>
144
+ decoder: !ref <decoder>
145
+ # The pretrainer allows a mapping between pretrained files and instances that
146
+ # are declared in the yaml.
147
+ pretrainer: !new:speechbrain.utils.parameter_transfer.Pretrainer
148
+ loadables:
149
+ normalizer: !ref <normalizer>
150
+ asr: !ref <asr_model>
151
+ lm: !ref <lm_model>
152
+ tokenizer: !ref <tokenizer>
lm.ckpt ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ee4a5a5d9ce11e24dcea93f24a241528b9b376798be6478c70fb279736515110
3
+ size 381074814
normalizer.ckpt ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f4866d96b29f5c97526c7469aa6f58cd50aeb9865b457daf599f0f42e5827be9
3
+ size 1783
record_0_16k.wav ADDED
Binary file
record_1_16k.wav ADDED
Binary file
record_2_16k.wav ADDED
Binary file
tokenizer.ckpt ADDED
@@ -0,0 +1,3 @@
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5e095c023a42b6bd25352512597a245db9bf9126ce6bf64082bd41d0a196b220
3
+ size 313899