nithinraok commited on
Commit
b202bcd
1 Parent(s): 225159b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +289 -0
README.md CHANGED
@@ -1,3 +1,292 @@
1
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
  license: cc-by-4.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - en
4
+ library_name: nemo
5
+ datasets:
6
+ - VOXCELEB-1
7
+ - VOXCELEB-2
8
+ - FISHER
9
+ - switchboard
10
+ - librispeech_asr
11
+ - SRE (2004-2010)
12
+ thumbnail: null
13
+ tags:
14
+ - speaker
15
+ - speech
16
+ - audio
17
+ - speaker-verification
18
+ - speaker-recognition
19
+ - speaker-diarization
20
+ - titanet
21
+ - NeMo
22
+ - pytorch
23
  license: cc-by-4.0
24
+ widget:
25
+ - example_title: Librispeech sample 1
26
+ src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
27
+ - example_title: Librispeech sample 2
28
+ src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
29
+ model-index:
30
+ - name: speakerverification_en_titanet_large
31
+ results:
32
+ - task:
33
+ name: Speaker Verification
34
+ type: speaker-verification
35
+ dataset:
36
+ name: VoxCeleb-1 (Cleaned)
37
+ type: voxceleb1-O
38
+ config: clean
39
+ split: test
40
+ args:
41
+ language: en
42
+ metrics:
43
+ - name: Test EER
44
+ type: eer
45
+ value: 0.66
46
+ - task:
47
+ type: Speaker Diarization
48
+ name: speaker-diarization
49
+ dataset:
50
+ name: AMI (MixHeadset)
51
+ type: ami_diarization
52
+ config: oracle-vad-known-number-of-speakers
53
+ split: test
54
+ args:
55
+ language: en
56
+ metrics:
57
+ - name: Test DER
58
+ type: der
59
+ value: 1.73
60
+ - task:
61
+ type: Speaker Diarization
62
+ name: speaker-diarization
63
+ dataset:
64
+ name: AMI (MixHeadset)
65
+ type: ami_diarization
66
+ config: oracle-vad-unknown-number-of-speakers
67
+ split: test
68
+ args:
69
+ language: en
70
+ metrics:
71
+ - name: Test DER
72
+ type: der
73
+ value: 1.89
74
+ - task:
75
+ type: Speaker Diarization
76
+ name: speaker-diarization
77
+ dataset:
78
+ name: AMI (Lapel)
79
+ type: ami_diarization
80
+ config: oracle-vad-known-number-of-speakers
81
+ split: test
82
+ args:
83
+ language: en
84
+ metrics:
85
+ - name: Test DER
86
+ type: der
87
+ value: 2.03
88
+ - task:
89
+ type: Speaker Diarization
90
+ name: speaker-diarization
91
+ dataset:
92
+ name: AMI (Lapel)
93
+ type: ami_diarization
94
+ config: oracle-vad-unknown-number-of-speakers
95
+ split: test
96
+ args:
97
+ language: en
98
+ metrics:
99
+ - name: Test DER
100
+ type: der
101
+ value: 2.03
102
+ - task:
103
+ type: Speaker Diarization
104
+ name: speaker-diarization
105
+ dataset:
106
+ name: CH109
107
+ type: callhome_diarization
108
+ config: oracle-vad-known-number-of-speakers
109
+ split: test
110
+ args:
111
+ language: en
112
+ metrics:
113
+ - name: Test DER
114
+ type: der
115
+ value: 1.19
116
+ - task:
117
+ type: Speaker Diarization
118
+ name: speaker-diarization
119
+ dataset:
120
+ name: CH109
121
+ type: callhome_diarization
122
+ config: oracle-vad-unknown-number-of-speakers
123
+ split: test
124
+ args:
125
+ language: en
126
+ metrics:
127
+ - name: Test DER
128
+ type: der
129
+ value: 1.63
130
+ - task:
131
+ type: Speaker Diarization
132
+ name: speaker-diarization
133
+ dataset:
134
+ name: NIST SRE 2000
135
+ type: nist-sre_diarization
136
+ config: oracle-vad-known-number-of-speakers
137
+ split: test
138
+ args:
139
+ language: en
140
+ metrics:
141
+ - name: Test DER
142
+ type: der
143
+ value: 6.73
144
+ - task:
145
+ type: Speaker Diarization
146
+ name: speaker-diarization
147
+ dataset:
148
+ name: NIST SRE 2000
149
+ type: nist-sre_diarization
150
+ config: oracle-vad-unknown-number-of-speakers
151
+ split: test
152
+ args:
153
+ language: en
154
+ metrics:
155
+ - name: Test DER
156
+ type: der
157
+ value: 5.38
158
  ---
159
+
160
+ # NVIDIA TitaNet-Large (en-US)
161
+
162
+ <style>
163
+ img {
164
+ display: inline;
165
+ }
166
+ </style>
167
+
168
+ | [![Model architecture](https://img.shields.io/badge/Model_Arch-TitaNet--Large-lightgrey#model-badge)](#model-architecture)
169
+ | [![Model size](https://img.shields.io/badge/Params-23M-lightgrey#model-badge)](#model-architecture)
170
+ | [![Language](https://img.shields.io/badge/Language-en--US-lightgrey#model-badge)](#datasets)
171
+
172
+
173
+ This model extracts speaker embeddings from given speech, which are backbone for speaker verification and diarization tasks.
174
+ It is a "large" version of TitaNet (around 23M parameters) models.
175
+ See the [model architecture](#model-architecture) section and [NeMo documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/asr/speaker_recognition/models.html#titanet) for complete architecture details.
176
+
177
+ ## NVIDIA NeMo: Training
178
+
179
+ To train, fine-tune or play with the model you will need to install [NVIDIA NeMo](https://github.com/NVIDIA/NeMo). We recommend you install it after you've installed latest Pytorch version.
180
+ ```
181
+ pip install nemo_toolkit['all']
182
+ ```
183
+
184
+ ## How to Use this Model
185
+
186
+ The model is available for use in the NeMo toolkit [3] and can be used as a pre-trained checkpoint for inference or for fine-tuning on another dataset.
187
+
188
+ ### Automatically instantiate the model
189
+
190
+ ```python
191
+ import nemo.collections.asr as nemo_asr
192
+ speaker_model = nemo_asr.models.EncDecSpeakerLabelModel.from_pretrained("nvidia/speakerverification_en_titanet_large")
193
+ ```
194
+
195
+ ### Embedding Extraction
196
+ First, let's get a sample
197
+ ```
198
+ wget https://dldata-public.s3.us-east-2.amazonaws.com/2086-149220-0033.wav
199
+ ```
200
+ Then do:
201
+ ```
202
+ emb = speaker_model.get_embedding('2086-149220-0033.wav']
203
+ ```
204
+
205
+ ### Verifying two utterances (Speaker Verification)
206
+ let's get another sample
207
+ ```
208
+ wget https://dldata-public.s3.us-east-2.amazonaws.com/2086-149220-0033.wav
209
+ ```
210
+ Now to check if two audio files are from same speaker or not, simply do:
211
+ ```
212
+ speaker_model.verify_speakers('path/to/one/audio_file','path/to/other/audio_file')
213
+ ```
214
+
215
+ ### Extracting Embeddings for more audio files
216
+
217
+ To extract embeddings from a bunch of audio files:
218
+
219
+ Write audio files to a `manifest.json` file with lines as in format:
220
+
221
+ ```json
222
+ {"audio_filepath": "<absolute path to dataset>/audio_file.wav", "duration": "duration of file in sec", "label": "speaker_id"}
223
+ ```
224
+ Then running following script will extract embeddings and writes to current working directory:
225
+ ```shell
226
+ python <NeMo_root>/examples/speaker_tasks/recognition/extract_speaker_embeddings.py --manifest=manifest.json
227
+ ```
228
+
229
+ ### Input
230
+
231
+ This model accepts 16000 KHz Mono-channel Audio (wav files) as input.
232
+
233
+ ### Output
234
+
235
+ This model provides speaker embeddings for an audio file.
236
+
237
+ ## Model Architecture
238
+
239
+ TitaNet model is a depth-wise separable conv1D model [1] for Speaker Verification and diarization tasks. You may find more info on the detail of this model here: [TitaNet-Model](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/asr/speaker_recognition/models.html).
240
+
241
+ ## Training
242
+
243
+ The NeMo toolkit [3] was used for training the models for over several hundred epochs. These model are trained with this [example script](https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/recognition/speaker_reco.py) and this [base config](https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/recognition/conf/titanet-large.yaml).
244
+
245
+ ### Datasets
246
+
247
+ All the models in this collection are trained on a composite dataset comprising several thousand hours of English speech:
248
+
249
+ - Voxceleb-1
250
+ - Voxceleb-2
251
+ - Fisher
252
+ - Switchboard
253
+ - Librispeech
254
+ - SRE (2004-2010)
255
+
256
+ ## Performance
257
+
258
+ Performances of the these models are reported in terms of Equal Error Rate (EER%) on speaker verification evaluation trial files and as Diarization Error Rate (DER%) on diarization test sessions.
259
+
260
+ * Speaker Verification (EER%)
261
+ | Version | Model | Model Size | VoxCeleb1 (Cleaned trial file) |
262
+ |---------|--------------|-----|---------------|
263
+ | 1.10.0 | TitaNet-Large | 23M | 0.66 |
264
+
265
+ * Speaker Diarization (DER%)
266
+ | Version | Model | Model Size | Evaluation Condition | NIST SRE 2000 | AMI (Lapel) | AMI (MixHeadset) | CH109 |
267
+ |---------|--------------|-----|----------------------|---------------|-------------|------------------|-------|
268
+ | 1.10.0 | TitaNet-Large | 23M | Oracle VAD KNOWN # of Speakers | 6.73 | 2.03 | 1.73 | 1.19 |
269
+ | 1.10.0 | TitaNet-Large | 23M | Oracle VAD UNKNOWN # of Speakers | 5.38 | 2.03 | 1.89 | 1.63 |
270
+
271
+ ## Limitations
272
+ This model is trained on both telephonic and non-telephonic speech from voxceleb datasets, Fisher and switch board. If your domain of data differs from trained data or doesnot show relatively good performance consider finetuning for that speech domain.
273
+
274
+ ## NVIDIA Riva: Deployment
275
+
276
+ [NVIDIA Riva](https://developer.nvidia.com/riva), is an accelerated speech AI SDK deployable on-prem, in all clouds, multi-cloud, hybrid, on edge, and embedded.
277
+ Additionally, Riva provides:
278
+
279
+ * World-class out-of-the-box accuracy for the most common languages with model checkpoints trained on proprietary data with hundreds of thousands of GPU-compute hours
280
+ * Best in class accuracy with run-time word boosting (e.g., brand and product names) and customization of acoustic model, language model, and inverse text normalization
281
+ * Streaming speech recognition, Kubernetes compatible scaling, and enterprise-grade support
282
+
283
+ Although this model isn’t supported yet by Riva, the [list of supported models is here](https://huggingface.co/models?other=Riva).
284
+ Check out [Riva live demo](https://developer.nvidia.com/riva#demos).
285
+
286
+ ## References
287
+ [1] [TitaNet: Neural Model for Speaker Representation with 1D Depth-wise Separable convolutions and global context](https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9746806)
288
+ [2] [NVIDIA NeMo Toolkit](https://github.com/NVIDIA/NeMo)
289
+
290
+ ## Licence
291
+
292
+ License to use this model is covered by the [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/). By downloading the public and release version of the model, you accept the terms and conditions of the [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) license.