sanchit-gandhi HF staff commited on
Commit
d47368d
1 Parent(s): 2bfe928

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +558 -0
README.md ADDED
@@ -0,0 +1,558 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - audio
6
+ - automatic-speech-recognition
7
+ widget:
8
+ - example_title: LibriSpeech sample 1
9
+ src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
10
+ - example_title: LibriSpeech sample 2
11
+ src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
12
+ pipeline_tag: automatic-speech-recognition
13
+ license: mit
14
+ library_name: transformers
15
+ ---
16
+
17
+ # Distil-Whisper: distil-small.en
18
+
19
+ Distil-Whisper was proposed in the paper [Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://arxiv.org/abs/2311.00430).
20
+ It is a distilled version of the Whisper model that is **6 times faster**, 49% smaller, and performs **within 1% WER**
21
+ on out-of-distribution evaluation sets.
22
+
23
+ This is the repository for distil-small.en, a distilled variant of [Whisper small.en](https://huggingface.co/openai/whisper-small.en).
24
+ It is the **smallest Distil-Whisper checkpoint**, with just 166M parameters, making it the ideal choice for memory
25
+ constrained applications (e.g. on-device).
26
+
27
+ For most other applications, the [distil-medium.en](https://huggingface.co/distil-whisper/distil-medium.en)
28
+ or [distil-large-v2](https://huggingface.co/distil-whisper/distil-large-v2) checkpoints are recommended, since they are
29
+ both faster and achieve better WER results:
30
+
31
+ | Model | Params / M | Rel. Latency | Short-Form WER | Long-Form WER |
32
+ |----------------------------------------------------------------------------|------------|--------------|----------------|---------------|
33
+ | [large-v2](https://huggingface.co/openai/whisper-large-v2) | 1550 | 1.0 | **9.1** | 11.7 |
34
+ | | | | | |
35
+ | [distil-large-v2](https://huggingface.co/distil-whisper/distil-large-v2) | 756 | 5.8 | 10.1 | **11.6** |
36
+ | [distil-medium.en](https://huggingface.co/distil-whisper/distil-medium.en) | 394 | **6.8** | 11.1 | 12.4 |
37
+ | [distil-small.en](https://huggingface.co/distil-whisper/distil-small.en) | **166** | 5.6 | 12.1 | 12.8 |
38
+
39
+ **Note:** Distil-Whisper is currently only available for English speech recognition. We are working with the community
40
+ to distill Whisper on other languages. If you are interested in distilling Whisper in your language, check out the
41
+ provided [training code](https://github.com/huggingface/distil-whisper/tree/main/training). We will update the
42
+ [Distil-Whisper repository](https://github.com/huggingface/distil-whisper/) with multilingual checkpoints when ready!
43
+
44
+ ### Why is `distil-small.en` slower than `distil-large-v2`?
45
+
46
+ While [distil-medium.en](https://huggingface.co/distil-whisper/distil-medium.en) and [distil-large-v2](https://huggingface.co/distil-whisper/distil-large-v2)
47
+ use two decoder layers each, distil-small.en uses four. Using more decoder layers improves the WER performance of the
48
+ model, at the expense of slower inference speed. We found that four layers was the minimum required to get reasonable
49
+ WER performance for `distil-small.en`, where it performs to within 3% WER of Whisper [large-v2](https://huggingface.co/openai/whisper-large-v2)
50
+ while being 5.6x faster. When we tried distilling with just two layers, the model was over 5% worse than large-v2, albeit
51
+ 7.8x faster. We leave distilling a two layer small.en model as future works.
52
+
53
+ ## Usage
54
+
55
+ Distil-Whisper is supported in Hugging Face 🤗 Transformers from version 4.35 onwards. To run the model, first
56
+ install the latest version of the Transformers library. For this example, we'll also install 🤗 Datasets to load toy
57
+ audio dataset from the Hugging Face Hub:
58
+
59
+ ```bash
60
+ pip install --upgrade pip
61
+ pip install --upgrade transformers accelerate datasets[audio]
62
+ ```
63
+
64
+ ### Short-Form Transcription
65
+
66
+ The model can be used with the [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline)
67
+ class to transcribe short-form audio files (< 30-seconds) as follows:
68
+
69
+ ```python
70
+ import torch
71
+ from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
72
+ from datasets import load_dataset
73
+
74
+
75
+ device = "cuda:0" if torch.cuda.is_available() else "cpu"
76
+ torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
77
+
78
+ model_id = "distil-whisper/distil-small.en"
79
+
80
+ model = AutoModelForSpeechSeq2Seq.from_pretrained(
81
+ model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
82
+ )
83
+ model.to(device)
84
+
85
+ processor = AutoProcessor.from_pretrained(model_id)
86
+
87
+ pipe = pipeline(
88
+ "automatic-speech-recognition",
89
+ model=model,
90
+ tokenizer=processor.tokenizer,
91
+ feature_extractor=processor.feature_extractor,
92
+ max_new_tokens=128,
93
+ torch_dtype=torch_dtype,
94
+ device=device,
95
+ )
96
+
97
+ dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
98
+ sample = dataset[0]["audio"]
99
+
100
+ result = pipe(sample)
101
+ print(result["text"])
102
+ ```
103
+
104
+ To transcribe a local audio file, simply pass the path to your audio file when you call the pipeline:
105
+ ```diff
106
+ - result = pipe(sample)
107
+ + result = pipe("audio.mp3")
108
+ ```
109
+
110
+ ### Long-Form Transcription
111
+
112
+ Distil-Whisper uses a chunked algorithm to transcribe long-form audio files (> 30-seconds). In practice, this chunked long-form algorithm
113
+ is 9x faster than the sequential algorithm proposed by OpenAI in the Whisper paper (see Table 7 of the [Distil-Whisper paper](https://arxiv.org/abs/2311.00430)).
114
+
115
+ To enable chunking, pass the `chunk_length_s` parameter to the `pipeline`. For Distil-Whisper, a chunk length of 15-seconds
116
+ is optimal. To activate batching, pass the argument `batch_size`:
117
+
118
+ ```python
119
+ import torch
120
+ from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
121
+ from datasets import load_dataset
122
+
123
+
124
+ device = "cuda:0" if torch.cuda.is_available() else "cpu"
125
+ torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
126
+
127
+ model_id = "distil-whisper/distil-small.en"
128
+
129
+ model = AutoModelForSpeechSeq2Seq.from_pretrained(
130
+ model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
131
+ )
132
+ model.to(device)
133
+
134
+ processor = AutoProcessor.from_pretrained(model_id)
135
+
136
+ pipe = pipeline(
137
+ "automatic-speech-recognition",
138
+ model=model,
139
+ tokenizer=processor.tokenizer,
140
+ feature_extractor=processor.feature_extractor,
141
+ max_new_tokens=128,
142
+ chunk_length_s=15,
143
+ batch_size=16,
144
+ torch_dtype=torch_dtype,
145
+ device=device,
146
+ )
147
+
148
+ dataset = load_dataset("distil-whisper/librispeech_long", "default", split="validation")
149
+ sample = dataset[0]["audio"]
150
+
151
+ result = pipe(sample)
152
+ print(result["text"])
153
+ ```
154
+
155
+ <!---
156
+ **Tip:** The pipeline can also be used to transcribe an audio file from a remote URL, for example:
157
+
158
+ ```python
159
+ result = pipe("https://huggingface.co/datasets/sanchit-gandhi/librispeech_long/resolve/main/audio.wav")
160
+ ```
161
+ --->
162
+
163
+ ### Speculative Decoding
164
+
165
+ Distil-Whisper can be used as an assistant model to Whisper for speculative decoding. Speculative decoding mathematically
166
+ ensures the exact same outputs as Whisper are obtained while being 2 times faster. This makes it the perfect drop-in
167
+ replacement for existing Whisper pipelines, since the same outputs are guaranteed.
168
+
169
+ In the following code-snippet, we load the assistant Distil-Whisper model standalone to the main Whisper pipeline. We then
170
+ specify it as the "assistant model" for generation:
171
+
172
+ ```python
173
+ from transformers import pipeline, AutoModelForCausalLM, AutoModelForSpeechSeq2Seq, AutoProcessor
174
+ import torch
175
+ from datasets import load_dataset
176
+
177
+ device = "cuda:0" if torch.cuda.is_available() else "cpu"
178
+ torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
179
+
180
+ assistant_model_id = "distil-whisper/distil-small.en"
181
+
182
+ assistant_model = AutoModelForSpeechSeq2Seq.from_pretrained(
183
+ assistant_model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
184
+ )
185
+ assistant_model.to(device)
186
+
187
+ model_id = "openai/whisper-medium.en"
188
+
189
+ model = AutoModelForSpeechSeq2Seq.from_pretrained(
190
+ model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
191
+ )
192
+ model.to(device)
193
+
194
+ processor = AutoProcessor.from_pretrained(model_id)
195
+
196
+ pipe = pipeline(
197
+ "automatic-speech-recognition",
198
+ model=model,
199
+ tokenizer=processor.tokenizer,
200
+ feature_extractor=processor.feature_extractor,
201
+ max_new_tokens=128,
202
+ generate_kwargs={"assistant_model": assistant_model},
203
+ torch_dtype=torch_dtype,
204
+ device=device,
205
+ )
206
+
207
+ dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
208
+ sample = dataset[0]["audio"]
209
+
210
+ result = pipe(sample)
211
+ print(result["text"])
212
+ ```
213
+
214
+ ## Additional Speed & Memory Improvements
215
+
216
+ You can apply additional speed and memory improvements to Distil-Whisper which we cover in the following.
217
+
218
+ ### Flash Attention
219
+
220
+ We recommend using [Flash-Attention 2](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#flashattention-2) if your GPU allows for it.
221
+ To do so, you first need to install [Flash Attention](https://github.com/Dao-AILab/flash-attention):
222
+
223
+ ```
224
+ pip install flash-attn --no-build-isolation
225
+ ```
226
+
227
+ and then all you have to do is to pass `use_flash_attention_2=True` to `from_pretrained`:
228
+
229
+ ```diff
230
+ - model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True)
231
+ + model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True, use_flash_attention_2=True)
232
+ ```
233
+
234
+ ### Torch Scale-Product-Attention (SDPA)
235
+
236
+ If your GPU does not support Flash Attention, we recommend making use of [BetterTransformers](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#bettertransformer).
237
+ To do so, you first need to install optimum:
238
+
239
+ ```
240
+ pip install --upgrade optimum
241
+ ```
242
+
243
+ And then convert your model to a "BetterTransformer" model before using it:
244
+
245
+ ```diff
246
+ model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True)
247
+ + model = model.to_bettertransformer()
248
+ ```
249
+
250
+ ### Running Distil-Whisper in `openai-whisper`
251
+
252
+ Coming soon!
253
+
254
+ <!---
255
+
256
+ To use the model in the original Whisper format, first ensure you have the [`openai-whisper`](https://pypi.org/project/openai-whisper/) package installed:
257
+
258
+ ```bash
259
+ pip install --upgrade openai-whisper
260
+ ```
261
+
262
+ The following code-snippet demonstrates how to transcribe a sample file from the LibriSpeech dataset loaded using
263
+ 🤗 Datasets:
264
+
265
+ ```python
266
+ import torch
267
+ from datasets import load_dataset
268
+ from huggingface_hub import hf_hub_download
269
+ from whisper import load_model, transcribe
270
+
271
+ medium_en = hf_hub_download(repo_id="distil-whisper/distil-small.en", filename="original-model.bin")
272
+ model = load_model(medium_en)
273
+
274
+ dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
275
+ sample = dataset[0]["audio"]["array"]
276
+ sample = torch.from_numpy(sample).float()
277
+
278
+ pred_out = transcribe(model, audio=sample)
279
+ print(pred_out["text"])
280
+ ```
281
+
282
+ To transcribe a local audio file, simply pass the path to the audio file as the `audio` argument to transcribe:
283
+
284
+ ```python
285
+ pred_out = transcribe(model, audio="audio.mp3")
286
+ ```
287
+ --->
288
+
289
+ ### Whisper.cpp
290
+
291
+ Coming soon!
292
+
293
+ <!---
294
+
295
+ Distil-Whisper can be run from the [Whisper.cpp](https://github.com/ggerganov/whisper.cpp) repository with the original
296
+ sequential long-form transcription algorithm. In a [provisional benchmark](https://github.com/ggerganov/whisper.cpp/pull/1424#issuecomment-1793513399)
297
+ on Mac M1, `distil-medium.en` is 4x faster than `large-v2`, while performing to within 1% WER over long-form audio.
298
+
299
+ Steps for getting started:
300
+ 1. Clone the Whisper.cpp repository:
301
+ ```
302
+ git clone https://github.com/ggerganov/whisper.cpp.git
303
+ cd whisper.cpp
304
+ ```
305
+ 2. Download the ggml weights for `distil-small.en` from the Hugging Face Hub:
306
+
307
+ ```bash
308
+ python -c "from huggingface_hub import hf_hub_download; hf_hub_download(repo_id='distil-whisper/distil-small.en', filename='ggml-medium-32-2.en.bin', local_dir='./models')"
309
+ ```
310
+
311
+ Note that if you do not have the `huggingface_hub` package installed, you can also download the weights with `wget`:
312
+
313
+ ```bash
314
+ wget https://huggingface.co/distil-whisper/distil-small.en/resolve/main/ggml-medium-32-2.en.bin -P ./models
315
+ ```
316
+
317
+ 3. Run inference using the provided sample audio:
318
+
319
+ ```bash
320
+ make -j && ./main -m models/ggml-medium-32-2.en.bin -f samples/jfk.wav
321
+ ```
322
+
323
+ --->
324
+
325
+ ### Transformers.js
326
+
327
+ ```js
328
+ import { pipeline } from '@xenova/transformers';
329
+
330
+ let transcriber = await pipeline('automatic-speech-recognition', 'distil-whisper/distil-small.en');
331
+
332
+ let url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav';
333
+ let output = await transcriber(url);
334
+ // { text: " And so my fellow Americans, ask not what your country can do for you. Ask what you can do for your country." }
335
+ ```
336
+
337
+ See the [docs](https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.AutomaticSpeechRecognitionPipeline) for more information.
338
+
339
+ ### Candle
340
+
341
+ Coming soon!
342
+
343
+ <!---
344
+
345
+ Through an integration with Hugging Face [Candle](https://github.com/huggingface/candle/tree/main) 🕯️, Distil-Whisper is
346
+ now available in the Rust library 🦀
347
+
348
+ Benefit from:
349
+ * Optimised CPU backend with optional MKL support for x86 and Accelerate for Macs
350
+ * CUDA backend for efficiently running on GPUs, multiple GPU distribution via NCCL
351
+ * WASM support: run Distil-Whisper in a browser
352
+
353
+ Steps for getting started:
354
+ 1. Install [`candle-core`](https://github.com/huggingface/candle/tree/main/candle-core) as explained [here](https://huggingface.github.io/candle/guide/installation.html)
355
+ 2. Clone the `candle` repository locally:
356
+ ```
357
+ git clone https://github.com/huggingface/candle.git
358
+ ```
359
+ 3. Enter the example directory for [Whisper](https://github.com/huggingface/candle/tree/main/candle-examples/examples/whisper):
360
+ ```
361
+ cd candle/candle-examples/examples/whisper
362
+ ```
363
+ 4. Run an example:
364
+ ```
365
+ cargo run --example whisper --release -- --model distil-small.en
366
+ ```
367
+ 5. To specify your own audio file, add the `--input` flag:
368
+ ```
369
+ cargo run --example whisper --release -- --model distil-small.en --input audio.wav
370
+ ```
371
+
372
+ --->
373
+
374
+ ### 8bit & 4bit Quantization
375
+
376
+ Coming soon!
377
+
378
+ ## Model Details
379
+
380
+ Distil-Whisper inherits the encoder-decoder architecture from Whisper. The encoder maps a sequence of speech vector
381
+ inputs to a sequence of hidden-state vectors. The decoder auto-regressively predicts text tokens, conditional on all
382
+ previous tokens and the encoder hidden-states. Consequently, the encoder is only run forward once, whereas the decoder
383
+ is run as many times as the number of tokens generated. In practice, this means the decoder accounts for over 90% of
384
+ total inference time. Thus, to optimise for latency, the focus is on minimising the inference time of the decoder.
385
+
386
+ To distill the Whisper model, we reduce the number of decoder layers while keeping the encoder fixed.
387
+ The encoder (shown in green) is entirely copied from the teacher to the student and frozen during training.
388
+ The student's decoder consists of a subset of the teacher decoder layers, which are intialised from maximally spaced layers.
389
+ The model is then trained on a weighted sum of the KL divergence and pseudo-label loss terms.
390
+
391
+ <p align="center">
392
+ <img src="https://huggingface.co/datasets/distil-whisper/figures/resolve/main/architecture.png?raw=true" width="600"/>
393
+ </p>
394
+
395
+ ## Evaluation
396
+
397
+ The following code-snippets demonstrates how to evaluate the Distil-Whisper model on the LibriSpeech validation.clean
398
+ dataset with [streaming mode](https://huggingface.co/blog/audio-datasets#streaming-mode-the-silver-bullet), meaning no
399
+ audio data has to be downloaded to your local device.
400
+
401
+ First, we need to install the required packages, including 🤗 Datasets to stream and load the audio data, and 🤗 Evaluate to
402
+ perform the WER calculation:
403
+
404
+ ```bash
405
+ pip install --upgrade pip
406
+ pip install --upgrade transformers datasets[audio] evaluate jiwer
407
+ ```
408
+
409
+ Evaluation can then be run end-to-end with the following example:
410
+
411
+ ```python
412
+ from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
413
+ from transformers.models.whisper.english_normalizer import EnglishTextNormalizer
414
+ from datasets import load_dataset
415
+ from evaluate import load
416
+ import torch
417
+ from tqdm import tqdm
418
+
419
+ # define our torch configuration
420
+ device = "cuda:0" if torch.cuda.is_available() else "cpu"
421
+ torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
422
+
423
+ model_id = "distil-whisper/distil-small.en"
424
+
425
+ # load the model + processor
426
+ model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, use_safetensors=True, low_cpu_mem_usage=True)
427
+ model = model.to(device)
428
+ processor = AutoProcessor.from_pretrained(model_id)
429
+
430
+ # load the dataset with streaming mode
431
+ dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)
432
+
433
+ # define the evaluation metric
434
+ wer_metric = load("wer")
435
+ normalizer = EnglishTextNormalizer(processor.tokenizer.english_spelling_normalizer)
436
+
437
+ def inference(batch):
438
+ # 1. Pre-process the audio data to log-mel spectrogram inputs
439
+ audio = [sample["array"] for sample in batch["audio"]]
440
+ input_features = processor(audio, sampling_rate=batch["audio"][0]["sampling_rate"], return_tensors="pt").input_features
441
+ input_features = input_features.to(device, dtype=torch_dtype)
442
+
443
+ # 2. Auto-regressively generate the predicted token ids
444
+ pred_ids = model.generate(input_features, max_new_tokens=128)
445
+
446
+ # 3. Decode the token ids to the final transcription
447
+ batch["transcription"] = processor.batch_decode(pred_ids, skip_special_tokens=True)
448
+ batch["reference"] = batch["text"]
449
+ return batch
450
+
451
+ dataset = dataset.map(function=inference, batched=True, batch_size=16)
452
+
453
+ all_transcriptions = []
454
+ all_references = []
455
+
456
+ # iterate over the dataset and run inference
457
+ for i, result in tqdm(enumerate(dataset), desc="Evaluating..."):
458
+ all_transcriptions.append(result["transcription"])
459
+ all_references.append(result["reference"])
460
+
461
+ # normalize predictions and references
462
+ all_transcriptions = [normalizer(transcription) for transcription in all_transcriptions]
463
+ all_references = [normalizer(reference) for reference in all_references]
464
+
465
+ # compute the WER metric
466
+ wer = 100 * wer_metric.compute(predictions=all_transcriptions, references=all_references)
467
+ print(wer)
468
+
469
+ ```
470
+ **Print Output:**
471
+ ```
472
+ 3.4326070294536297
473
+ ```
474
+
475
+ ## Intended Use
476
+
477
+ Distil-Whisper is intended to be a drop-in replacement for Whisper on English speech recognition. In particular, it
478
+ achieves comparable WER results over out-of-distribution test data, while being 6x faster over both short and long-form
479
+ audio.
480
+
481
+ ## Data
482
+
483
+ Distil-Whisper is trained on 22,000 hours of audio data from 9 open-source, permissively licensed speech datasets on the
484
+ Hugging Face Hub:
485
+
486
+ | Dataset | Size / h | Speakers | Domain | Licence |
487
+ |-----------------------------------------------------------------------------------------|----------|----------|-----------------------------|-----------------|
488
+ | [People's Speech](https://huggingface.co/datasets/MLCommons/peoples_speech) | 12,000 | unknown | Internet Archive | CC-BY-SA-4.0 |
489
+ | [Common Voice 13](https://huggingface.co/datasets/mozilla-foundation/common_voice_13_0) | 3,000 | unknown | Narrated Wikipedia | CC0-1.0 |
490
+ | [GigaSpeech](https://huggingface.co/datasets/speechcolab/gigaspeech) | 2,500 | unknown | Audiobook, podcast, YouTube | apache-2.0 |
491
+ | Fisher | 1,960 | 11,900 | Telephone conversations | LDC |
492
+ | [LibriSpeech](https://huggingface.co/datasets/librispeech_asr) | 960 | 2,480 | Audiobooks | CC-BY-4.0 |
493
+ | [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) | 540 | 1,310 | European Parliament | CC0 |
494
+ | [TED-LIUM](https://huggingface.co/datasets/LIUM/tedlium) | 450 | 2,030 | TED talks | CC-BY-NC-ND 3.0 |
495
+ | SwitchBoard | 260 | 540 | Telephone conversations | LDC |
496
+ | [AMI](https://huggingface.co/datasets/edinburghcstr/ami) | 100 | unknown | Meetings | CC-BY-4.0 |
497
+ ||||||
498
+ | **Total** | 21,770 | 18,260+ | | |
499
+
500
+ The combined dataset spans 10 distinct domains and over 50k speakers. The diversity of this dataset is crucial to ensuring
501
+ the distilled model is robust to audio distributions and noise.
502
+
503
+ The audio data is then pseudo-labelled using the Whisper large-v2 model: we use Whisper to generate predictions for all
504
+ the audio in our training set and use these as the target labels during training. Using pseudo-labels ensures that the
505
+ transcriptions are consistently formatted across datasets and provides sequence-level distillation signal during training.
506
+
507
+ ## WER Filter
508
+
509
+ The Whisper pseudo-label predictions are subject to mis-transcriptions and hallucinations. To ensure we only train on
510
+ accurate pseudo-labels, we employ a simple WER heuristic during training. First, we normalise the Whisper pseudo-labels
511
+ and the ground truth labels provided by each dataset. We then compute the WER between these labels. If the WER exceeds
512
+ a specified threshold, we discard the training example. Otherwise, we keep it for training.
513
+
514
+ Section 9.2 of the [Distil-Whisper paper](https://arxiv.org/abs/2311.00430) demonstrates the effectiveness of this filter for improving downstream performance
515
+ of the distilled model. We also partially attribute Distil-Whisper's robustness to hallucinations to this filter.
516
+
517
+ ## Training
518
+
519
+ The model was trained for 50,000 optimisation steps (or 12 epochs) with batch size 2056. The Tensorboard training logs can
520
+ be found under: https://huggingface.co/distil-whisper/distil-small.en/tensorboard?params=scalars#frame
521
+
522
+ ## Results
523
+
524
+ The distilled model performs to within 1% WER of Whisper on out-of-distribution (OOD) short-form audio, and outperforms Whisper
525
+ by 0.1% on OOD long-form audio. This performance gain is attributed to lower hallucinations.
526
+
527
+ For a detailed per-dataset breakdown of the evaluation results, refer to Tables 16 and 17 of the [Distil-Whisper paper](https://arxiv.org/abs/2311.00430)
528
+
529
+ Distil-Whisper is also evaluated on the [ESB benchmark](https://arxiv.org/abs/2210.13352) datasets as part of the [OpenASR leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard),
530
+ where it performs to within 0.2% WER of Whisper.
531
+
532
+ ## Reproducing Distil-Whisper
533
+
534
+ Training and evaluation code to reproduce Distil-Whisper is available under the Distil-Whisper repository: https://github.com/huggingface/distil-whisper/tree/main/training
535
+
536
+ ## License
537
+
538
+ Distil-Whisper inherits the [MIT license](https://github.com/huggingface/distil-whisper/blob/main/LICENSE) from OpenAI's Whisper model.
539
+
540
+ ## Citation
541
+
542
+ If you use this model, please consider citing the [Distil-Whisper paper](https://arxiv.org/abs/2311.00430):
543
+ ```
544
+ @misc{gandhi2023distilwhisper,
545
+ title={Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling},
546
+ author={Sanchit Gandhi and Patrick von Platen and Alexander M. Rush},
547
+ year={2023},
548
+ eprint={2311.00430},
549
+ archivePrefix={arXiv},
550
+ primaryClass={cs.CL}
551
+ }
552
+ ```
553
+
554
+ ## Acknowledgements
555
+ * OpenAI for the Whisper [model](https://huggingface.co/openai/whisper-large-v2) and [original codebase](https://github.com/openai/whisper)
556
+ * Hugging Face 🤗 [Transformers](https://github.com/huggingface/transformers) for the model integration
557
+ * Google's [TPU Research Cloud (TRC)](https://sites.research.google/trc/about/) programme for Cloud TPU v4s
558
+ * [`@rsonavane`](https://huggingface.co/rsonavane/distil-whisper-large-v2-8-ls) for releasing an early iteration of Distil-Whisper on the LibriSpeech dataset