sanchit-gandhi HF staff commited on
Commit
38b4c8a
1 Parent(s): a179016
Files changed (1) hide show
  1. README.md +155 -90
README.md CHANGED
@@ -157,172 +157,235 @@ pipeline_tag: automatic-speech-recognition
157
  license: apache-2.0
158
  ---
159
 
160
- # Whisper
161
 
162
- [OpenAI's Whisper](https://openai.com/blog/whisper/)
 
 
163
 
164
- The Whisper model was proposed in [Robust Speech Recognition via Large-Scale Weak Supervision](https://cdn.openai.com/papers/whisper.pdf) by Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever.
 
165
 
166
- **Disclaimer**: Content from **this** model card has been written by the Hugging Face team, and parts of it were copy pasted from the original model card.
 
167
 
 
168
 
169
- ## Intro
 
170
 
171
- The first paragraphs of the abstract read as follows :
 
 
 
172
 
173
- > We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zeroshot transfer setting without the need for any finetuning.
174
- > When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.
 
 
 
175
 
176
- The original code repository can be found [here](https://github.com/openai/whisper).
 
 
 
 
 
 
 
177
 
178
- ## Model details
179
-
180
- The Whisper models are trained for speech recognition and translation tasks, capable of transcribing speech audio into the text in the language it is spoken (ASR) as well as translated into English (speech translation). Researchers at OpenAI developed the models to study the robustness of speech processing systems trained under large-scale weak supervision. There are 9 models of different sizes and capabilities, summarised in the following table.
181
-
182
- | Size | Parameters | English-only model | Multilingual model |
183
- |:------:|:----------:|:------------------:|:------------------:|
184
- | tiny | 39 M | ✓ | ✓ |
185
- | base | 74 M | ✓ | ✓ |
186
- | small | 244 M | ✓ | ✓ |
187
- | medium | 769 M | ✓ | ✓ |
188
- | large | 1550 M | | ✓ |
189
-
190
-
191
-
192
- ## Model description
193
 
194
- Whisper is an auto-regressive automatic speech recognition encoder-decoder model that was trained on 680 000 hours of 16kHz sampled multilingual audio. It was fully trained in a supervised manner, with multiple tasks :
195
 
196
- - English transcription
197
- - Any-to-English speech translation
198
- - Non-English transcription
199
- - No speech prediction
200
 
201
- To each task corresponds a sequence of tokens that are given to the decoder as *context tokens*. The beginning of a transcription always starts with `<|startoftranscript|>` which is why the `decoder_start_token` is always set to `tokenizer.encode("<|startoftranscript|>")`. The following token should be the language token, which is automatically detected in the original code. Finally, the task is define using either `<|transcribe|>` or `<|translate|>`. In addition, a `<|notimestamps|>` token is added if the task does not include timestamp prediction.
 
 
 
 
 
202
 
 
 
 
 
 
203
 
204
- # Usage
 
 
205
 
206
- To transcribe or translate audio files, the model has to be used along a `WhisperProcessor`. The `WhisperProcessor.get_decoder_prompt_ids` function is used to get a list of `( idx, token )` tuples, which can either be set in the config, or directly passed to the generate function, as `forced_decoder_ids`.
207
 
 
 
 
208
 
209
- ## Transcription
210
- In the following example, the english only model is used. We set the `decoder_input_ids` accordingly.
211
 
 
212
 
213
- ### English to english
214
- The "<|en|>" token is used to specify that the speech is in english and should be transcribed to english
 
215
 
216
  ```python
217
  >>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
218
  >>> from datasets import load_dataset
219
- >>> import torch
220
 
221
  >>> # load model and processor
222
  >>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")
223
  >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")
 
224
 
225
- >>> # load dummy dataset and read soundfiles
226
  >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
227
- >>> input_features = processor(ds[0]["audio"]["array"], return_tensors="pt").input_features
 
228
 
229
- >>> # Generate logits
230
- >>> logits = model(input_features, decoder_input_ids = torch.tensor([[50258]])).logits
231
- >>> # take argmax and decode
232
- >>> predicted_ids = torch.argmax(logits, dim=-1)
233
- >>> transcription = processor.batch_decode(predicted_ids)
234
- ['<|startoftranscript|>']
 
 
235
  ```
 
236
 
237
  ### French to French
238
- In order to obtain the full transcription, the `generate()` function is used. The following example demonstrates a french to french
239
- transcription.
240
 
241
  ```python
242
  >>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
243
- >>> from datasets import load_dataset
244
- >>> import torch
245
 
246
  >>> # load model and processor
247
  >>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")
248
  >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")
 
249
 
250
- >>> # load dummy dataset and read soundfiles
251
  >>> ds = load_dataset("common_voice", "fr", split="test", streaming=True)
252
- >>> ds = ds.cast_column("audio", datasets.Audio(sampling_rate=16_000))
253
- >>> input_speech = next(iter(ds))["audio"]["array"]
254
- >>> model.config.forced_decoder_ids = processor.get_decoder_prompt_ids(language = "fr", task = "transcribe")
255
- >>> input_features = processor(input_speech, return_tensors="pt").input_features
256
- >>> predicted_ids = model.generate(input_features)
 
 
257
  >>> transcription = processor.batch_decode(predicted_ids)
258
  ['<|startoftranscript|><|fr|><|transcribe|><|notimestamps|> Un vrai travail intéressant va enfin être mené sur ce sujet.<|endoftext|>']
259
 
260
- >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens = True)
261
  [' Un vrai travail intéressant va enfin être mené sur ce sujet.']
262
  ```
263
 
264
  ## Translation
265
- The "<|translate|>" is used as the first decoder input token to specify the transcription task.
266
 
267
  ### French to English
268
 
269
  ```python
270
  >>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
271
- >>> from datasets import load_dataset
272
- >>> import torch
273
 
274
  >>> # load model and processor
275
  >>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")
276
  >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")
 
277
 
278
- >>> # load dummy dataset and read soundfiles
279
  >>> ds = load_dataset("common_voice", "fr", split="test", streaming=True)
280
- >>> ds = ds.cast_column("audio", datasets.Audio(sampling_rate=16_000))
281
- >>> input_speech = next(iter(ds))["audio"]["array"]
282
- >>> # tokenize
283
- >>> input_features = processor(input_speech, return_tensors="pt").input_features
284
- >>> forced_decoder_ids = processor.get_decoder_prompt_ids(language = "fr", task = "translate")
285
-
286
- >>> predicted_ids = model.generate(input_features, forced_decoder_ids = forced_decoder_ids)
287
- >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens = True)
288
  [' A very interesting work, we will finally be given on this subject.']
289
  ```
290
 
291
  ## Evaluation
292
 
293
- This code snippet shows how to evaluate **openai/whisper-tiny** on LibriSpeech's "clean" and "other" test data.
294
 
295
  ```python
296
  >>> from datasets import load_dataset
297
  >>> from transformers import WhisperForConditionalGeneration, WhisperProcessor
298
- >>> import soundfile as sf
299
  >>> import torch
300
- >>> from jiwer import wer
301
-
302
 
303
- >>> librispeech_eval = load_dataset("librispeech_asr", "clean", split="test")
304
 
305
- >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny").to("cuda")
306
  >>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")
 
307
 
308
  >>> def map_to_pred(batch):
309
- >>> input_features = processor(batch["audio"]["array"], return_tensors="pt").input_features
310
-
 
 
311
  >>> with torch.no_grad():
312
- >>> logits = model(input_features.to("cuda")).logits
313
-
314
- >>> predicted_ids = torch.argmax(logits, dim=-1)
315
- >>> transcription = processor.batch_decode(predicted_ids, normalize = True)
316
- >>> batch['text'] = processor.tokenizer._normalize(batch['text'])
317
- >>> batch["transcription"] = transcription
318
  >>> return batch
319
 
320
- >>> result = librispeech_eval.map(map_to_pred, batched=True, batch_size=1, remove_columns=["speech"])
 
 
 
 
 
 
 
 
 
 
 
 
 
321
 
322
- >>> print("WER:", wer(result["text"], result["transcription"]))
323
- 0.07547098647858638
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
324
  ```
325
 
 
 
 
 
 
 
326
 
327
  ### Evaluated Use
328
 
@@ -359,12 +422,14 @@ There are also potential dual use concerns that come with releasing Whisper. Whi
359
 
360
 
361
  ### BibTeX entry and citation info
362
- *Since no official citation was provided, we use the following in the mean time*
363
  ```bibtex
364
  @misc{radford2022whisper,
365
- title={Robust Speech Recognition via Large-Scale Weak Supervision.},
366
- author={Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever},
367
- year={2022},
368
- url={https://cdn.openai.com/papers/whisper.pdf},
 
 
 
369
  }
370
  ```
 
157
  license: apache-2.0
158
  ---
159
 
160
+ # Whisper
161
 
162
+ Whisper is a pre-trained model for automatic speech recognition (ASR) and speech translation. Trained on 680k hours
163
+ of labelled data, Whisper models demonstrate a strong ability to generalise to many datasets and domains **without** the need
164
+ for fine-tuning.
165
 
166
+ Whisper was proposed in the paper [Robust Speech Recognition via Large-Scale Weak Supervision](https://arxiv.org/abs/2212.04356)
167
+ by Alec Radford et al from OpenAI. The original code repository can be found [here](https://github.com/openai/whisper).
168
 
169
+ **Disclaimer**: Content for this model card has partly been written by the Hugging Face team, and parts of it were
170
+ copied and pasted from the original model card.
171
 
172
+ ## Model details
173
 
174
+ Whisper is a Transformer based encoder-decoder model, also referred to as a _sequence-to-sequence_ model.
175
+ It was trained on 680k hours of labelled speech data annotated using large-scale weak supervision.
176
 
177
+ The models were trained on either English-only data or multilingual data. The English-only models were trained
178
+ on the task of speech recognition. The multilingual models were trained on both speech recognition and speech
179
+ translation. For speech recognition, the model predicts transcriptions in the *same* language as the audio.
180
+ For speech translation, the model predicts transcriptions to a *different* language to the audio.
181
 
182
+ Whisper checkpoints come in five configurations of varying model sizes.
183
+ The smallest four are trained on either English-only or multilingual data.
184
+ The largest checkpoints are multilingual only. All ten of the pre-trained checkpoints
185
+ are available on the [Hugging Face Hub](https://huggingface.co/models?search=openai/whisper). The
186
+ checkpoints are summarised in the following table with links to the models on the Hub:
187
 
188
+ | Size | Parameters | English-only | Multilingual |
189
+ |----------|------------|------------------------------------------------------|-----------------------------------------------------|
190
+ | tiny | 39 M | [✓](https://huggingface.co/openai/whisper-tiny.en) | [✓](https://huggingface.co/openai/whisper-tiny) |
191
+ | base | 74 M | [✓](https://huggingface.co/openai/whisper-base.en) | [✓](https://huggingface.co/openai/whisper-base) |
192
+ | small | 244 M | [✓](https://huggingface.co/openai/whisper-small.en) | [✓](https://huggingface.co/openai/whisper-small) |
193
+ | medium | 769 M | [✓](https://huggingface.co/openai/whisper-medium.en) | [✓](https://huggingface.co/openai/whisper-medium) |
194
+ | large | 1550 M | x | [✓](https://huggingface.co/openai/whisper-large) |
195
+ | large-v2 | 1550 M | x | [✓](https://huggingface.co/openai/whisper-large-v2) |
196
 
197
+ # Usage
 
 
 
 
 
 
 
 
 
 
 
 
 
 
198
 
199
+ To transcribe audio samples, the model has to be used alongside a [`WhisperProcessor`](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperProcessor).
200
 
201
+ The `WhisperProcessor` is used to:
202
+ 1. Pre-process the audio inputs (converting them to log-Mel spectrograms for the model)
203
+ 2. Post-process the model outputs (converting them from tokens to text)
 
204
 
205
+ The model is informed of which task to perform (transcription or translation) by passing the appropriate "context tokens". These context tokens
206
+ are a sequence of tokens that are given to the decoder at the start of the decoding process, and take the following order:
207
+ 1. The transcription always starts with the `<|startoftranscript|>` token
208
+ 2. The second token is the language token (e.g. `<|en|>` for English)
209
+ 3. The third token is the "task token". It can take one of two values: `<|transcribe|>` for speech recognition or `<|translate|>` for speech translation
210
+ 4. In addition, a `<|notimestamps|>` token is added if the model should not include timestamp prediction
211
 
212
+ Thus, a typical sequence of context tokens might look as follows:
213
+ ```
214
+ <|startoftranscript|> <|en|> <|transcribe|> <|notimestamps|>
215
+ ```
216
+ Which tells the model to decode in English, under the task of speech recognition, and not to predict timestamps.
217
 
218
+ These tokens can either be forced or un-forced. If they are forced, the model is made to predict each token at
219
+ each position. This allows one to control the output language and task for the Whisper model. If they are un-forced,
220
+ the Whisper model will automatically predict the output langauge and task itself.
221
 
222
+ The context tokens can be set accordingly:
223
 
224
+ ```python
225
+ model.config.forced_decoder_ids = WhisperProcessor.get_decoder_prompt_ids(language="english", task="transcribe")
226
+ ```
227
 
228
+ Which forces the model to predict in English under the task of speech recognition.
 
229
 
230
+ ## Transcription
231
 
232
+ ### English to English
233
+ In this example, the context tokens are 'unforced', meaning the model automatically predicts the output language
234
+ (English) and task (transcribe).
235
 
236
  ```python
237
  >>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
238
  >>> from datasets import load_dataset
 
239
 
240
  >>> # load model and processor
241
  >>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")
242
  >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")
243
+ >>> model.config.forced_decoder_ids = None
244
 
245
+ >>> # load dummy dataset and read audio files
246
  >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
247
+ >>> sample = ds[0]["audio"]
248
+ >>> input_features = processor(sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt").input_features
249
 
250
+ >>> # generate token ids
251
+ >>> predicted_ids = model.generate(input_features)
252
+ >>> # decode token ids to text
253
+ >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=False)
254
+ ['<|startoftranscript|><|en|><|transcribe|><|notimestamps|> Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.<|endoftext|>']
255
+
256
+ >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
257
+ [' Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.']
258
  ```
259
+ The context tokens can be removed from the start of the transcription by setting `skip_special_tokens=True`.
260
 
261
  ### French to French
262
+ The following example demonstrates French to French transcription by setting the decoder ids appropriately.
 
263
 
264
  ```python
265
  >>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
266
+ >>> from datasets import Audio, load_dataset
 
267
 
268
  >>> # load model and processor
269
  >>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")
270
  >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")
271
+ >>> forced_decoder_ids = processor.get_decoder_prompt_ids(language="french", task="transcribe")
272
 
273
+ >>> # load streaming dataset and read first audio sample
274
  >>> ds = load_dataset("common_voice", "fr", split="test", streaming=True)
275
+ >>> ds = ds.cast_column("audio", Audio(sampling_rate=16_000))
276
+ >>> input_speech = next(iter(ds))["audio"]
277
+ >>> input_features = processor(input_speech["array"], sampling_rate=input_speech["sampling_rate"], return_tensors="pt").input_features
278
+
279
+ >>> # generate token ids
280
+ >>> predicted_ids = model.generate(input_features, forced_decoder_ids=forced_decoder_ids)
281
+ >>> # decode token ids to text
282
  >>> transcription = processor.batch_decode(predicted_ids)
283
  ['<|startoftranscript|><|fr|><|transcribe|><|notimestamps|> Un vrai travail intéressant va enfin être mené sur ce sujet.<|endoftext|>']
284
 
285
+ >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
286
  [' Un vrai travail intéressant va enfin être mené sur ce sujet.']
287
  ```
288
 
289
  ## Translation
290
+ Setting the task to "translate" forces the Whisper model to perform speech translation.
291
 
292
  ### French to English
293
 
294
  ```python
295
  >>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
296
+ >>> from datasets import Audio, load_dataset
 
297
 
298
  >>> # load model and processor
299
  >>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")
300
  >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")
301
+ >>> forced_decoder_ids = processor.get_decoder_prompt_ids(language="french", task="translate")
302
 
303
+ >>> # load streaming dataset and read first audio sample
304
  >>> ds = load_dataset("common_voice", "fr", split="test", streaming=True)
305
+ >>> ds = ds.cast_column("audio", Audio(sampling_rate=16_000))
306
+ >>> input_speech = next(iter(ds))["audio"]
307
+ >>> input_features = processor(input_speech["array"], sampling_rate=input_speech["sampling_rate"], return_tensors="pt").input_features
308
+
309
+ >>> # generate token ids
310
+ >>> predicted_ids = model.generate(input_features, forced_decoder_ids=forced_decoder_ids)
311
+ >>> # decode token ids to text
312
+ >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
313
  [' A very interesting work, we will finally be given on this subject.']
314
  ```
315
 
316
  ## Evaluation
317
 
318
+ This code snippet shows how to evaluate Whisper Tiny on [LibriSpeech test-clean](https://huggingface.co/datasets/librispeech_asr):
319
 
320
  ```python
321
  >>> from datasets import load_dataset
322
  >>> from transformers import WhisperForConditionalGeneration, WhisperProcessor
 
323
  >>> import torch
324
+ >>> from evaluate import load
 
325
 
326
+ >>> librispeech_test_clean = load_dataset("librispeech_asr", "clean", split="test")
327
 
 
328
  >>> processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")
329
+ >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny").to("cuda")
330
 
331
  >>> def map_to_pred(batch):
332
+ >>> audio = batch["audio"]
333
+ >>> input_features = processor(audio["array"], sampling_rate=audio["sampling_rate"], return_tensors="pt").input_features
334
+ >>> batch["reference"] = processor.tokenizer._normalize(batch['text'])
335
+ >>>
336
  >>> with torch.no_grad():
337
+ >>> predicted_ids = model.generate(input_features.to("cuda"))[0]
338
+ >>> transcription = processor.decode(predicted_ids)
339
+ >>> batch["prediction"] = processor.tokenizer._normalize(transcription)
 
 
 
340
  >>> return batch
341
 
342
+ >>> result = librispeech_test_clean.map(map_to_pred)
343
+
344
+ >>> wer = load("wer")
345
+ >>> print(100 * wer.compute(references=result["reference"], predictions=result["prediction"]))
346
+ 7.547098647858638
347
+ ```
348
+
349
+ ## Long-Form Transcription
350
+
351
+ The Whisper model is intrinsically designed to work on audio samples of up to 30s in duration. However, by using a chunking
352
+ algorithm, it can be used to transcribe audio samples of up to arbitrary length. This is possible through Transformers
353
+ [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline)
354
+ method. Chunking is enabled by setting `chunk_length_s=30` when instantiating the pipeline. It can also be extended to
355
+ predict utterance level timestamps by passing `return_timestamps=True`:
356
 
357
+ ```python
358
+ >>> import torch
359
+ >>> from transformers import pipeline
360
+ >>> from datasets import load_dataset
361
+
362
+ >>> device = "cuda:0" if torch.cuda.is_available() else "cpu"
363
+
364
+ >>> pipe = pipeline(
365
+ >>> "automatic-speech-recognition",
366
+ >>> model="openai/whisper-tiny",
367
+ >>> chunk_length_s=30,
368
+ >>> device=device,
369
+ >>> )
370
+
371
+ >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
372
+ >>> sample = ds[0]["audio"]
373
+
374
+ >>> prediction = pipe(sample)["text"]
375
+ " Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel."
376
+
377
+ >>> # we can also return timestamps for the predictions
378
+ >>> prediction = pipe(sample, return_timestamps=True)["chunks"]
379
+ [{'text': ' Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.',
380
+ 'timestamp': (0.0, 5.44)}]
381
  ```
382
 
383
+ ## Fine-Tuning
384
+
385
+ The pre-trained Whisper model demonstrates a strong ability to generalise to different datasets and domains. However,
386
+ its predictive capabilities can be improved further for certain languages and tasks through *fine-tuning*. The blog
387
+ post [Fine-Tune Whisper with 🤗 Transformers](https://huggingface.co/blog/fine-tune-whisper) provides a step-by-step
388
+ guide to fine-tuning the Whisper model with as little as 5 hours of labelled data.
389
 
390
  ### Evaluated Use
391
 
 
422
 
423
 
424
  ### BibTeX entry and citation info
 
425
  ```bibtex
426
  @misc{radford2022whisper,
427
+ doi = {10.48550/ARXIV.2212.04356},
428
+ url = {https://arxiv.org/abs/2212.04356},
429
+ author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
430
+ title = {Robust Speech Recognition via Large-Scale Weak Supervision},
431
+ publisher = {arXiv},
432
+ year = {2022},
433
+ copyright = {arXiv.org perpetual, non-exclusive license}
434
  }
435
  ```