distil-small.en / README.md

Update README.md

281458e 11 months ago

24.6 kB

	---
	language:
	- en
	tags:
	- audio
	- automatic-speech-recognition
	- transformers.js
	widget:
	- example_title: LibriSpeech sample 1
	src: https://cdn-media.huggingface.co/speech_samples/sample1.flac
	- example_title: LibriSpeech sample 2
	src: https://cdn-media.huggingface.co/speech_samples/sample2.flac
	pipeline_tag: automatic-speech-recognition
	license: mit
	library_name: transformers
	---

	# Distil-Whisper: distil-small.en

	Distil-Whisper was proposed in the paper [Robust Knowledge Distillation via Large-Scale Pseudo Labelling](https://arxiv.org/abs/2311.00430).
	It is a distilled version of the Whisper model that is 6 times faster, 49% smaller, and performs within 1% WER
	on out-of-distribution evaluation sets.

	This is the repository for distil-small.en, a distilled variant of [Whisper small.en](https://huggingface.co/openai/whisper-small.en).
	It is the smallest Distil-Whisper checkpoint, with just 166M parameters, making it the ideal choice for memory
	constrained applications (e.g. on-device).

	For most other applications, the [distil-medium.en](https://huggingface.co/distil-whisper/distil-medium.en)
	or [distil-large-v2](https://huggingface.co/distil-whisper/distil-large-v2) checkpoints are recommended, since they are
	both faster and achieve better WER results:

	\| Model \| Params / M \| Rel. Latency \| Short-Form WER \| Long-Form WER \|
	\|----------------------------------------------------------------------------\|------------\|--------------\|----------------\|---------------\|
	\| [large-v2](https://huggingface.co/openai/whisper-large-v2) \| 1550 \| 1.0 \| 9.1 \| 11.7 \|
	\| \| \| \| \| \|
	\| [distil-large-v2](https://huggingface.co/distil-whisper/distil-large-v2) \| 756 \| 5.8 \| 10.1 \| 11.6 \|
	\| [distil-medium.en](https://huggingface.co/distil-whisper/distil-medium.en) \| 394 \| 6.8 \| 11.1 \| 12.4 \|
	\| [distil-small.en](https://huggingface.co/distil-whisper/distil-small.en) \| 166 \| 5.6 \| 12.1 \| 12.8 \|

	Note: Distil-Whisper is currently only available for English speech recognition. We are working with the community
	to distill Whisper on other languages. If you are interested in distilling Whisper in your language, check out the
	provided [training code](https://github.com/huggingface/distil-whisper/tree/main/training). We will update the
	[Distil-Whisper repository](https://github.com/huggingface/distil-whisper/) with multilingual checkpoints when ready!

	### Why is distil-small.en slower than distil-large-v2?

	While [distil-medium.en](https://huggingface.co/distil-whisper/distil-medium.en) and [distil-large-v2](https://huggingface.co/distil-whisper/distil-large-v2)
	use two decoder layers each, distil-small.en uses four. Using more decoder layers improves the WER performance of the
	model, at the expense of slower inference speed. We found that four layers was the minimum required to get reasonable
	WER performance for `distil-small.en`, where it performs to within 3% WER of Whisper [large-v2](https://huggingface.co/openai/whisper-large-v2)
	while being 5.6x faster. When we tried distilling with just two layers, the model was over 5% worse than large-v2, albeit
	7.8x faster. We leave distilling a two layer small.en model as future works.

	## Usage

	Distil-Whisper is supported in Hugging Face 🤗 Transformers from version 4.35 onwards. To run the model, first
	install the latest version of the Transformers library. For this example, we'll also install 🤗 Datasets to load toy
	audio dataset from the Hugging Face Hub:

	```bash
	pip install --upgrade pip
	pip install --upgrade transformers accelerate datasets[audio]
	```

	### Short-Form Transcription

	The model can be used with the [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline)
	class to transcribe short-form audio files (< 30-seconds) as follows:

	```python
	import torch
	from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
	from datasets import load_dataset


	device = "cuda:0" if torch.cuda.is_available() else "cpu"
	torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

	model_id = "distil-whisper/distil-small.en"

	model = AutoModelForSpeechSeq2Seq.from_pretrained(
	model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
	)
	model.to(device)

	processor = AutoProcessor.from_pretrained(model_id)

	pipe = pipeline(
	"automatic-speech-recognition",
	model=model,
	tokenizer=processor.tokenizer,
	feature_extractor=processor.feature_extractor,
	max_new_tokens=128,
	torch_dtype=torch_dtype,
	device=device,
	)

	dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
	sample = dataset[0]["audio"]

	result = pipe(sample)
	print(result["text"])
	```

	To transcribe a local audio file, simply pass the path to your audio file when you call the pipeline:
	```diff
	- result = pipe(sample)
	+ result = pipe("audio.mp3")
	```

	### Long-Form Transcription

	Distil-Whisper uses a chunked algorithm to transcribe long-form audio files (> 30-seconds). In practice, this chunked long-form algorithm
	is 9x faster than the sequential algorithm proposed by OpenAI in the Whisper paper (see Table 7 of the [Distil-Whisper paper](https://arxiv.org/abs/2311.00430)).

	To enable chunking, pass the `chunk_length_s` parameter to the `pipeline`. For Distil-Whisper, a chunk length of 15-seconds
	is optimal. To activate batching, pass the argument `batch_size`:

	```python
	import torch
	from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
	from datasets import load_dataset


	device = "cuda:0" if torch.cuda.is_available() else "cpu"
	torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

	model_id = "distil-whisper/distil-small.en"

	model = AutoModelForSpeechSeq2Seq.from_pretrained(
	model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
	)
	model.to(device)

	processor = AutoProcessor.from_pretrained(model_id)

	pipe = pipeline(
	"automatic-speech-recognition",
	model=model,
	tokenizer=processor.tokenizer,
	feature_extractor=processor.feature_extractor,
	max_new_tokens=128,
	chunk_length_s=15,
	batch_size=16,
	torch_dtype=torch_dtype,
	device=device,
	)

	dataset = load_dataset("distil-whisper/librispeech_long", "default", split="validation")
	sample = dataset[0]["audio"]

	result = pipe(sample)
	print(result["text"])
	```

	<!---
	Tip: The pipeline can also be used to transcribe an audio file from a remote URL, for example:

	```python
	result = pipe("https://huggingface.co/datasets/sanchit-gandhi/librispeech_long/resolve/main/audio.wav")
	```
	--->

	### Speculative Decoding

	Distil-Whisper can be used as an assistant model to Whisper for speculative decoding. Speculative decoding mathematically
	ensures the exact same outputs as Whisper are obtained while being 2 times faster. This makes it the perfect drop-in
	replacement for existing Whisper pipelines, since the same outputs are guaranteed.

	In the following code-snippet, we load the assistant Distil-Whisper model standalone to the main Whisper pipeline. We then
	specify it as the "assistant model" for generation:

	```python
	from transformers import pipeline, AutoModelForSpeechSeq2Seq, AutoProcessor
	import torch
	from datasets import load_dataset

	device = "cuda:0" if torch.cuda.is_available() else "cpu"
	torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

	assistant_model_id = "distil-whisper/distil-small.en"

	assistant_model = AutoModelForSpeechSeq2Seq.from_pretrained(
	assistant_model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
	)
	assistant_model.to(device)

	model_id = "openai/whisper-medium.en"

	model = AutoModelForSpeechSeq2Seq.from_pretrained(
	model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
	)
	model.to(device)

	processor = AutoProcessor.from_pretrained(model_id)

	pipe = pipeline(
	"automatic-speech-recognition",
	model=model,
	tokenizer=processor.tokenizer,
	feature_extractor=processor.feature_extractor,
	max_new_tokens=128,
	generate_kwargs={"assistant_model": assistant_model},
	torch_dtype=torch_dtype,
	device=device,
	)

	dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
	sample = dataset[0]["audio"]

	result = pipe(sample)
	print(result["text"])
	```

	## Additional Speed & Memory Improvements

	You can apply additional speed and memory improvements to Distil-Whisper which we cover in the following.

	### Flash Attention

	We recommend using [Flash-Attention 2](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#flashattention-2) if your GPU allows for it.
	To do so, you first need to install [Flash Attention](https://github.com/Dao-AILab/flash-attention):

	```
	pip install flash-attn --no-build-isolation
	```

	and then all you have to do is to pass `use_flash_attention_2=True` to `from_pretrained`:

	```diff
	- model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True)
	+ model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True, use_flash_attention_2=True)
	```

	### Torch Scale-Product-Attention (SDPA)

	If your GPU does not support Flash Attention, we recommend making use of [BetterTransformers](https://huggingface.co/docs/transformers/main/en/perf_infer_gpu_one#bettertransformer).
	To do so, you first need to install optimum:

	```
	pip install --upgrade optimum
	```

	And then convert your model to a "BetterTransformer" model before using it:

	```diff
	model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True)
	+ model = model.to_bettertransformer()
	```

	### Running Distil-Whisper in `openai-whisper`

	To use the model in the original Whisper format, first ensure you have the [`openai-whisper`](https://pypi.org/project/openai-whisper/) package installed:

	```bash
	pip install --upgrade openai-whisper
	```

	The following code-snippet demonstrates how to transcribe a sample file from the LibriSpeech dataset loaded using
	🤗 Datasets:

	```python
	import torch
	from datasets import load_dataset
	from huggingface_hub import hf_hub_download
	from whisper import load_model, transcribe

	distil_small_en = hf_hub_download(repo_id="distil-whisper/distil-small.en", filename="original-model.bin")
	model = load_model(distil_small_en)

	dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
	sample = dataset[0]["audio"]["array"]
	sample = torch.from_numpy(sample).float()

	pred_out = transcribe(model, audio=sample)
	print(pred_out["text"])
	```

	Note that the model weights will be downloaded and saved to your cache the first time you run the example. Subsequently,
	you can re-use the same example, and the weights will be loaded directly from your cache without having to download them
	again.

	To transcribe a local audio file, simply pass the path to the audio file as the `audio` argument to transcribe:

	```python
	pred_out = transcribe(model, audio="audio.mp3")
	```

	### Whisper.cpp

	Distil-Whisper can be run from the [Whisper.cpp](https://github.com/ggerganov/whisper.cpp) repository with the original
	sequential long-form transcription algorithm. In a [provisional benchmark](https://github.com/ggerganov/whisper.cpp/pull/1424#issuecomment-1793513399)
	on Mac M1, `distil-small.en` is over 4x faster than `large-v2`, while performing to within 1.4% WER over long-form audio.

	Steps for getting started:
	1. Clone the Whisper.cpp repository:
	```
	git clone https://github.com/ggerganov/whisper.cpp.git
	cd whisper.cpp
	```
	2. Download the ggml weights for `distil-small.en` from the Hugging Face Hub:

	```bash
	python -c "from huggingface_hub import hf_hub_download; hf_hub_download(repo_id='distil-whisper/distil-small.en', filename='ggml-distil-small.en.bin', local_dir='./models')"
	```

	Note that if you do not have the `huggingface_hub` package installed, you can also download the weights with `wget`:

	```bash
	wget https://huggingface.co/distil-whisper/distil-small.en/resolve/main/ggml-distil-small.en.bin -P ./models
	```

	3. Run inference using the provided sample audio:

	```bash
	make -j && ./main -m models/ggml-distil-small.en.bin -f samples/jfk.wav
	```

	### Transformers.js

	```js
	import { pipeline } from '@xenova/transformers';

	let transcriber = await pipeline('automatic-speech-recognition', 'distil-whisper/distil-small.en');

	let url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/jfk.wav';
	let output = await transcriber(url);
	// { text: " And so my fellow Americans, ask not what your country can do for you. Ask what you can do for your country." }
	```

	See the [docs](https://huggingface.co/docs/transformers.js/api/pipelines#module_pipelines.AutomaticSpeechRecognitionPipeline) for more information.

	### Candle

	Coming soon!

	<!---

	Through an integration with Hugging Face [Candle](https://github.com/huggingface/candle/tree/main) 🕯️, Distil-Whisper is
	now available in the Rust library 🦀

	Benefit from:
	* Optimised CPU backend with optional MKL support for x86 and Accelerate for Macs
	* CUDA backend for efficiently running on GPUs, multiple GPU distribution via NCCL
	* WASM support: run Distil-Whisper in a browser

	Steps for getting started:
	1. Install [`candle-core`](https://github.com/huggingface/candle/tree/main/candle-core) as explained [here](https://huggingface.github.io/candle/guide/installation.html)
	2. Clone the `candle` repository locally:
	```
	git clone https://github.com/huggingface/candle.git
	```
	3. Enter the example directory for [Whisper](https://github.com/huggingface/candle/tree/main/candle-examples/examples/whisper):
	```
	cd candle/candle-examples/examples/whisper
	```
	4. Run an example:
	```
	cargo run --example whisper --release -- --model distil-small.en
	```
	5. To specify your own audio file, add the `--input` flag:
	```
	cargo run --example whisper --release -- --model distil-small.en --input audio.wav
	```

	--->

	### 8bit & 4bit Quantization

	Coming soon!

	## Model Details

	Distil-Whisper inherits the encoder-decoder architecture from Whisper. The encoder maps a sequence of speech vector
	inputs to a sequence of hidden-state vectors. The decoder auto-regressively predicts text tokens, conditional on all
	previous tokens and the encoder hidden-states. Consequently, the encoder is only run forward once, whereas the decoder
	is run as many times as the number of tokens generated. In practice, this means the decoder accounts for over 90% of
	total inference time. Thus, to optimise for latency, the focus is on minimising the inference time of the decoder.

	To distill the Whisper model, we reduce the number of decoder layers while keeping the encoder fixed.
	The encoder (shown in green) is entirely copied from the teacher to the student and frozen during training.
	The student's decoder consists of a subset of the teacher decoder layers, which are intialised from maximally spaced layers.
	The model is then trained on a weighted sum of the KL divergence and pseudo-label loss terms.

	<p align="center">
	<img src="https://huggingface.co/datasets/distil-whisper/figures/resolve/main/architecture.png?raw=true" width="600"/>
	</p>

	## Evaluation

	The following code-snippets demonstrates how to evaluate the Distil-Whisper model on the LibriSpeech validation.clean
	dataset with [streaming mode](https://huggingface.co/blog/audio-datasets#streaming-mode-the-silver-bullet), meaning no
	audio data has to be downloaded to your local device.

	First, we need to install the required packages, including 🤗 Datasets to stream and load the audio data, and 🤗 Evaluate to
	perform the WER calculation:

	```bash
	pip install --upgrade pip
	pip install --upgrade transformers datasets[audio] evaluate jiwer
	```

	Evaluation can then be run end-to-end with the following example:

	```python
	from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
	from transformers.models.whisper.english_normalizer import EnglishTextNormalizer
	from datasets import load_dataset
	from evaluate import load
	import torch
	from tqdm import tqdm

	# define our torch configuration
	device = "cuda:0" if torch.cuda.is_available() else "cpu"
	torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

	model_id = "distil-whisper/distil-small.en"

	# load the model + processor
	model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, use_safetensors=True, low_cpu_mem_usage=True)
	model = model.to(device)
	processor = AutoProcessor.from_pretrained(model_id)

	# load the dataset with streaming mode
	dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)

	# define the evaluation metric
	wer_metric = load("wer")
	normalizer = EnglishTextNormalizer(processor.tokenizer.english_spelling_normalizer)

	def inference(batch):
	# 1. Pre-process the audio data to log-mel spectrogram inputs
	audio = [sample["array"] for sample in batch["audio"]]
	input_features = processor(audio, sampling_rate=batch["audio"][0]["sampling_rate"], return_tensors="pt").input_features
	input_features = input_features.to(device, dtype=torch_dtype)

	# 2. Auto-regressively generate the predicted token ids
	pred_ids = model.generate(input_features, max_new_tokens=128)

	# 3. Decode the token ids to the final transcription
	batch["transcription"] = processor.batch_decode(pred_ids, skip_special_tokens=True)
	batch["reference"] = batch["text"]
	return batch

	dataset = dataset.map(function=inference, batched=True, batch_size=16)

	all_transcriptions = []
	all_references = []

	# iterate over the dataset and run inference
	for i, result in tqdm(enumerate(dataset), desc="Evaluating..."):
	all_transcriptions.append(result["transcription"])
	all_references.append(result["reference"])

	# normalize predictions and references
	all_transcriptions = [normalizer(transcription) for transcription in all_transcriptions]
	all_references = [normalizer(reference) for reference in all_references]

	# compute the WER metric
	wer = 100 * wer_metric.compute(predictions=all_transcriptions, references=all_references)
	print(wer)

	```
	Print Output:
	```
	3.4326070294536297
	```

	## Intended Use

	Distil-Whisper is intended to be a drop-in replacement for Whisper on English speech recognition. In particular, it
	achieves comparable WER results over out-of-distribution test data, while being 6x faster over both short and long-form
	audio.

	## Data

	Distil-Whisper is trained on 22,000 hours of audio data from 9 open-source, permissively licensed speech datasets on the
	Hugging Face Hub:

	\| Dataset \| Size / h \| Speakers \| Domain \| Licence \|
	\|-----------------------------------------------------------------------------------------\|----------\|----------\|-----------------------------\|-----------------\|
	\| [People's Speech](https://huggingface.co/datasets/MLCommons/peoples_speech) \| 12,000 \| unknown \| Internet Archive \| CC-BY-SA-4.0 \|
	\| [Common Voice 13](https://huggingface.co/datasets/mozilla-foundation/common_voice_13_0) \| 3,000 \| unknown \| Narrated Wikipedia \| CC0-1.0 \|
	\| [GigaSpeech](https://huggingface.co/datasets/speechcolab/gigaspeech) \| 2,500 \| unknown \| Audiobook, podcast, YouTube \| apache-2.0 \|
	\| Fisher \| 1,960 \| 11,900 \| Telephone conversations \| LDC \|
	\| [LibriSpeech](https://huggingface.co/datasets/librispeech_asr) \| 960 \| 2,480 \| Audiobooks \| CC-BY-4.0 \|
	\| [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli) \| 540 \| 1,310 \| European Parliament \| CC0 \|
	\| [TED-LIUM](https://huggingface.co/datasets/LIUM/tedlium) \| 450 \| 2,030 \| TED talks \| CC-BY-NC-ND 3.0 \|
	\| SwitchBoard \| 260 \| 540 \| Telephone conversations \| LDC \|
	\| [AMI](https://huggingface.co/datasets/edinburghcstr/ami) \| 100 \| unknown \| Meetings \| CC-BY-4.0 \|
	\|\|\|\|\|\|
	\| Total \| 21,770 \| 18,260+ \| \| \|

	The combined dataset spans 10 distinct domains and over 50k speakers. The diversity of this dataset is crucial to ensuring
	the distilled model is robust to audio distributions and noise.

	The audio data is then pseudo-labelled using the Whisper large-v2 model: we use Whisper to generate predictions for all
	the audio in our training set and use these as the target labels during training. Using pseudo-labels ensures that the
	transcriptions are consistently formatted across datasets and provides sequence-level distillation signal during training.

	## WER Filter

	The Whisper pseudo-label predictions are subject to mis-transcriptions and hallucinations. To ensure we only train on
	accurate pseudo-labels, we employ a simple WER heuristic during training. First, we normalise the Whisper pseudo-labels
	and the ground truth labels provided by each dataset. We then compute the WER between these labels. If the WER exceeds
	a specified threshold, we discard the training example. Otherwise, we keep it for training.

	Section 9.2 of the [Distil-Whisper paper](https://arxiv.org/abs/2311.00430) demonstrates the effectiveness of this filter for improving downstream performance
	of the distilled model. We also partially attribute Distil-Whisper's robustness to hallucinations to this filter.

	## Training

	The model was trained for 50,000 optimisation steps (or 12 epochs) with batch size 2056. The Tensorboard training logs can
	be found under: https://huggingface.co/distil-whisper/distil-small.en/tensorboard?params=scalars#frame

	## Results

	The distilled model performs to within 1% WER of Whisper on out-of-distribution (OOD) short-form audio, and outperforms Whisper
	by 0.1% on OOD long-form audio. This performance gain is attributed to lower hallucinations.

	For a detailed per-dataset breakdown of the evaluation results, refer to Tables 16 and 17 of the [Distil-Whisper paper](https://arxiv.org/abs/2311.00430)

	Distil-Whisper is also evaluated on the [ESB benchmark](https://arxiv.org/abs/2210.13352) datasets as part of the [OpenASR leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard),
	where it performs to within 0.2% WER of Whisper.

	## Reproducing Distil-Whisper

	Training and evaluation code to reproduce Distil-Whisper is available under the Distil-Whisper repository: https://github.com/huggingface/distil-whisper/tree/main/training

	## License

	Distil-Whisper inherits the [MIT license](https://github.com/huggingface/distil-whisper/blob/main/LICENSE) from OpenAI's Whisper model.

	## Citation

	If you use this model, please consider citing the [Distil-Whisper paper](https://arxiv.org/abs/2311.00430):
	```
	@misc{gandhi2023distilwhisper,
	title={Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling},
	author={Sanchit Gandhi and Patrick von Platen and Alexander M. Rush},
	year={2023},
	eprint={2311.00430},
	archivePrefix={arXiv},
	primaryClass={cs.CL}
	}
	```

	## Acknowledgements
	* OpenAI for the Whisper [model](https://huggingface.co/openai/whisper-large-v2) and [original codebase](https://github.com/openai/whisper)
	* Hugging Face 🤗 [Transformers](https://github.com/huggingface/transformers) for the model integration
	* Google's [TPU Research Cloud (TRC)](https://sites.research.google/trc/about/) programme for Cloud TPU v4s
	* [`@rsonavane`](https://huggingface.co/rsonavane/distil-whisper-large-v2-8-ls) for releasing an early iteration of Distil-Whisper on the LibriSpeech dataset