--- language: is datasets: - malromur tags: - audio - automatic-speech-recognition - speech - xlsr-fine-tuning-week license: apache-2.0 widget: - label: Malromur sample 1608 src: https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-icelandic/resolve/main/sample1608.flac - label: Malromur sample 3860 src: https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-icelandic/resolve/main/sample3860.flac model-index: - name: XLSR Wav2Vec2 Icelandic by Mehrdad Farahani results: - task: name: Speech Recognition type: automatic-speech-recognition dataset: name: Malromur is type: malromur args: lt metrics: - name: Test WER type: wer value: 10.74 --- # Wav2Vec2-Large-XLSR-53-Icelandic Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) in Icelandic using [Malromur](https://clarin.is/en/resources/malromur/). When using this model, make sure that your speech input is sampled at 16kHz. ## Usage The model can be used directly (without a language model) as follows: **Requirements** ```bash # requirement packages !pip install git+https://github.com/huggingface/datasets.git !pip install git+https://github.com/huggingface/transformers.git !pip install torchaudio !pip install librosa !pip install jiwer !pip install num2words ``` **Normalizer** ```bash # num2word packages # Original source: https://github.com/savoirfairelinux/num2words !mkdir -p ./num2words !wget -O num2words/__init__.py https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-icelandic/raw/main/num2words/__init__.py !wget -O num2words/base.py https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-icelandic/raw/main/num2words/base.py !wget -O num2words/compat.py https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-icelandic/raw/main/num2words/compat.py !wget -O num2words/currency.py https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-icelandic/raw/main/num2words/currency.py !wget -O num2words/lang_EU.py https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-icelandic/raw/main/num2words/lang_EU.py !wget -O num2words/lang_IS.py https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-icelandic/raw/main/num2words/lang_IS.py !wget -O num2words/utils.py https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-icelandic/raw/main/num2words/utils.py # Malromur_test selected based on gender and age !wget -O malromur_test.csv https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-icelandic/raw/main/malromur_test.csv # Normalizer !wget -O normalizer.py https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-icelandic/raw/main/normalizer.py ``` **Prediction** ```python import librosa import torch import torchaudio from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor from datasets import load_dataset import numpy as np import re import string import IPython.display as ipd from normalizer import Normalizer normalizer = Normalizer(lang="is") def speech_file_to_array_fn(batch): speech_array, sampling_rate = torchaudio.load(batch["path"]) speech_array = speech_array.squeeze().numpy() speech_array = librosa.resample(np.asarray(speech_array), sampling_rate, 16_000) batch["speech"] = speech_array return batch def predict(batch): features = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True) input_values = features.input_values.to(device) attention_mask = features.attention_mask.to(device) with torch.no_grad(): logits = model(input_values, attention_mask=attention_mask).logits pred_ids = torch.argmax(logits, dim=-1) batch["predicted"] = processor.batch_decode(pred_ids) return batch device = torch.device("cuda" if torch.cuda.is_available() else "cpu") processor = Wav2Vec2Processor.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-icelandic") model = Wav2Vec2ForCTC.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-icelandic").to(device) dataset = load_dataset("csv", data_files={"test": "./malromur_test.csv"})["test"] dataset = dataset.map( normalizer, fn_kwargs={"do_lastspace_removing": True, "text_key_name": "cleaned_sentence"}, remove_columns=list(set(dataset.column_names) - set(['cleaned_sentence', 'path'])) ) dataset = dataset.map(speech_file_to_array_fn) result = dataset.map(predict, batched=True, batch_size=8) max_items = np.random.randint(0, len(result), 20).tolist() for i in max_items: reference, predicted = result["cleaned_sentence"][i], result["predicted"][i] print("reference:", reference) print("predicted:", predicted) print('---') ``` **Output:** ```text reference: lögregla rakti sporin í snjónum predicted: lögregla rakti sporinn í snjónum --- reference: vaðlatúni predicted: vaðlatúni --- reference: mykjunesi predicted: mikjunesi --- reference: miðey predicted: miðey --- reference: tveir mótmæla við stjórnarráðsbygginguna predicted: tveir mótmæla við stjórnarráðsbegginguna --- reference: furðustrandir mest selda bók ársins predicted: furðustrandir mest seldabók ársins --- reference: flekar brenndir í kvöld predicted: flekar brenndir í kvöld --- reference: ástæðan er sögð eldgosið í grímsvötnum predicted: ástæðan er sögð eldgosið í grímsvötnum --- reference: birtingur predicted: birtingur --- reference: tvöþúsund og átján predicted: tvöþúsund og átján --- reference: einfríður predicted: einfríður --- reference: dalhúsum predicted: dalhúsum --- reference: sex stútar á ferð predicted: sex stútar á ferð --- reference: eyjamenn áfram í toppbaráttu predicted: eyjamenn áfram í toppbaráttu --- reference: þetta októberkvöld sýndi sitt rétta andlit með hráslagakulda frá vatninu predicted: þetta októberkvöld sýnsint réttla andlit með hráslagakulda frá vatninu --- reference: jes predicted: js --- reference: hersveitirnar benda hvor á aðra predicted: hersveitirnar benda hvor á aðra --- reference: þetta er hráskinnsleikur stórvelda eins og hver maður vissi predicted: þetta er hráskinnsleikur stórvelda eins og hver maður vissi --- reference: umferð efstu deildar hófst predicted: umferð efstu deildar hófst --- reference: freisting is predicted: freisting is --- ``` ## Evaluation The model can be evaluated as follows on the test data of Malromur. ```python import librosa import torch import torchaudio from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor from datasets import load_dataset, load_metric import numpy as np import re import string from normalizer import Normalizer normalizer = Normalizer(lang="is") def speech_file_to_array_fn(batch): speech_array, sampling_rate = torchaudio.load(batch["path"]) speech_array = speech_array.squeeze().numpy() speech_array = librosa.resample(np.asarray(speech_array), sampling_rate, 16_000) batch["speech"] = speech_array return batch def predict(batch): features = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True) input_values = features.input_values.to(device) attention_mask = features.attention_mask.to(device) with torch.no_grad(): logits = model(input_values, attention_mask=attention_mask).logits pred_ids = torch.argmax(logits, dim=-1) batch["predicted"] = processor.batch_decode(pred_ids) return batch device = torch.device("cuda" if torch.cuda.is_available() else "cpu") processor = Wav2Vec2Processor.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-icelandic") model = Wav2Vec2ForCTC.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-icelandic").to(device) dataset = load_dataset("csv", data_files={"test": "./malromur_test.csv"})["test"] dataset = dataset.map( normalizer, fn_kwargs={"do_lastspace_removing": True, "text_key_name": "cleaned_sentence"}, remove_columns=list(set(dataset.column_names) - set(['cleaned_sentence', 'path'])) ) dataset = dataset.map(speech_file_to_array_fn) result = dataset.map(predict, batched=True, batch_size=8) wer = load_metric("wer") print("WER: {:.2f}".format(100 * wer.compute(predictions=result["predicted"], references=result["cleaned_sentence"]))) ``` **Test Result**: - WER: 10.74% ## Training & Report The Common Voice `train`, `validation` datasets were used for training. You can see the training states [here](#) The script used for training can be found [here](#) ## Questions? Post a Github issue on the [Wav2Vec](https://github.com/m3hrdadfi/wav2vec) repo.