Update README.md

README.md

@@ -21,7 +21,7 @@ model-index:
     metrics:
        - name: Test WER
          type: wer
-         value: 39.60
+         value: 30.92
 ---
 
 # Greek (el) version of the XLSR-Wav2Vec2 automatic speech recognition (ASR) model
@@ -33,7 +33,7 @@ model-index:
 * model: XLSR-Wav2Vec2, trained for 60 epochs
 * metrics: Word Error Rate (WER)
 
-### Model description
+## Model description
 
 Wav2Vec2 is a pretrained model for Automatic Speech Recognition (ASR) and was released in September 2020 by Alexei Baevski, Michael Auli, and Alex Conneau. Soon after the superior performance of Wav2Vec2 was demonstrated on the English ASR dataset LibriSpeech, Facebook AI presented XLSR-Wav2Vec2. XLSR stands for cross-lingual speech representations and refers to XLSR-Wav2Vec2`s ability to learn speech representations that are useful across multiple languages.
 
@@ -41,7 +41,7 @@ Similar to Wav2Vec2, XLSR-Wav2Vec2 learns powerful speech representations from h
 
 This model was trained on Greek CommonVoice speech data (364MB) for 60 epochs on a single NVIDIA RTX 3080, for aprox. 8hrs.
 
-### How to use for inference:
+## How to use for inference:
 
 For live demo, make sure that speech files are sampled at 16kHz.
 
@@ -63,7 +63,7 @@ import numpy as np
 from datasets import load_dataset, load_metric
 import torch
 
-chars_to_ignore_regex = '[\,\?\.\!\-\;\:\"\“\%\‘\”\�]'
+chars_to_ignore_regex = '[\\,\\?\\.\\!\\-\\;\\:\\"\\“\\%\\‘\\”\\�]'
 
 def remove_special_characters(batch):
     batch["text"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower() + " "
@@ -96,13 +96,13 @@ def prepare_dataset(batch):
 
 # Loading model and dataset processor
 
-model = Wav2Vec2ForCTC.from_pretrained(".").to("cuda")
-processor = Wav2Vec2Processor.from_pretrained(".")
+model = Wav2Vec2ForCTC.from_pretrained("lighteternal/wav2vec2-large-xlsr-53-greek").to("cuda")
+processor = Wav2Vec2Processor.from_pretrained("lighteternal/wav2vec2-large-xlsr-53-greek")
 
 
 # Preparing speech dataset to be suitable for inference
 
-common_voice_test = load_dataset("common_voice", "el", data_dir="cv-corpus-6.1-2020-12-11", split="test")
+common_voice_test = load_dataset("common_voice", "el", split="test")
 
 common_voice_test = common_voice_test.remove_columns(["accent", "age", "client_id", "down_votes", "gender", "locale", "segment", "up_votes"])
 
@@ -117,7 +117,7 @@ common_voice_test = common_voice_test.map(prepare_dataset, remove_columns=common
 
 # Loading test dataset
 
-common_voice_test_transcription = load_dataset("common_voice", "el", data_dir="./cv-corpus-6.1-2020-12-11", split="test")
+common_voice_test_transcription = load_dataset("common_voice", "el", split="test")
 
 
 #Performing inference on a random sample. Change the "example" value to try inference on different CommonVoice extracts
@@ -134,12 +134,63 @@ print("Prediction:")
 print(processor.decode(pred_ids[0]))
 # πού θέλεις να πάμε ρώτησε φοβισμένα ο βασιλιάς
 
-print("\nReference:")
+print("\
+Reference:")
 print(common_voice_test_transcription["sentence"][example].lower())
 # πού θέλεις να πάμε; ρώτησε φοβισμένα ο βασιλιάς.
 
 ```
 
+## Evaluation
+
+The model can be evaluated as follows on the Greek test data of Common Voice. 
+
+
+```python
+import torch
+import torchaudio
+from datasets import load_dataset, load_metric
+from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
+import re
+
+test_dataset = load_dataset("common_voice", "el", split="test") 
+wer = load_metric("wer")
+
+processor = Wav2Vec2Processor.from_pretrained("lighteternal/wav2vec2-large-xlsr-53-greek") 
+model = Wav2Vec2ForCTC.from_pretrained("lighteternal/wav2vec2-large-xlsr-53-greek")
+model.to("cuda")
+
+chars_to_ignore_regex = '[\\,\\?\\.\\!\\-\\;\\:\\"\\“\\%\\‘\\”\\�]'
+resampler = torchaudio.transforms.Resample(48_000, 16_000)
+
+# Preprocessing the datasets.
+# We need to read the aduio files as arrays
+
+def speech_file_to_array_fn(batch):
+  batch["sentence"] = re.sub(chars_to_ignore_regex, '', batch["sentence"]).lower()
+  speech_array, sampling_rate = torchaudio.load(batch["path"])
+  batch["speech"] = resampler(speech_array).squeeze().numpy()
+  return batch
+  
+test_dataset = test_dataset.map(speech_file_to_array_fn)
+
+# Preprocessing the datasets.
+# We need to read the aduio files as arrays
+
+def evaluate(batch):
+  inputs = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
+  with torch.no_grad():
+    logits = model(inputs.input_values.to("cuda"), attention_mask=inputs.attention_mask.to("cuda")).logits
+  pred_ids = torch.argmax(logits, dim=-1)
+  batch["pred_strings"] = processor.batch_decode(pred_ids)
+  return batch
+
+result = test_dataset.map(evaluate, batched=True, batch_size=8)
+print("WER: {:2f}".format(100 * wer.compute(predictions=result["pred_strings"], references=result["sentence"])))
+```
+
+**Test Result**: 30.92 % 
+
 ### How to use for training:
 
 Instructions and code to replicate the process are provided in the Fine_Tune_XLSR_Wav2Vec2_on_Greek_ASR_with_���_Transformers.ipynb notebook.
@@ -151,7 +202,7 @@ Instructions and code to replicate the process are provided in the Fine_Tune_XLS
 | ----------- | ----------- |
 | Training Loss | 0.0287 |
 | Validation Loss | 0.6062 |
-| WER on CommonVoice Test *| 0.3960 |
+| WER on CommonVoice Test *| 0.3092 |
 * Reference transcripts were lower-cased and striped of punctuation and special characters.
 
 Full metrics log here: