Initial model

README.md

@@ -0,0 +1,182 @@
+---
+language: ka
+datasets:
+- common_voice
+tags:
+- audio
+- automatic-speech-recognition
+- speech
+- xlsr-fine-tuning-week
+license: apache-2.0
+widget:
+- label: Common Voice sample 566
+  src: https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-georgian/resolve/main/sample566.flac
+- label: Common Voice sample 95
+  src: https://huggingface.co/m3hrdadfi/wav2vec2-large-xlsr-georgian/resolve/main/sample95.flac
+model-index:
+- name: XLSR Wav2Vec2 Georgian by Mehrdad Farahani
+  results:
+  - task: 
+      name: Speech Recognition
+      type: automatic-speech-recognition
+    dataset:
+      name: Common Voice ka
+      type: common_voice
+      args: ka
+    metrics:
+       - name: Test WER
+         type: wer
+         value: 54.00
+        
+---
+
+# Wav2Vec2-Large-XLSR-53 Georgian
+
+Fine-tuned [facebook/wav2vec2-large-xlsr-53](https://huggingface.co/facebook/wav2vec2-large-xlsr-53) in Georgian using [Common Voice](https://huggingface.co/datasets/common_voice). When using this model, make sure that your speech input is sampled at 16kHz.
+
+## Usage
+The model can be used directly (without a language model) as follows:
+
+```bash
+!pip install git+https://github.com/huggingface/datasets.git
+!pip install git+https://github.com/huggingface/transformers.git
+!pip install torchaudio
+!pip install librosa
+!pip install jiwer
+```
+
+```python
+import torch
+import torchaudio
+from datasets import load_dataset, load_metric
+from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor
+
+import librosa
+
+import pandas as pd
+import numpy as np
+
+import random
+import os
+import string
+import six
+import re
+
+import IPython.display as ipd
+
+# Loading the datasets
+dataset = load_dataset("common_voice", "ka", split="test")
+print(dataset)
+
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+processor = Wav2Vec2Processor.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-georgian")
+model = Wav2Vec2ForCTC.from_pretrained("m3hrdadfi/wav2vec2-large-xlsr-georgian").to(device)
+
+
+# Preprocessing the datasets.
+chars_to_ignore_regex = f"""[{"".join([
+    ",", "?", ".", "!", "-", ";", ":", '""', "%", "'", '"', "�",
+    "#", "!", "?", "«", "»", "(", ")", "؛", ",", "?", ".", "!", "-", ";", ":", '"', 
+    "“", "%", "‘", "�", "–", "…", "_", "”", '“', '„'
+])}]"""
+
+def remove_special_characters(text, chars_to_ignore):
+    text = re.sub(chars_to_ignore, '', text).lower() + " "
+    return text
+
+def normalizer(batch, chars_to_ignore):
+    text = batch["sentence"]
+    text = remove_special_characters(text, chars_to_ignore)
+    batch["sentence"] = text
+    return batch
+
+# We need to read the aduio files as arrays
+def speech_file_to_array_fn(batch):
+    speech_array, sampling_rate = torchaudio.load(batch["path"])
+    speech_array = speech_array.squeeze().numpy()
+    speech_array = librosa.resample(np.asarray(speech_array), sampling_rate, 16_000)
+
+    batch["speech"] = speech_array
+    return batch
+
+def predict(batch):
+    features = processor(batch["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)
+
+    input_values = features.input_values.to(device)
+    attention_mask = features.attention_mask.to(device)
+
+    with torch.no_grad():
+        logits = model(input_values, attention_mask=attention_mask).logits 
+        
+    pred_ids = torch.argmax(logits, dim=-1)
+
+    batch["predicted"] = processor.batch_decode(pred_ids)[0]
+    return batch
+
+dataset = dataset.map(normalizer, fn_kwargs={"chars_to_ignore": chars_to_ignore_regex})
+dataset = dataset.map(speech_file_to_array_fn, remove_columns=list(set(dataset.column_names) - set(['sentence', 'path'])))
+result = dataset.map(predict)
+```
+
+## Prediction
+
+```python
+max_items = np.random.randint(0, len(result), 20).tolist()
+for i in max_items:
+    reference, predicted =  result["sentence"][i], result["predicted"][i]
+    print("reference:", reference)
+    print("predicted:", predicted)
+    print('---')
+```
+
+```text
+reference: ადმინისტრაციული ცენტრი ქალაქი იმიშლი 
+predicted: ადმინისტრაციული ცენტრი ქალაქი იმიშლი
+---
+reference: დაიბადა ადვოკატის ოჯახში 
+predicted: აიბადა ადმოკატის ოჯახში
+---
+reference: აღსანიშნავია რომ სიმღერა წარმოადგენს პოლ მაკკარტნისა და ჯორჯ ჰარისონის იშვიათ ვოკალურ დუეტს 
+predicted: აღსენიშნავიარო სიმღე რაწარმოადგემს ბოლ მაკარდნის და ჯორჩხარისონის იშვიად ვოკალურ დუეთს
+---
+reference: იკრძალებოდა წირვალოცვა ქართულ ენაზე 
+predicted: იკრძალებოდ��� წირვა ლოცვა ქართულ ენაზე
+---
+reference: აღმართულია ვალესა და ბერნის კანტონების საზღვარზე 
+predicted: აღმართულია ვალესა და ბერნის კანთონების საზღვარზე
+---
+reference: აქ იგი მიიწვიეს სამხატვრო აკადემიაში სადაც სიცოცხლის ბოლომდე ეწეოდა პედაგოგიურ მოღვაწეობას 
+predicted: აქ იგი მიისწრვიეს სამხატრო აკადემი აშისა და ციცაცხლის ბოლომდე ეწყებობ და პედაგუდივირ მოყვაწევებას
+---
+reference: კლარისა თანხმდება შემოთავაზებაზე და ლექტერის დახმარებით სერიული მკვლელის კვალს დაადგება 
+predicted: კლარის თან ხვდება შემუთავაზე ბაზე და ლექტერის დახმარებიც სერიური მკვლელის კველს დაადგებაა
+---
+reference: იბრძოდა ტყვეებით ვაჭრობის წინააღმდეგ 
+predicted: დიბრძოტო ტყვეებით ვაჭრობის წინააღდეგ
+---
+reference: სათავსს აღმოსავლეთით და დასავლეთით თითო სარკმელი აქვს 
+predicted: სათავს აღმოსაველეთი და დასავლეთ მთიდო სარკმელი აქვს
+---
+reference: იგი მდებარეობს ქალაქის ჩრდილოაღმოსავლეთ ნაწილში 
+predicted: იგი მდებარეობს ქალაქის ჩრდილო აღმოსავლეთ ნაწილში
+---
+```
+
+## Evaluation
+
+```python
+wer = load_metric("wer")
+
+print("WER: {:.2f}".format(100 * wer.compute(predictions=result["predicted"], references=result["sentence"])))
+```
+
+**Test Result**: 
+- WER: 54.00%
+
+
+## Training & Report
+The Common Voice `train`, `validation` datasets were used for training.
+
+You can see the training states [here](https://wandb.ai/m3hrdadfi/finetuned_wav2vec_xlsr_georgian/reports/Fine-Tuning-for-Wav2Vec2-Large-XLSR-53-Georgian--Vmlldzo1NTg5MDQ?accessToken=rsmd0p83iln13yq23b9kzj8bim6nco21w8cqn2tb19v51okakqk92c71h6hbxmfj)
+
+The script used for training can be found [here](https://colab.research.google.com/github/m3hrdadfi/notebooks/blob/main/Fine_Tune_XLSR_Wav2Vec2_on_Georgian_ASR_with_%F0%9F%A4%97_Transformers_ipynb.ipynb)

all_results.json

@@ -0,0 +1,24 @@
+{
+    "epoch": 30.0,
+    "eval_loss": 0.4455166161060333,
+    "eval_mem_cpu_alloc_delta": 52916700,
+    "eval_mem_cpu_peaked_delta": 92345080,
+    "eval_mem_gpu_alloc_delta": 0,
+    "eval_mem_gpu_peaked_delta": 5249111040,
+    "eval_runtime": 81.4513,
+    "eval_samples": 654,
+    "eval_samples_per_second": 8.029,
+    "eval_wer": 0.5288702928870292,
+    "init_mem_cpu_alloc_delta": 9478038,
+    "init_mem_cpu_peaked_delta": 18306,
+    "init_mem_gpu_alloc_delta": 1261911040,
+    "init_mem_gpu_peaked_delta": 0,
+    "total_flos": 8.556740517881789e+18,
+    "train_mem_cpu_alloc_delta": 12260352,
+    "train_mem_cpu_peaked_delta": 186508822,
+    "train_mem_gpu_alloc_delta": 3794085376,
+    "train_mem_gpu_peaked_delta": 6038033408,
+    "train_runtime": 8781.3793,
+    "train_samples": 1585,
+    "train_samples_per_second": 0.109
+}

config.json

@@ -0,0 +1,76 @@
+{
+  "_name_or_path": "facebook/wav2vec2-large-xlsr-53",
+  "activation_dropout": 0.0,
+  "apply_spec_augment": true,
+  "architectures": [
+    "Wav2Vec2ForCTC"
+  ],
+  "attention_dropout": 0.1,
+  "bos_token_id": 1,
+  "conv_bias": true,
+  "conv_dim": [
+    512,
+    512,
+    512,
+    512,
+    512,
+    512,
+    512
+  ],
+  "conv_kernel": [
+    10,
+    3,
+    3,
+    3,
+    3,
+    2,
+    2
+  ],
+  "conv_stride": [
+    5,
+    2,
+    2,
+    2,
+    2,
+    2,
+    2
+  ],
+  "ctc_loss_reduction": "mean",
+  "ctc_zero_infinity": true,
+  "do_stable_layer_norm": true,
+  "eos_token_id": 2,
+  "feat_extract_activation": "gelu",
+  "feat_extract_dropout": 0.0,
+  "feat_extract_norm": "layer",
+  "feat_proj_dropout": 0.0,
+  "final_dropout": 0.0,
+  "gradient_checkpointing": true,
+  "hidden_act": "gelu",
+  "hidden_dropout": 0.1,
+  "hidden_size": 1024,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-05,
+  "layerdrop": 0.1,
+  "mask_channel_length": 10,
+  "mask_channel_min_space": 1,
+  "mask_channel_other": 0.0,
+  "mask_channel_prob": 0.0,
+  "mask_channel_selection": "static",
+  "mask_feature_length": 10,
+  "mask_feature_prob": 0.0,
+  "mask_time_length": 10,
+  "mask_time_min_space": 1,
+  "mask_time_other": 0.0,
+  "mask_time_prob": 0.05,
+  "mask_time_selection": "static",
+  "model_type": "wav2vec2",
+  "num_attention_heads": 16,
+  "num_conv_pos_embedding_groups": 16,
+  "num_conv_pos_embeddings": 128,
+  "num_feat_extract_layers": 7,
+  "num_hidden_layers": 24,
+  "pad_token_id": 0,
+  "transformers_version": "4.5.0.dev0",
+  "vocab_size": 38
+}

eval_results.json

@@ -0,0 +1,12 @@
+{
+    "epoch": 30.0,
+    "eval_loss": 0.4455166161060333,
+    "eval_mem_cpu_alloc_delta": 52916700,
+    "eval_mem_cpu_peaked_delta": 92345080,
+    "eval_mem_gpu_alloc_delta": 0,
+    "eval_mem_gpu_peaked_delta": 5249111040,
+    "eval_runtime": 81.4513,
+    "eval_samples": 654,
+    "eval_samples_per_second": 8.029,
+    "eval_wer": 0.5288702928870292
+}

preprocessor_config.json

@@ -0,0 +1,8 @@
+{
+  "do_normalize": true,
+  "feature_size": 1,
+  "padding_side": "right",
+  "padding_value": 0.0,
+  "return_attention_mask": true,
+  "sampling_rate": 16000
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c0938b613c1c3fe1abd582f27dfec45efeef27402c30a8bc0de2408aef51c21
+size 1262089623

result.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c42bc8fa4f5eca7ff0ab10d3692e0b33144189969b2b23f107e68c3f4e47803
+size 3183

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": "<s>", "eos_token": "</s>", "unk_token": "<unk>", "pad_token": "<pad>"}

tokenizer_config.json

@@ -0,0 +1 @@
+{"unk_token": "<unk>", "bos_token": "<s>", "eos_token": "</s>", "pad_token": "<pad>", "do_lower_case": false, "word_delimiter_token": "|"}

train_results.json

@@ -0,0 +1,15 @@
+{
+    "epoch": 30.0,
+    "init_mem_cpu_alloc_delta": 9478038,
+    "init_mem_cpu_peaked_delta": 18306,
+    "init_mem_gpu_alloc_delta": 1261911040,
+    "init_mem_gpu_peaked_delta": 0,
+    "total_flos": 8.556740517881789e+18,
+    "train_mem_cpu_alloc_delta": 12260352,
+    "train_mem_cpu_peaked_delta": 186508822,
+    "train_mem_gpu_alloc_delta": 3794085376,
+    "train_mem_gpu_peaked_delta": 6038033408,
+    "train_runtime": 8781.3793,
+    "train_samples": 1585,
+    "train_samples_per_second": 0.109
+}

trainer_state.json

@@ -0,0 +1,297 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 30.0,
+  "global_step": 960,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 1.56,
+      "learning_rate": 7.5e-05,
+      "loss": 13.0978,
+      "step": 50
+    },
+    {
+      "epoch": 1.56,
+      "eval_loss": 13.780136108398438,
+      "eval_runtime": 82.8799,
+      "eval_samples_per_second": 7.891,
+      "eval_wer": 1.0,
+      "step": 50
+    },
+    {
+      "epoch": 3.12,
+      "learning_rate": 0.00015,
+      "loss": 7.3093,
+      "step": 100
+    },
+    {
+      "epoch": 3.12,
+      "eval_loss": 3.198237419128418,
+      "eval_runtime": 81.5893,
+      "eval_samples_per_second": 8.016,
+      "eval_wer": 1.0,
+      "step": 100
+    },
+    {
+      "epoch": 4.69,
+      "learning_rate": 0.000225,
+      "loss": 3.0745,
+      "step": 150
+    },
+    {
+      "epoch": 4.69,
+      "eval_loss": 3.1082892417907715,
+      "eval_runtime": 82.4037,
+      "eval_samples_per_second": 7.937,
+      "eval_wer": 1.0,
+      "step": 150
+    },
+    {
+      "epoch": 6.25,
+      "learning_rate": 0.0003,
+      "loss": 3.0551,
+      "step": 200
+    },
+    {
+      "epoch": 6.25,
+      "eval_loss": 3.0994772911071777,
+      "eval_runtime": 82.7226,
+      "eval_samples_per_second": 7.906,
+      "eval_wer": 1.0,
+      "step": 200
+    },
+    {
+      "epoch": 7.81,
+      "learning_rate": 0.00028026315789473683,
+      "loss": 3.0632,
+      "step": 250
+    },
+    {
+      "epoch": 7.81,
+      "eval_loss": 3.0916755199432373,
+      "eval_runtime": 83.5323,
+      "eval_samples_per_second": 7.829,
+      "eval_wer": 1.0,
+      "step": 250
+    },
+    {
+      "epoch": 9.38,
+      "learning_rate": 0.0002605263157894737,
+      "loss": 3.0391,
+      "step": 300
+    },
+    {
+      "epoch": 9.38,
+      "eval_loss": 3.0707435607910156,
+      "eval_runtime": 82.7328,
+      "eval_samples_per_second": 7.905,
+      "eval_wer": 1.0,
+      "step": 300
+    },
+    {
+      "epoch": 10.94,
+      "learning_rate": 0.00024078947368421052,
+      "loss": 3.0321,
+      "step": 350
+    },
+    {
+      "epoch": 10.94,
+      "eval_loss": 3.0443670749664307,
+      "eval_runtime": 84.1437,
+      "eval_samples_per_second": 7.772,
+      "eval_wer": 1.0,
+      "step": 350
+    },
+    {
+      "epoch": 12.5,
+      "learning_rate": 0.00022105263157894733,
+      "loss": 3.0069,
+      "step": 400
+    },
+    {
+      "epoch": 12.5,
+      "eval_loss": 2.998474359512329,
+      "eval_runtime": 83.9178,
+      "eval_samples_per_second": 7.793,
+      "eval_wer": 1.0,
+      "step": 400
+    },
+    {
+      "epoch": 14.06,
+      "learning_rate": 0.0002013157894736842,
+      "loss": 2.9623,
+      "step": 450
+    },
+    {
+      "epoch": 14.06,
+      "eval_loss": 2.866849184036255,
+      "eval_runtime": 82.5906,
+      "eval_samples_per_second": 7.919,
+      "eval_wer": 1.0,
+      "step": 450
+    },
+    {
+      "epoch": 15.62,
+      "learning_rate": 0.00018157894736842105,
+      "loss": 2.4771,
+      "step": 500
+    },
+    {
+      "epoch": 15.62,
+      "eval_loss": 1.5367902517318726,
+      "eval_runtime": 85.6456,
+      "eval_samples_per_second": 7.636,
+      "eval_wer": 0.9838912133891213,
+      "step": 500
+    },
+    {
+      "epoch": 17.19,
+      "learning_rate": 0.00016184210526315788,
+      "loss": 1.0561,
+      "step": 550
+    },
+    {
+      "epoch": 17.19,
+      "eval_loss": 0.6924143433570862,
+      "eval_runtime": 85.1658,
+      "eval_samples_per_second": 7.679,
+      "eval_wer": 0.7548117154811715,
+      "step": 550
+    },
+    {
+      "epoch": 18.75,
+      "learning_rate": 0.0001421052631578947,
+      "loss": 0.5288,
+      "step": 600
+    },
+    {
+      "epoch": 18.75,
+      "eval_loss": 0.5334728956222534,
+      "eval_runtime": 83.737,
+      "eval_samples_per_second": 7.81,
+      "eval_wer": 0.6569037656903766,
+      "step": 600
+    },
+    {
+      "epoch": 20.31,
+      "learning_rate": 0.00012236842105263157,
+      "loss": 0.3581,
+      "step": 650
+    },
+    {
+      "epoch": 20.31,
+      "eval_loss": 0.48591092228889465,
+      "eval_runtime": 86.2479,
+      "eval_samples_per_second": 7.583,
+      "eval_wer": 0.605857740585774,
+      "step": 650
+    },
+    {
+      "epoch": 21.88,
+      "learning_rate": 0.00010263157894736841,
+      "loss": 0.2638,
+      "step": 700
+    },
+    {
+      "epoch": 21.88,
+      "eval_loss": 0.4631027579307556,
+      "eval_runtime": 84.0825,
+      "eval_samples_per_second": 7.778,
+      "eval_wer": 0.5648535564853556,
+      "step": 700
+    },
+    {
+      "epoch": 23.44,
+      "learning_rate": 8.289473684210526e-05,
+      "loss": 0.2284,
+      "step": 750
+    },
+    {
+      "epoch": 23.44,
+      "eval_loss": 0.4597685933113098,
+      "eval_runtime": 86.122,
+      "eval_samples_per_second": 7.594,
+      "eval_wer": 0.5594142259414226,
+      "step": 750
+    },
+    {
+      "epoch": 25.0,
+      "learning_rate": 6.315789473684209e-05,
+      "loss": 0.1965,
+      "step": 800
+    },
+    {
+      "epoch": 25.0,
+      "eval_loss": 0.4614764153957367,
+      "eval_runtime": 86.0272,
+      "eval_samples_per_second": 7.602,
+      "eval_wer": 0.5535564853556485,
+      "step": 800
+    },
+    {
+      "epoch": 26.56,
+      "learning_rate": 4.342105263157895e-05,
+      "loss": 0.1837,
+      "step": 850
+    },
+    {
+      "epoch": 26.56,
+      "eval_loss": 0.4499300718307495,
+      "eval_runtime": 89.3292,
+      "eval_samples_per_second": 7.321,
+      "eval_wer": 0.5349372384937239,
+      "step": 850
+    },
+    {
+      "epoch": 28.12,
+      "learning_rate": 2.3684210526315787e-05,
+      "loss": 0.187,
+      "step": 900
+    },
+    {
+      "epoch": 28.12,
+      "eval_loss": 0.45425695180892944,
+      "eval_runtime": 85.6275,
+      "eval_samples_per_second": 7.638,
+      "eval_wer": 0.5345188284518828,
+      "step": 900
+    },
+    {
+      "epoch": 29.69,
+      "learning_rate": 3.947368421052631e-06,
+      "loss": 0.1568,
+      "step": 950
+    },
+    {
+      "epoch": 29.69,
+      "eval_loss": 0.4458238184452057,
+      "eval_runtime": 84.9753,
+      "eval_samples_per_second": 7.696,
+      "eval_wer": 0.5290794979079498,
+      "step": 950
+    },
+    {
+      "epoch": 30.0,
+      "step": 960,
+      "total_flos": 8.556740517881789e+18,
+      "train_runtime": 8781.3793,
+      "train_samples_per_second": 0.109
+    },
+    {
+      "epoch": 30.0,
+      "eval_loss": 0.4455166161060333,
+      "eval_runtime": 81.4513,
+      "eval_samples_per_second": 8.029,
+      "eval_wer": 0.5288702928870292,
+      "step": 960
+    }
+  ],
+  "max_steps": 960,
+  "num_train_epochs": 30,
+  "total_flos": 8.556740517881789e+18,
+  "trial_name": null,
+  "trial_params": null
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b09499473860372d1a5755c32562e264aa7cbc7d9f4c4491ed862c399a413bb7
+size 2351

vocab.json

@@ -0,0 +1 @@
+{"<pad>": 0, "<s>": 1, "</s>": 2, "<unk>": 3, "|": 4, "ა": 5, "ბ": 6, "გ": 7, "დ": 8, "ე": 9, "ვ": 10, "ზ": 11, "თ": 12, "ი": 13, "კ": 14, "ლ": 15, "მ": 16, "ნ": 17, "ო": 18, "პ": 19, "ჟ": 20, "რ": 21, "ს": 22, "ტ": 23, "უ": 24, "ფ": 25, "ქ": 26, "ღ": 27, "ყ": 28, "შ": 29, "ჩ": 30, "ც": 31, "ძ": 32, "წ": 33, "ჭ": 34, "ხ": 35, "ჯ": 36, "ჰ": 37}