[Question]What are the correct steps to follow to fine-tune a pre-trained ASR model on right-to-left language not using latin letters? #1267
State: closed | Pull request: no

I have followed all steps outlined in the link: 
[](https://developer.nvidia.com/blog/jump-start-training-for-speech-recognition-models-with-nemo/)

I am using Quartznet 15*15 model for pretraining. 
As mentioned, I have changed the vocabulary and set normalized transcription to false using the code below. 
```
quartznet.change_vocabulary(  new_vocabulary=[' ', 'ا', 'ب', 'پ', 'ت', 'ٹ', 'ث', 'ج', 'چ', 'ح', 'خ', 'د', 'ذ', 'ڈ', 'ر',
    'ض', 'س', 'ش', 'ص', 'ط', 'ظ', 'ع', 'غ', 'ف', 'ق', 'ک', 'گ', 'ل', 'م','ن','و','ہ' , 'ھ' , 'ي', 'ے', 'ۃ', 'آ', 'ء',
         ]
)
```

And

```
params['model']['train_ds']['normalize_transcripts'] = False
params['model']['validation_ds']['normalize_transcripts'] = False
```

I have also changed the labels to the Urdu alphabet with:
```
params['model']['train_ds']['labels'] = [' ', 'ا', 'ب', 'پ', 'ت', 'ٹ', 'ث', 'ج', 'چ', 'ح', 'خ', 'د', 'ذ', 'ڈ', 'ر',
    'ض', 'س', 'ش', 'ص', 'ط', 'ظ', 'ع', 'غ', 'ف', 'ق', 'ک', 'گ', 'ل', 'م','ن','و','ہ' , 'ھ' , 'ي', 'ے', 'ۃ', 'آ', 'ء',
         ]
params['model']['validation_ds']['labels'] = [' ', 'ا', 'ب', 'پ', 'ت', 'ٹ', 'ث', 'ج', 'چ', 'ح', 'خ', 'د', 'ذ', 'ڈ', 'ر',
    'ض', 'س', 'ش', 'ص', 'ط', 'ظ', 'ع', 'غ', 'ف', 'ق', 'ک', 'گ', 'ل', 'م','ن','و','ہ' , 'ھ' , 'ي', 'ے', 'ۃ', 'آ', 'ء',
         ]
```

I have used learning rate of 0.000001 as mentioned on link: 
[](https://github.com/NVIDIA/NeMo/issues/1107)

I am not sure why my WER is so high. The audio files are sampled at 16k. Please see sample screen shot below...

Is someone who attempted something similar able to help resolve this? 

```

[NeMo I 2020-10-27 22:07:37 wer:160] reference:مران بگٹ شاد امرتا سنگھ کا بھس بدل کر منڈھے پر چھ آا
[NeMo I 2020-10-27 22:07:37 wer:161] decoded  : ابڈب گبگب  بذب ب خء ابف بمٹشفگبءبارتخروتبءخب مبمتاف بم اخءخڈابخبڈبمبڈذ ب بابٹگۃءمٹحڈاءمءمپخفابڈفبامافاڈ
[NeMo I 2020-10-27 22:07:38 wer:159] 
    
[NeMo I 2020-10-27 22:07:38 wer:160] reference:عمر چور اور اغوا کار پراچہ کو رشم باھ سے باندھ کر ٹھڈے لگاتا موضع کے باہر مقبوضہ تھانہ تک لاا
[NeMo I 2020-10-27 22:07:38 wer:161] decoded  :ت سبڈب متمتڈخ طابابمبمگبلابف بچابرببمبڈراتخبمبمبخ پتمخڈخذ رخباخاباخ ت اب مبتءتءبء اب مب تبمرامڈ ءما ب خبب خبڈ بمبم بمربڈ ءطڈبخب سباڈف مبڈابمبٹڈف تذم مذءمبخگبمرمڈمبثبخگبٹمبگبءبخءٹبمگ
[NeMo I 2020-10-27 22:07:40 wer:159] 
    
[NeMo I 2020-10-27 22:07:40 wer:160] reference:ورلڈکپ نے اقوام کا افسوسناک تنا اور انگخت ہچ کا اور دنا م شگفتہ کلچر بدار کا
[NeMo I 2020-10-27 22:07:40 wer:161] decoded  :ت اڈباڈ ذگذافڈچ بٹمتواخڈا مخابخ بمبو پ ءتطءطشخ گمتمب بءبخگبمبڈخا ب ابباتبء  خ فتف بءبم ڈب بب مءٹمب ا خ ب راذخف ءا رلءستڈبببابڈا ابلرخبخفگ

```

**Environment overview (please complete the following information)**

 - Environment location: Kaggle
 - Method of NeMo install: [pip install or from source]. !python -m pip install git+https://github.com/NVIDIA/NeMo.git@main#egg=nemo_toolkit[all]
**Environment details**
- OS version: Kaggle
- PyTorch version: 1.0.3
- Python version: 3.7



Sadly, I don't know any right-to-left language so I can not comment on specifics.

However, I would imagine that right-to-left could pose a problem for CTC mechanism because it tries to learn alignments left-to-right.  Have you tried doing a mirror reflection of your transcripts (e.g. converting them to left-to-right), then trying to learn such model first. Then, if it works, you can just mirror predictions and obtain the proper spelling?

Also, learning rate of 0.000001 is terribly small. Are you using Novograd? Try 0.001 maybe higher, maybe lower - this is something that you need to experiment with.