Reference string blank
State: closed | Pull request: no

**Describe the bug**

I have changed the vocabulary on pretrained QuartzNet:

```
vocabulary=list(sorted(set(''.join([x.target.value for x in samples]))))
quartznet.change_vocabulary(new_vocabulary=vocabulary)
```
giving

```
NeMo I 2020-10-13 10:39:05 ctc_models:189] Changed decoder to output to [' ', 'ሀ', 'ሁ', 'ሂ', 'ሃ', 'ሄ', 'ህ', 'ሆ', 'ለ', 'ሉ', 'ሊ', 'ላ', 'ሌ', 'ል', 'ሎ', 'ሏ', 'ሐ', 'ሑ', 'ሒ', 'ሔ', 'ሕ', 'መ', 'ሙ', 'ሚ', 'ማ', 'ሜ', 'ም', 'ሞ', 'ሟ', 'ሠ', 'ሡ', 'ሣ', 'ሥ', 'ሦ', 'ረ', 'ሩ', 'ሪ', 'ራ', 'ሬ', 'ር', 'ሮ', 'ሯ', 'ሰ', 'ሱ', 'ሲ', 'ሳ', 'ሴ', 'ስ', 'ሶ', 'ሷ', 'ሸ', 'ሹ', 'ሺ', 'ሻ', 'ሼ', 'ሽ', 'ሾ', 'ሿ', 'ቀ', 'ቁ', 'ቂ', 'ቃ', 'ቄ', 'ቅ', 'ቆ', 'ቋ', 'በ', 'ቡ', 'ቢ', 'ባ', 'ቤ', 'ብ', 'ቦ', 'ቧ', 'ቨ', 'ቪ', 'ቫ', 'ቬ', 'ቭ', 'ተ', 'ቱ', 'ቲ', 'ታ', 'ቴ', 'ት', 'ቶ', 'ቷ', 'ቸ', 'ቹ', 'ቺ', 'ቻ', 'ቼ', 'ች', 'ቾ', 'ቿ', 'ኋ', 'ነ', 'ኑ', 'ኒ', 'ና', 'ኔ', 'ን', 'ኖ', 'ኗ', 'ኘ', 'ኙ', 'ኚ', 'ኛ', 'ኜ', 'ኝ', 'ኞ', 'አ', 'ኡ', 'ኢ', 'ኤ', 'እ', 'ኦ', 'ኧ', 'ከ', 'ኩ', 'ኪ', 'ካ', 'ኬ', 'ክ', 'ኮ', 'ኳ', 'ኸ', 'ኻ', 'ኼ', 'ኽ', 'ወ', 'ዉ', 'ዊ', 'ዋ', 'ዌ', 'ው', 'ዎ', 'ዐ', 'ዑ', 'ዓ', 'ዕ', 'ዘ', 'ዙ', 'ዚ', 'ዛ', 'ዜ', 'ዝ', 'ዞ', 'ዟ', 'ዠ', 'ዢ', 'ዣ', 'ዤ', 'ዥ', 'ዦ', 'የ', 'ዩ', 'ዪ', 'ያ', 'ዬ', 'ይ', 'ዮ', 'ደ', 'ዱ', 'ዲ', 'ዳ', 'ዴ', 'ድ', 'ዶ', 'ዷ', 'ጀ', 'ጁ', 'ጂ', 'ጃ', 'ጄ', 'ጅ', 'ጆ', 'ጇ', 'ገ', 'ጉ', 'ጊ', 'ጋ', 'ጌ', 'ግ', 'ጎ', 'ጓ', 'ጠ', 'ጡ', 'ጢ', 'ጣ', 'ጤ', 'ጥ', 'ጦ', 'ጧ', 'ጨ', 'ጩ', 'ጪ', 'ጫ', 'ጬ', 'ጭ', 'ጮ', 'ጱ', 'ጲ', 'ጳ', 'ጴ', 'ጵ', 'ጸ', 'ጹ', 'ጺ', 'ጻ', 'ጼ', 'ጽ', 'ጾ', 'ጿ', 'ፀ', 'ፁ', 'ፃ', 'ፄ', 'ፅ', 'ፆ', 'ፈ', 'ፉ', 'ፊ', 'ፋ', 'ፌ', 'ፍ', 'ፎ', 'ፏ', 'ፐ', 'ፑ', 'ፒ', 'ፓ', 'ፔ', 'ፕ', 'ፖ'] vocabulary.
```

I defined training samples such as

```
{'audio_filepath': 'NIST/openasr20_amharic/build/audio_split/BABEL_OP3_307_15902_20140422_235151_outLine_2606600_2615480.wav',
  'duration': 1.11,
  'text': 'ጉርድ ሾላ ምነው'},
 {'audio_filepath': 'NIST/openasr20_amharic/build/audio_split/BABEL_OP3_307_25767_20140403_234644_outLine_3567640_3637240.wav',
  'duration': 8.7,
  'text': 'እንትን እንትኑን አይቼዋለሁ ሳምፕሉን አይቼዋለሁ ግን ሳምፕሉን አይቼዋለሁ በሳምፕሉ ጥራት ደረጃ ከሆነ ሌላውም ቲ ቲሸርት የሚመጣው ትንሽ እንትን አለበት'},
```
When I run the trainer as follows:

```
quartznet.setup_training_data(train_data_config=params['model']['train_ds'])
quartznet.setup_validation_data(val_data_config=params['model']['validation_ds'])
trainer = pl.Trainer(gpus=[0], max_epochs=2, amp_level='O1', precision=16)
```

I see this output, where the decoded shows in Amharic but the reference is blank:

```
[NeMo I 2020-10-13 10:39:14 wer:149] reference:   
[NeMo I 2020-10-13 10:39:14 wer:150] decoded  :ቸሡፐሸዞሡፖገሡቸፖሦጡሦዮገሡቸጴዞቸቦችሡፉዞፁሾሦሡቸሡቸሡሁዞቸቼሡቸሡቋአቸቿሾኢፐሡ
[NeMo I 2020-10-13 10:39:14 wer:148] 
    
[NeMo I 2020-10-13 10:39:14 wer:149] reference:     
[NeMo I 2020-10-13 10:39:14 wer:150] decoded  :ቸሡቸፖጤንቸሲጴሡቸክቃሣፉሡኢሀዞሸዞፖጴዞጎጂሦዉቻዷዞቸጓንዷቸኗናፐሁዞንቸዉፐጴዞሡቃሦሡጄቋፁማንዉቫሸፁኼጊዉቺዎዞፁፉንቸትዊዞፁቃቸሦዉፐጤሡጮሸዞኗፐጤሁቸኗሡሸሡጿሦሀፖፐሟቸሡቸንፑፖቪሀዞቸፑቸሡኢዞፐቼጹሡ
[NeMo I 2020-10-13 10:39:15 wer:148] 
```
**Steps/Code to reproduce bug**

Here is my whole training script:

```
import logging, json, copy, nemo
from Cfg import Cfg
from ruamel.yaml import YAML
from omegaconf import DictConfig
import nemo.collections.asr as nemo_asr
import pytorch_lightning as pl
from amharic_vocabulary import amharic_vocabulary

logger = logging.getLogger()
logger.setLevel(logging.INFO)
C = Cfg('NIST', 8000, 'amharic') 
config_path = './NeMo/examples/asr/conf/config.yaml'
yaml = YAML(typ='safe')
with open(config_path) as f:
    params = yaml.load(f)
train_manifest=f'{C.build_dir}/train_manifest.json'
test_manifest=f'{C.build_dir}/test_manifest.json'
params['model']['train_ds']['manifest_filepath'] = train_manifest
params['model']['validation_ds']['manifest_filepath'] = test_manifest
quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")
quartznet.change_vocabulary(new_vocabulary=amharic_vocabulary)
quartznet.setup_training_data(train_data_config=params['model']['train_ds'])
quartznet.setup_validation_data(val_data_config=params['model']['validation_ds'])
trainer = pl.Trainer(gpus=[0], max_epochs=1, amp_level='O1', precision=16)
trainer.fit(quartznet)
```

**Expected behavior**

The "reference" string should match the manifest.

**Environment overview (please complete the following information)**

 - Environment location: Ubuntu 20.04, Anaconda Python 3.7
 - Method of NeMo install: pip install

**Environment details**

- OS version: Ubuntu 20.04LTS
- PyTorch version: 1.6.0
- Python version: 3.7.4
- GPU model: NVidia GeForce RTX 2080TI
- CUDA: 10.1.243
- cuDNN: 7.6.5

**Additional context**

Trying to follow notebook ASR_with_Nemo.ipynb to transfer learn a new language.

**NOTE**
I fixed this problem by starting from scratch from a customized YAML file and not by trying to change the vocabulary on the fly, which seems iffy.

Of course that means I got a whole lotta training to do, but at least I've flipped the problem around a little bit:

```
[NeMo I 2020-10-13 12:51:10 wer:149] reference:እ ሀብቱን በደምብ ሰላም በይልኝ ኮራ ኩራት ቀልዶብሻል ስልክ ስደውልልሽ እማታነሺው
[NeMo I 2020-10-13 12:51:10 wer:150] decoded  :
[NeMo I 2020-10-13 12:51:27 wer:148] 
```


Empty references typically mean that training/validation_ds was not setup correctly - e.g. vocabulary is not passed to it as "labels" parameter.

Looking at your code:
```
import logging, json, copy, nemo
from Cfg import Cfg
from ruamel.yaml import YAML
from omegaconf import DictConfig
import nemo.collections.asr as nemo_asr
import pytorch_lightning as pl
from amharic_vocabulary import amharic_vocabulary

logger = logging.getLogger()
logger.setLevel(logging.INFO)
C = Cfg('NIST', 8000, 'amharic') 
config_path = './NeMo/examples/asr/conf/config.yaml'
yaml = YAML(typ='safe')
with open(config_path) as f:
    params = yaml.load(f)
train_manifest=f'{C.build_dir}/train_manifest.json'
test_manifest=f'{C.build_dir}/test_manifest.json'
params['model']['train_ds']['manifest_filepath'] = train_manifest
params['model']['validation_ds']['manifest_filepath'] = test_manifest
quartznet = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")
quartznet.change_vocabulary(new_vocabulary=amharic_vocabulary)
quartznet.setup_training_data(train_data_config=params['model']['train_ds'])
quartznet.setup_validation_data(val_data_config=params['model']['validation_ds'])
trainer = pl.Trainer(gpus=[0], max_epochs=1, amp_level='O1', precision=16)
trainer.fit(quartznet)
```
It looks fined to me, but I have one question - is params['model']['train_ds']['labels'] (and params['model']['validation_ds']['labels']) set to the amharic_vocabulary ? My guess is not - which is why references would be empty.

Fine-tuning should give much better results and faster, especially if you have only 1 GPU. Let me know if setting params['model']['train_ds'/'validation_ds']['labels']=amharic_vocabulary unblocks fine-tuning for you.



Thanks.  I am training 10 languages for NIST ASR20.  The NIST Sphere files are sample rate 8K but the pretrained models are 16K sample rate.  It's probably best to stay with the 8K sample rate.  So I am training one model from scratch. For the other 9 I will try the change_vocabulary method plus the params label fields fix.  In a few days...with one 2080TI this is taking a while on 10 hours of training data with clips up to 33 seconds in length.  

The demonstration notebook ASR_with_NeMo.ipynb should be updated in the Transfer Learning section to include the additional steps to update the params labels fields.  Also I recommend, for demonstration purposes, changing to a language with a different number of graphemes, as this is a better test.

NOTE: For this project I applied to the NVidia Inception program for a discount on a new GPU but I was rejected -- silently, no email notice of the decision.  If there is a free-tier NVidia Developer Cloud available that you can point me to, I can try this on cloud.  I guess it will need about 100 hours of training on the equivalent of the 2080TI.

I am going with the QuartzNet 15x5 YAML on the assumption that it is big enough to represent all the information that can be inferred from 20 hours of 8K sample data.  I widened the longest sequence parameter to 33 seconds, which is the length of the longest labelled clip in the training data.  Typically I can break speech down to 4 seconds or so per clip based on long silences, for inference on unlabelled data.   If you have suggestions on how to increase the net size for the 10 hour case, I'd appreciate.   My weights checkpoints are only 75MB or so in size.  Most weights files for this kind of problem that I have experienced are more in the 2GB size range.  So I am concerned that I am no retaining enough data.

Also I should note I spent several hours trying to figure out how checkpointing works.  Something like the following code should be incorporated into the ASR_with_NeMo.ipynb notebook, as it is not at all easy to get it right based on the docs:

```
from pytorch_lightning.callbacks import ModelCheckpoint

checkpoint_callback = ModelCheckpoint(
    filepath=model_save_dir+'/amharic_{epoch:02d}_loss_{current:.4f}_trainWER_{training_batch_wer:.4f}_valWER_{validation_wer:.4f}',
    save_top_k=-1,
    verbose=True,
    monitor='loss',
    mode='min',
    period=1,
    save_weights_only=True,
)

trainer = pl.Trainer(checkpoint_callback=checkpoint_callback, gpus=[0], max_epochs=1000, amp_level='O1', precision=16)

trainer.fit(model)
```

and then to resume training:

```
from glob import glob
models=list(sorted(glob('save/nemo_amharic/*.ckpt')))
if len(models):
    last_fn=models[-1]
    model.load_from_checkpoint(last_fn)
    print('loaded', last_fn)
```
In particular, the Checkpoint method produces files in a different format from the `save_to` and `restore_from` methods.  It took an insane amount of Googling to figure out that the checkpoint files could be read with `load_from_checkpoint`.  The notebook should show how to configure training to store checkpoints and how to load the resulting checkpoints.

One other feature that would be helpful is some automated cross-validation so that every epoch the training and validation sets are combined, shuffled, and split.  I'm training to good WER numbers on the training set and the validation set is stuck in the 90's.  So I have to manually stop and reshuffle which means rebuilding and reloading the weights in the middle and maybe losing the state of the optimizer.  This is something that should be a lot easier.

"It's probably best to stay with the 8K sample rate." 
- I would actually recommend to upsample to 16K if you want to fine-tune from one of our checkpoints (which are all 16Khz).
For 20hours, I would definitely recommend fine-tuning from one of the checkpoints. 

"My weights checkpoints are only 75MB or so in size"
- This looks correct. QuartzNet model uses separable convolutions and QuartzNet15x5 has about 20million weights only.

Thanks, I'm trying that now!  First I copied the original `./NeMo/examples/asr/conf/config.yaml` and changed these things:

- The labels at the top, from English letters to Amharic.  The config copies the labels from the top declaration into `train_ds` and `validation_ds` so there is no need to restate them there.
- The batch size (best I can do on 2080TI is 12)
- The  max duration (33 seconds up from 16.7 in the original)

Here is the new code after upsampling:
 ```
C = Cfg('NIST', 16000, 'amharic') 

config_path = 'amharic_16000.yaml'
yaml = YAML(typ='safe')
with open(config_path) as f:
    params = yaml.load(f)

params['model']['train_ds']['manifest_filepath'] = f'{C.build_dir}/train_manifest.json'
params['model']['validation_ds']['manifest_filepath'] = f'{C.build_dir}/test_manifest.json'

model = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")
model.change_vocabulary(new_vocabulary=params['labels'])
model.setup_optimization(optim_config=DictConfig(params['model']['optim']))
model.setup_training_data(train_data_config=params['model']['train_ds'])
model.setup_validation_data(val_data_config=params['model']['validation_ds'])
```

NOTE: This leaves the learning rate pretty high at 0.01.  I don't know if that's normal.  I couldn't find where the learning rate is stored in the pretrained model, so I'm assuming 0.01 is legit.

It works, I am training now with the following loop:

```
model_save_dir='save/nemo_amharic'

class ModelCheckpointAtEpochEnd(pl.callbacks.ModelCheckpoint):
    def on_epoch_end(self, trainer, pl_module):
        metrics = trainer.callback_metrics
        metrics['epoch'] = trainer.current_epoch
        trainer.checkpoint_callback.on_validation_end(trainer, pl_module)

pid=os.getpid()
dt=datetime.datetime.now().strftime("%Y%m%d_%H%M%S")

checkpoint_callback = ModelCheckpointAtEpochEnd(
    filepath=model_save_dir+'/amharic_'+f'{dt}_{pid}'+'_{epoch:02d}',
    verbose=True,
    save_top_k=-1,
    save_weights_only=False,
    period=1)

trainer = pl.Trainer(gpus=[0], max_epochs=10, amp_level='O1', precision=16, checkpoint_callback=checkpoint_callback)

while True:
    reshuffle_samples(C)
    trainer.fit(model)
```
I am not exactly sure how I would reload the model from a checkpoint.  I guess I have to do the same as first step above to configure the pretrained model for new language and then `load_from_checkpoint`, so

```
C = Cfg('NIST', 16000, 'amharic') 

config_path = 'amharic_16000.yaml'
yaml = YAML(typ='safe')
with open(config_path) as f:
    params = yaml.load(f)

params['model']['train_ds']['manifest_filepath'] = f'{C.build_dir}/train_manifest.json'
params['model']['validation_ds']['manifest_filepath'] = f'{C.build_dir}/test_manifest.json'

model = nemo_asr.models.EncDecCTCModel.from_pretrained(model_name="QuartzNet15x5Base-En")
model.change_vocabulary(new_vocabulary=params['labels'])
model.setup_optimization(optim_config=DictConfig(params['model']['optim']))
model.setup_training_data(train_data_config=params['model']['train_ds'])
model.setup_validation_data(val_data_config=params['model']['validation_ds'])

model.load_from_checkpoint('foo.ckpt')
```


Some comments/answers to your questions.

> The batch size (best I can do on 2080TI is 12)
The max duration (33 seconds up from 16.7 in the original)

- If you can, for training, I would recommend either splitting longer fragments or discarding them (if there are very few of them). The maximum audio duration affects GPU RAM consumption a lot. If you can fit batch size of 12 with (actual) max duration of 33, then you should be able to fit batch size of 2*12 with (actual) max duration of 33/2=16.5. If you just set max duration to X in datalayer config it will drop anything longer than that and will print stats on how much of data was dropped.
- Additional trick to increase "algorithmic" batch size is to set ```accumulate_grad_batches``` to a small integer more than 1. https://github.com/PyTorchLightning/pytorch-lightning/blob/master/pytorch_lightning/trainer/trainer.py#L105 


>> ```model.load_from_checkpoint('foo.ckpt')``` 
- Sorry, this is confusing but it will not affect ```model``` object instance because ```load_from_checkpoint``` is a class method: https://github.com/PyTorchLightning/pytorch-lightning/blob/8f2bba10e198e42b9fcc3656719c571a404230a0/pytorch_lightning/core/saving.py#L53 

try ```model = model.load_from_checkpoint('foo.ckpt')```


oh, and definitely try a smaller learning rate (0.0015) when fine-tuning. Overwriting it from command line should work for your script: ```python yourscript.py ... model.optim.lr=0.0015 ...```.
 

My model is loaded with

`model = nemo_asr.models.EncDecCTCModel.load_from_checkpoint(checkpt_filename)`

This doesn't seem to work inside a Jupyter notebook:

`model.optim.lr=0.0015`

To change `max_duration`, and `lr`, I change these in the YAML file and then do the following, it is the easiest way?
```
config_path = 'amharic_16000.yaml'
yaml = YAML(typ='safe')
with open(config_path) as f:
    params = yaml.load(f)
train_manifest=f'{C.build_dir}/train_manifest.json'
test_manifest=f'{C.build_dir}/test_manifest.json'
params['model']['train_ds']['manifest_filepath'] = train_manifest
params['model']['validation_ds']['manifest_filepath'] = test_manifest

model.setup_training_data(train_data_config=params['model']['train_ds'])
model.setup_validation_data(val_data_config=params['model']['validation_ds'])
model.setup_optimization(optim_config=DictConfig(params['model']['optim']))
```

**NOTE:** I made it 33 assuming that there might be some long-term dependencies for longer sentences that the model learns.  Are you saying it doesn't keep more weights for a longer sequence, i.e. same # of weights if max_duration=1 versus max_duration=33?  If so, how do you capture long-range dependencies in sentence structure?

**NOTE2:** There doesn't seem to be any kind of adaptive scheduler, the learning rate stays pretty much constant over 200 epochs on 10,000 samples per epoch.  Is that correct?  Here is the config:

```
  optim:
    name: novograd
    # cls: nemo.core.optim.optimizers.Novograd
    lr: .0015
    # optimizer arguments
    betas: [0.8, 0.5]
    weight_decay: 0.001

    # scheduler setup
    sched:
      name: CosineAnnealing

      # pytorch lightning args
      monitor: val_loss
      reduce_on_plateau: false

      # Scheduler params
      warmup_steps: null
      warmup_ratio: null
      min_lr: 0.0
      last_epoch: -1
```