RIR and noise augmentation for ASR Quartznet
State: closed | Pull request: no
Labels: name: bug

Hi all,
I'm still having trouble implementing RIR in my ASR model after i solved [this](https://github.com/NVIDIA/NeMo/issues/1648) problem.
Let's assume i have alredy created manifest file for both noise and rir aug. Let's assume also that i want to implement augmentation with implicit definition, as suggested in [tutorial n.5](https://github.com/NVIDIA/NeMo/blob/main/tutorials/asr/05_Online_Noise_Augmentation.ipynb). 
With this in mind, i created the nested dictionary:

```
audio_augmentations = dict(
        rir_noise_aug = dict(
        prob=0.5,
        rir_manifest_path = 'rir_manifest.json',
        rir_prob = 0.5,
        noise_manifest_paths = 'noise_manifest.json',
        min_snr_db = [0,0],
        max_snr_db = [30,30],
        bg_noise_manifest_paths = 'noise_manifest.json',
        bg_min_snr_db = [10,10],
        bg_max_snr_db = [40,40]
    )
)

```
And supplied the 'augmentor' to the model.train_ds config:

`config.model.train_ds.augmentor = audio_augmentations`

But when i go for:

`asr_model.setup_training_data(train_data_config=config.model.train_ds)`

It returns an error:

```
TypeError                                 Traceback (most recent call last)
<ipython-input-29-e364e6754a27> in <module>
----> 1 asr_model.setup_training_data(train_data_config=config.model.train_ds)
      2 asr_model.setup_validation_data(val_data_config=config.model.validation_ds)
      3 asr_model.set_trainer(trainer)
      4 asr_model.setup_optimization(optim_config=config.model.optim)

/opt/conda/lib/python3.6/site-packages/nemo/collections/asr/models/ctc_models.py in setup_training_data(self, train_data_config)
    326         self._update_dataset_config(dataset_name='train', config=train_data_config)
    327 
--> 328         self._train_dl = self._setup_dataloader_from_config(config=train_data_config)
    329 
    330         # Need to set this because if using an IterableDataset, the length of the dataloader is the total number

/opt/conda/lib/python3.6/site-packages/nemo/collections/asr/models/ctc_models.py in _setup_dataloader_from_config(self, config)
    224     def _setup_dataloader_from_config(self, config: Optional[Dict]):
    225         if 'augmentor' in config:
--> 226             augmentor = process_augmentations(config['augmentor'])
    227         else:
    228             augmentor = None

/opt/conda/lib/python3.6/site-packages/nemo/collections/asr/parts/perturb.py in process_augmentations(augmenter)
    753 
    754             try:
--> 755                 augmentation = perturbation_types[augment_name](**augment_kwargs)
    756                 augmentations.append([prob, augmentation])
    757             except KeyError:

/opt/conda/lib/python3.6/site-packages/nemo/collections/asr/parts/perturb.py in __init__(self, rir_manifest_path, rir_prob, noise_manifest_paths, min_snr_db, max_snr_db, rir_tar_filepaths, rir_shuffle_n, noise_tar_filepaths, apply_noise_rir, orig_sample_rate, max_additions, max_duration, bg_noise_manifest_paths, bg_min_snr_db, bg_max_snr_db, bg_noise_tar_filepaths, bg_orig_sample_rate)
    508                     min_snr_db=min_snr_db[i],
    509                     max_snr_db=max_snr_db[i],
--> 510                     audio_tar_filepaths=noise_tar_filepaths[i],
    511                     orig_sr=orig_sr,
    512                 )

TypeError: 'NoneType' object is not subscriptable
```

Actually i can't figure it out where is the problem. I tried to search in perturb.py but i guess i'am too naive to find the solution. It seems the problem relies in noise_tar_filepath. But my rir and noise are not tarred!! 
Thanks in advance

@lodm94 You are right, there seems to be an issue with the default value of noise_tar_paths. Until we push a fix, as a workaround you could change your dict to the following:

audio_augmentations = dict(
        rir_noise_aug = dict(
        prob=0.5,
        rir_manifest_path = 'rir_manifest.json',
        rir_prob = 0.5,
        noise_manifest_paths = ['noise_manifest.json'],
        min_snr_db = [0],
        max_snr_db = [30],
        bg_noise_manifest_paths = ['noise_manifest.json'],
        bg_min_snr_db = [10],
        bg_max_snr_db = [40],
        noise_tar_filepaths=[None], 
        bg_noise_tar_filepaths=[None]
    )
)


Hi @jbalam-nv thanks for the feedback. That was exactly how i managed to un the augmentation, i just forgot to reply here but it works.
Anyway things went somehow bad. Take a look here:
![Immagine 2021-02-03 105243](https://user-images.githubusercontent.com/70609842/106729992-5feee880-660e-11eb-997a-bc69a0f14cc2.png)
![Immagine 2021-02-03 105318](https://user-images.githubusercontent.com/70609842/106730023-69785080-660e-11eb-89a8-0c1c4a0055e2.png)

Orange line: model from pre-trained fine tuned with spec-augment only and 90h of data.
Blue line: model from pre-trained fine tuned with spec-augment, rir_and_noise and 240h of data.
Is there some tips you could give me to improve wer? Best result i got was 15.04% (ORAGE LINE)
If details are needed i will supply asap.
Thanks

@lodm94 I would suggest reducing augmentation probability (0.1 or 0.2) . What is the learning rate for your fine-tuning?

@jbalam-nv  tomorrow i will run an experiment with reduced probs (either noise and rir). Here's my optim dict:

```
optim:
    name: novograd
    # _target_: nemo.core.optim.optimizers.Novograd
    lr: .001
    betas: [0.9, 0.25]
    weight_decay: 0.001

    sched:
      name: CosineAnnealing
      warmup_steps: null
      warmup_ratio: 0.12
      min_lr: 0.0
      last_epoch: -1
```

Sorry i accidentally closed the issue. One more question about optim: should i try a different min_lr? Should training stop when i reaach the min_lr or maybe it could be worth to let the model train for some epochs at costant min_lr?

Hi @jbalam-nv i tried with reduced probs as you said. Training was going well but then

![image](https://user-images.githubusercontent.com/70609842/107620940-c8187c80-6c55-11eb-8f26-48d73bd91b93.png)
![image](https://user-images.githubusercontent.com/70609842/107621030-ef6f4980-6c55-11eb-823a-20007ef6836c.png)

during validation check at the end of the epoch, i can see only the reference sentence. Decode is blank.
During training, instead, i can see the decoded sentence and it is perfect, so perfect that i am afraid model is overfitting. Look that u-shape in val_loss.  

Hi All, Thanks for this thread @titu1994  and @jbalam-nv. I followed these instructions and could successfully train a musan noise augmented nemo ASR model. However, I have few queries:

1. Where does the noise augmented files are stored? Or Are these noise augmented files are simply produced on the fly, as and when needed, and discarded once the job is done (without retaining them)?

2. When I use augmentors during training, does it include original clean files also for training along-with noise augmented files? Or it only uses noise augmented files for training?

3. For the kind of experiments & datasets that we used, we had expected lower WERs  with noise augmentation compared to WERs obtained on clean datasets. However the obtained results are other way (noise augmented datasets have 1% higher WER than clean data). This made us to suspect the implementation part of nemo asr training. I have used exactly the same config of **audio_augmentations** mentioned by @jbalam-nv.  There were no errors during training as well. Do you still feel there might be some implementation gaps (which has led to lower WER). Any suggestions on this would be very helpful.

4. I have used musan free sound noise and RIR. Is it fine? or any suggestions on this?

Thanks
Manjunath


Hi  @titu1994  or @jbalam-nv , Could you please suggest me on my above query. Thanks

Do not reopen discussing on a months old thread. Create a new one.