[Question] Fine-tuning Quartznet not going as expected, looking for advice/best practices
State: closed | Pull request: no

I'm trying to do some fine-tuning on Quartznet15-5. Specifically, I'm trying to see if I do fine-tuning on Switchboard how much I can increase the performance on Switchboard and how that affects the other datasets it was trained with.
I've simply split swb into an 80/10/10 train/eval/test split.

I read through the paper on fine-tuning and based hyperparameters for the optimizer on that. What I'm running into is that, universally, it becomes substantially worse as training begins, then trends back downwards towards where it started. The un-tuned quartznet is getting around ~12% WER on val/test, and the fine-tuning is typically ending worse than that. Image and optimizer below. Training for more epochs ended up stretching out the curve rather than achieving a lower validation WER.

In addition to it getting substantially worse at first, I'm also surprised that I can't seem to fine-tune it to perform better on switchboard specifically - since it was trained on several other datasets, I figured it would be possible to fine-tune to the sorts of speech we get for switchboard. 

Does anyone have advice on what might be happening here, and other things to try to accomplish what I want to do?

![image](https://user-images.githubusercontent.com/16962414/100906021-b2891880-3496-11eb-8ad7-147dc62b63b8.png)

Here's an example of an optimizer I tried
new_opt = {
    "betas": [0.95, 0.25],
    "lr": 0.001, # 0.001 caused increase in error, decrease LR suggested
    "name": "novograd",
    "sched": {
        "last_epoch": -1,
        "min_lr": 0.0,
        "monitor": "val_loss",
        "name": "CosineAnnealing",
        "reduce_on_plateau": False,
        "warmup_ratio": 0.12,
        "warmup_steps": None,
    },
    "weight_decay": 0.001,
}

Using Docker v1.0.0b1 right now.



can you please share: (1) your python script, (2) yaml config and (3) command line args? I suspect smaller LR is not actually set.
Also, 1.0.0b2 is out - so I'd recommend you use it, but it shouldn't be the root cause.

@gtcooke94 which Quartznet15x5 are you using to finetune? Some of our models have been trained with complete Switcboard data. That may explain this discrepancy. Your dev and eval set may already be in the training data of the base model.

fine-tuning script, configs are just written in here directly, simply called with `python fine-tuning.py`: https://gist.github.com/gtcooke94/89d933cda31ee75fec3c32e295b5b718
Using LR=0.001, have tried with 0.0005 and varying warmup rates as well and see a similar behavior (albeit less exaggerated).

@jbalam-nv I'm using 15x5-base-en. I know there's definitely leakage going on with using parts of switchboard as validation/test data, but my hope was that I'd be able to tune more aggressively towards conversational speech specifically (I don't care about narrated speech). My thinking being, if this has trained on narrated speech as well, perhaps by training more on switchboard specifically it'll learn to better handle conversational speech. Does that make sense, or am I making bad assumptions that this will be able to learn more on switchboard?

>> https://gist.github.com/gtcooke94/89d933cda31ee75fec3c32e295b5b718
- this looks fine to me. I don't remember though is Switchboard initially 8Khz or 16Khz? Do you know the actual sample rate of the wav files you are using for training?

I've upsampled them to all to 16khz wav files before doing anything with them.
I sort of just assumed that's what was done at training and Quartznet15x5 wants to run with 16khz?

@gtcooke94 In our experiments, we found that this model has enough capacity to learn from multiple domains. I doubt if you can improve performance by using a subset of the original training set. Finetuning with new data in your domain of interest would certainly help but I do not see this experiment helping. The fact that your validation set was already used for training in the base model and you are holding that out while fine-tuning should make WER a little worse. Looks like your other settings are good, so if you have access to any other dataset not used in the base model, you should try that.



Thanks for the discussion and advice, closing this now

Hi, @gtcooke94 I am seeing a similar behavior as yours. FInetuning on another dataset results in a worse WER than using the Quartnzent 15x5-base-en.

@sciai-ai  how are you exporting JasperDecoderForCTC-STEP-81000.pt   JasperEncoder-STEP-81000.pt  trainer-STEP-81000.pt files into .nemo?