problem with fine-tuning th e pre-trained `SpeakerNet` model for speaker recognition
State: closed | Pull request: no

Hi!

As an experiment, I was fine-tuning the pre-trained `SpeakerNet` model with the `an4` dataset plus my own voice recordings. I used the example script `speaker_reco_finetune.py` and changed the configuration file according to the documentation and the colab notebook tutorial. But it appears something is always wrong with setting up the optimizer:

```
Traceback (most recent call last):
  File "/Users/xujinghua/speaker-recognition-nemo/scripts/experimental/fine_tune_an4.py", line 119, in <module>
    main()
  File "/Users/xujinghua/speaker-recognition-nemo/scripts/experimental/fine_tune_an4.py", line 108, in main
    speaker_model.setup_optimization(speaker_model._cfg.optim)
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/nemo/core/classes/modelPT.py", line 603, in setup_optimization
    optimizer=self._optimizer, scheduler_config=scheduler_config, train_dataloader=self._train_dl
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/nemo/core/optim/lr_scheduler.py", line 539, in prepare_lr_scheduler
    max_epochs, accumulate_grad_batches, num_workers, num_samples, batch_size, drop_last,
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/nemo/core/optim/lr_scheduler.py", line 587, in compute_max_steps
    sampler_num_samples = math.ceil(num_samples / num_workers)
ZeroDivisionError: division by zero
```

I tried a few ways to debug but nothing helped I have no clue what is going on wrong. Has anyone got an idea or run into such problems before? Help would be super appreciated!

Cheers,
JX


Hi, again! I just tried with the `hi-mia` dataset and received the same error 

Hi, I just ran `speaker_reco_finetune.py` and everything looks good to me. 
Can you make sure you have set the following params in conf/SpeakerNet_verification_3x2x512.yaml:
train_set
dev_set
decoder.emb_sizes: 256
decoder.num_classes: (num of uniq classes in your train set) 
optim.name: sgd
optim.lr: 0.006 (# make sure to remove betas from args and replace with momentum) 
max_epochs: 10 

And use --pretrained_model="speakerverification_speakernet" while running speaker_reco_finetune.py script(speakerrecognition also works, change pretrained_model name accordingly)

Hi, @nithinraok  Thank you for your reply! I made sure train_set, dev_set, and num_classes all correct, though the default optim.name was novograd and the max_epoch I went with was 5, I am changing them to the numbers you said. Though can I ask how to replace betas with momentum? I did not just find useful documentation for nemo configurations...Thank you again for your help! 

btw I set gpus: 0, since torch.cuda.is_available() returns false (according the colab tutorial), could this be the cause of the problem? Does fine-tuning the pretrained model absolutely require gpu?

```
  optim:
    name: sgd
    lr: .006
    # optimizer arguments
    args:
      name: auto
      weight_decay: 0.001
      momentum: 0.9 # replaced here
```

Yes, I just tried with cpu, looks like optimizer running into an issue. Can you run on a gpu machine meanwhile, while I find out what could have been the cause of it. Thanks. 

Thank you! I will try running on a GPU machine. First I will need to set up a few things and make sure torch.cuda.is_available() returns true. 
btw I just changed the configuration file according to your descriptions with cpu and I get a different error this time
```
Traceback (most recent call last):
  File "/Users/xujinghua/speaker-recognition-nemo/scripts/experimental/fine_tune_an4.py", line 119, in <module>
    main()
  File "/Users/xujinghua/speaker-recognition-nemo/scripts/experimental/fine_tune_an4.py", line 76, in main
    finetune_config = OmegaConf.load(args.finetune_config_file)
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/omegaconf/omegaconf.py", line 254, in load
    obj = yaml.load(f, Loader=get_yaml_loader())
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/yaml/__init__.py", line 114, in load
    return loader.get_single_data()
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/yaml/constructor.py", line 43, in get_single_data
    return self.construct_document(node)
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/yaml/constructor.py", line 47, in construct_document
    data = self.construct_object(node)
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/yaml/constructor.py", line 92, in construct_object
    data = constructor(self, node)
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/omegaconf/_utils.py", line 95, in no_duplicates_constructor
    value = loader.construct_object(value_node, deep=deep)
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/yaml/constructor.py", line 92, in construct_object
    data = constructor(self, node)
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/omegaconf/_utils.py", line 95, in no_duplicates_constructor
    value = loader.construct_object(value_node, deep=deep)
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/yaml/constructor.py", line 92, in construct_object
    data = constructor(self, node)
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/omegaconf/_utils.py", line 95, in no_duplicates_constructor
    value = loader.construct_object(value_node, deep=deep)
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/yaml/constructor.py", line 92, in construct_object
    data = constructor(self, node)
  File "/Users/xujinghua/miniconda3/lib/python3.7/site-packages/omegaconf/_utils.py", line 101, in no_duplicates_constructor
    key_node.start_mark,
yaml.constructor.ConstructorError: while constructing a mapping
  in "/Users/xujinghua/speaker-recognition-nemo/conf/SpeakerNet_recognition_3x2x512.yaml", line 126, column 7
found duplicate key name
  in "/Users/xujinghua/speaker-recognition-nemo/conf/SpeakerNet_recognition_3x2x512.yaml", line 132, column 7
```
Thank you again for your help! I will first try gpu


You are using a key name twice in config file. In the error you posted looks like the key is in line 126 and line 132. 

ah, sorry! that was an oversight, it turned out to be an indentation error in the config file. I corrected the indentation and now it gets the same `ZeroDivisionError` again...

Can you try running locally if possible? Probably something with colab

I was running locally (still training on cpu though), still trying to set up gpu, then I will try again

Hi @nithinraok  I set up cuda, gpu and everything, this time instead of errors it shuts down my computer at the beginning of training...

Not sure, why you are facing such issues. Let's take this step by step. Instead of finetuning are you able to train? 


I just tried it again and now it is working! I am not sure why the computer shut down in the previous tries, probably due to low battery... As a turns out the error were indeed caused by cpu training, I think to fine-tune the pretrained model it is mandatory to train on gpu too (which might be a small problem for mac users who don't have a NVIDIA graphics card :p). Anyways, thank you very much for your help & the great work of building and training the models!

Hi @JINHXu ... To finetune on cpu, you have to make sure to set accelerator to `null` in config, along with gpus=0. This should work 

Hi @nithinraok  I just tried, it works for me too. Thank you again for all your help!