How can I resume the training of a SpeakerNet?
State: closed | Pull request: no
Labels: name: bug

name: stale

Hi,

How can I resume the training of a SpeakerNet?

I already set `resume_if_exists: True `. 

But get error:

`nemo.utils.exp_manager.NotFoundError: There was no checkpoint folder at checkpoint_dir :exp/SpeakerNet/SpeakerNet/checkpoints. Cannot resume.`

I also set `create_checkpoint_callback: True`, I thought this would generate a checkpoint, but in reality, it did not.

This is my exp folder architecture:

- SpeakerNet
  -  2023-11-29_19-21-04
     - cmd-args.log 
     - events.out.tfevents.1701256897.biic188.12314.0
     - git-info.log
     - hparams.yaml
     - labels.txt
     - lightning_logs.txt
     - nemo_error_log.txt
     - nemo_log_globalrank-0_localrank-0.txt
  - spkr.nemo
 
Tool version
python 3.10.13
NeMo 1.20.0
pytorch-lightning 1.9.4
torch 2.1.0

looks like there are no existing checkpoints for it to restore from. Hence you also need to add `ignore_no_checkpoints` option as well to ignore when there are no checkpoints. Note that checkpoints need to be in `ckpt` format. 

Hi, @nithinraok 

I trained for one epoch using [speaker_reco.py](https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/recognition/speaker_reco.py) with [SpeakerNet_verification_3x2x256.yaml](https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/recognition/conf/SpeakerNet_verification_3x2x256.yaml). I thought `create_checkpoint_callback` in [SpeakerNet_verification_3x2x256.yaml](https://github.com/NVIDIA/NeMo/blob/274a21b492f337702f83f7b82abca338d6c9c09e/examples/speaker_tasks/recognition/conf/SpeakerNet_verification_3x2x256.yaml#L133C3-L133C29) would generate checkpoints every epoch, but if not, how do I generate checkpoints?

Can you switch to https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/recognition/conf/titanet-small.yaml 
You need to enable checkpointing as mentioned here: https://github.com/NVIDIA/NeMo/blob/d81ea6a3f786b46cd204d41d5dc18268763f401c/examples/speaker_tasks/recognition/conf/titanet-small.yaml#L172 

try titanet configs. 

Hi, @nithinraok 

I try to switch to [titanet-small.yaml](https://github.com/NVIDIA/NeMo/blob/main/examples/speaker_tasks/recognition/conf/titanet-small.yaml), but got an error: `TypeError: 'NoneType' object is not iterable`

This is full error message:

```
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
[NeMo I 2024-01-09 17:31:12 exp_manager:374] Experiments will be logged at exp/titanet_small/TitaNet-S/2024-01-09_17-31-12
[NeMo I 2024-01-09 17:31:12 exp_manager:797] TensorboardLogger has been set up
[NeMo I 2024-01-09 17:31:24 collections:301] Filtered duration for loading collection is  0.00 hours.
[NeMo I 2024-01-09 17:31:24 collections:302] Dataset loaded with 1240651 items, total duration of  2700.55 hours.
[NeMo I 2024-01-09 17:31:24 collections:304] # 1240651 files loaded accounting to # 7205 labels
[NeMo W 2024-01-09 17:31:24 label_models:187] Total number of 7205 found in all the manifest files.
Error executing job with overrides: []
Traceback (most recent call last):
  File "/mnt/HDD/HDD2/DTDwind/NeMo/examples/speaker_tasks/recognition/speaker_reco.py", line 85, in <module>
    main()
  File "/mnt/HDD/HDD2/DTDwind/NeMo/nemo/core/config/hydra_runner.py", line 105, in wrapper
    _run_hydra(
  File "/home/DTDwind/.conda/envs/nemo/lib/python3.10/site-packages/hydra/_internal/utils.py", line 389, in _run_hydra
    _run_app(
  File "/home/DTDwind/.conda/envs/nemo/lib/python3.10/site-packages/hydra/_internal/utils.py", line 452, in _run_app
    run_and_report(
  File "/home/DTDwind/.conda/envs/nemo/lib/python3.10/site-packages/hydra/_internal/utils.py", line 216, in run_and_report
    raise ex
  File "/home/DTDwind/.conda/envs/nemo/lib/python3.10/site-packages/hydra/_internal/utils.py", line 213, in run_and_report
    return func()
  File "/home/DTDwind/.conda/envs/nemo/lib/python3.10/site-packages/hydra/_internal/utils.py", line 453, in <lambda>
    lambda: hydra.run(
  File "/home/DTDwind/.conda/envs/nemo/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/home/DTDwind/.conda/envs/nemo/lib/python3.10/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/home/DTDwind/.conda/envs/nemo/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/mnt/HDD/HDD2/DTDwind/NeMo/examples/speaker_tasks/recognition/speaker_reco.py", line 61, in main
    speaker_model = EncDecSpeakerLabelModel(cfg=cfg.model, trainer=trainer)
  File "/mnt/HDD/HDD2/DTDwind/NeMo/nemo/collections/asr/models/label_models.py", line 125, in __init__
    super().__init__(cfg=cfg, trainer=trainer)
  File "/mnt/HDD/HDD2/DTDwind/NeMo/nemo/core/classes/modelPT.py", line 143, in __init__
    self.setup_training_data(self._cfg.train_ds)
  File "/mnt/HDD/HDD2/DTDwind/NeMo/nemo/collections/asr/models/label_models.py", line 274, in setup_training_data
    self._train_dl = self.__setup_dataloader_from_config(config=train_data_layer_config)
  File "/mnt/HDD/HDD2/DTDwind/NeMo/nemo/collections/asr/models/label_models.py", line 192, in __setup_dataloader_from_config
    augmentor = process_augmentations(config['augmentor'])
  File "/mnt/HDD/HDD2/DTDwind/NeMo/nemo/collections/asr/parts/preprocessing/perturb.py", line 1230, in process_augmentations
    augmentation = augmentation_class(**augment_kwargs)
  File "/mnt/HDD/HDD2/DTDwind/NeMo/nemo/collections/asr/parts/preprocessing/perturb.py", line 462, in __init__
    self._manifest = collections.ASRAudioText(manifest_path, parser=parsers.make_parser([]), index_by_file_id=True)
  File "/mnt/HDD/HDD2/DTDwind/NeMo/nemo/collections/common/parts/preprocessing/collections.py", line 223, in __init__
    for item in manifest.item_iter(manifests_files):
  File "/mnt/HDD/HDD2/DTDwind/NeMo/nemo/collections/common/parts/preprocessing/manifest.py", line 73, in item_iter
    for manifest_file in manifests_files:
TypeError: 'NoneType' object is not iterable

```

I don't understand why I'm getting this error. The message indicates that it should have successfully retrieved my dataset, so manifest_files should not be NoneType.

Looks like your noise augmentation manifest file is null, you may remove the noise augmentation from config or add the noise manifest 

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.