Getting Key Error while finetuning Speaker Recognition model 
State: closed | Pull request: no
Labels: name: bug

name: stale

Hi @nithinraok ,

Please have a look into it.

I am trying to finetune speaker recognition model on open source dataset.
When I set train_manifest_path and dev_manifest_path both as same either train.json or dev.json. Its working fine for me.
When I set train_manifest and dev_manifest as train.jsom and dev.json (two different file) then I get key error in dev_manifest.
I prepared data using data_preparation script from nemo and doing above experiment on opensource data only.

python speaker_reco_finetune.py
Error:
 File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 768, in fit
    self._call_and_handle_interrupt(
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 719, in _call_and_handle_interrupt
    return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
    return function(*args, **kwargs)
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run
    results = self._run_stage()
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1321, in _run_stage
    return self._run_train()
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1343, in _run_train
    self._run_sanity_check()
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1411, in _run_sanity_check
    val_loop.run()
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 154, in advance
    dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 111, in advance
    batch = next(data_fetcher)
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 184, in __next__
    return self.fetching_function()
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 259, in fetching_function
    self._fetch_next_batch(self.dataloader_iter)
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/pytorch_lightning/utilities/fetching.py", line 273, in _fetch_next_batch
    batch = next(iterator)
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 561, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/nasim-pc/Nasim/stt_nemo_1.9.0_venv/lib/python3.8/site-packages/nemo/collections/asr/data/audio_to_label.py", line 314, in __getitem__
    t = torch.tensor(self.label2id[sample.label]).long()
KeyError: 'SV0257'

I have attached error logs and testing_cases that I am assuming.
[https://docs.google.com/spreadsheets/d/1MqSfsFuPtg4LhJgEAl7x9tXfsOrsDIE7/edit?usp=sharing&ouid=102994177770185617606&rtpof=true&sd=true](url)

Training Log:
[https://drive.google.com/file/d/14MdNwZzQI5_RuwfFY8XRMYYtSGwewqz5/view?usp=sharing](url)

I prepared data using python scripts/dataset_processing/get_hi-mia_data.py

My Config file:
[https://drive.google.com/file/d/1ya77Dqqh-wvj0h9ufo9gl7nulHRaaY_4/view?usp=sharing](url)

Installed nemo v1.9.0, torch: 1.10.2+cu102 on my local machine through pip.
OS Version: ubuntu 1.18.04
python: 3.8.0
GPU: Quardro M2000


These two train.json and dev.json should be disjoint but if I am using same file in train and dev manifest then it work else give me key error in dev manifest as I mentioned in error log attachment.


Please help me to resolve this issue.



Thanks