Unable to train with more than 0 dataloader workers
State: closed | Pull request: no
Labels: name: bug

name: stale

**Describe the bug**

When running the tutorial to train a FastPitch model with an adapter (https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_Adapter_Finetuning.ipynb), I get the error `EOFError: Caught EOFError in DataLoader worker process 0` when starting the step that does the model training. Following a similar issue I was able to avoid the error by setting the number of dataloader workers to 0 which is not ideal for training speed.

**Steps/Code to reproduce bug**

Running the tutorial as is should produce the error

**Expected behavior**

Training should start regardless of the number of workers

**Environment overview (please complete the following information)**

 - Environment location: EC2
 - Method of NeMo install: version 1.23.0 pip install

**Environment details**

If NVIDIA docker image is used you don't need to specify these.
Otherwise, please provide:
- OS version - ubuntu 20
- PyTorch version 2.2.1
- Python version 3.10.13

**Additional context**

full traceback below
```

  | Name                | Type                              | Params
--------------------------------------------------------------------------
0 | mel_loss_fn         | MelLoss                           | 0     
1 | pitch_loss_fn       | PitchLoss                         | 0     
2 | duration_loss_fn    | DurationLoss                      | 0     
3 | energy_loss_fn      | EnergyLoss                        | 0     
4 | aligner             | AlignmentEncoderAdapter           | 1.6 M 
5 | forward_sum_loss_fn | ForwardSumLoss                    | 0     
6 | bin_loss_fn         | BinLoss                           | 0     
7 | preprocessor        | AudioToMelSpectrogramPreprocessor | 0     
8 | fastpitch           | FastPitchModule                   | 58.0 M
--------------------------------------------------------------------------
4.5 M     Trainable params
53.5 M    Non-trainable params
58.0 M    Total params
231.999   Total estimated model params size (MB)
terminate called after throwing an instance of 'c10::Error'                                                                                                      
  what():  CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /opt/conda/conda-bld/pytorch_1708025831440/work/c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9049bd0d87 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f9049b8175f in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f9049ca28a8 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::cuda::ExchangeDevice(int) + 0x39 (0x7f9049ca2c49 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0xf9dedc (0x7f904ac72edc in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x541280 (0x7f90a6c33280 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x649bf (0x7f9049bb59bf in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x7f9049baec8b in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f9049baee39 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: <unknown function> + 0x803c08 (0x7f90a6ef5c08 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7f90a6ef5f86 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /opt/conda/conda-bld/pytorch_1708025831440/work/c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9049bd0d87 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f9049b8175f in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f9049ca28a8 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::cuda::ExchangeDevice(int) + 0x39 (0x7f9049ca2c49 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0xf9dedc (0x7f904ac72edc in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x541280 (0x7f90a6c33280 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x649bf (0x7f9049bb59bf in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x7f9049baec8b in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f9049baee39 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: <unknown function> + 0x803c08 (0x7f90a6ef5c08 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7f90a6ef5f86 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /opt/conda/conda-bld/pytorch_1708025831440/work/c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9049bd0d87 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f9049b8175f in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f9049ca28a8 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::cuda::ExchangeDevice(int) + 0x39 (0x7f9049ca2c49 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0xf9dedc (0x7f904ac72edc in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x541280 (0x7f90a6c33280 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x649bf (0x7f9049bb59bf in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x7f9049baec8b in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f9049baee39 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: <unknown function> + 0x803c08 (0x7f90a6ef5c08 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7f90a6ef5f86 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /opt/conda/conda-bld/pytorch_1708025831440/work/c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9049bd0d87 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f9049b8175f in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f9049ca28a8 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::cuda::ExchangeDevice(int) + 0x39 (0x7f9049ca2c49 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0xf9dedc (0x7f904ac72edc in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x541280 (0x7f90a6c33280 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x649bf (0x7f9049bb59bf in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x7f9049baec8b in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f9049baee39 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: <unknown function> + 0x803c08 (0x7f90a6ef5c08 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7f90a6ef5f86 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /opt/conda/conda-bld/pytorch_1708025831440/work/c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9049bd0d87 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f9049b8175f in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f9049ca28a8 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::cuda::ExchangeDevice(int) + 0x39 (0x7f9049ca2c49 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0xf9dedc (0x7f904ac72edc in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x541280 (0x7f90a6c33280 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x649bf (0x7f9049bb59bf in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x7f9049baec8b in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f9049baee39 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: <unknown function> + 0x803c08 (0x7f90a6ef5c08 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7f90a6ef5f86 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>

Epoch 0:   0%|                                                                                                                            | 0/37 [00:00<?, ?it/s]terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /opt/conda/conda-bld/pytorch_1708025831440/work/c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9049bd0d87 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f9049b8175f in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f9049ca28a8 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::cuda::ExchangeDevice(int) + 0x39 (0x7f9049ca2c49 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0xf9dedc (0x7f904ac72edc in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x541280 (0x7f90a6c33280 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x649bf (0x7f9049bb59bf in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x7f9049baec8b in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f9049baee39 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: <unknown function> + 0x803c08 (0x7f90a6ef5c08 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7f90a6ef5f86 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /opt/conda/conda-bld/pytorch_1708025831440/work/c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9049bd0d87 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f9049b8175f in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f9049ca28a8 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::cuda::ExchangeDevice(int) + 0x39 (0x7f9049ca2c49 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0xf9dedc (0x7f904ac72edc in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x541280 (0x7f90a6c33280 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x649bf (0x7f9049bb59bf in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x7f9049baec8b in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f9049baee39 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: <unknown function> + 0x803c08 (0x7f90a6ef5c08 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7f90a6ef5f86 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: initialization error
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at /opt/conda/conda-bld/pytorch_1708025831440/work/c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f9049bd0d87 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f9049b8175f in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f9049ca28a8 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10::cuda::ExchangeDevice(int) + 0x39 (0x7f9049ca2c49 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #4: <unknown function> + 0xf9dedc (0x7f904ac72edc in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0x541280 (0x7f90a6c33280 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x649bf (0x7f9049bb59bf in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x21b (0x7f9049baec8b in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f9049baee39 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: <unknown function> + 0x803c08 (0x7f90a6ef5c08 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #10: THPVariable_subclass_dealloc(_object*) + 0x2f6 (0x7f90a6ef5f86 in /opt/conda/envs/nemo/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>

Error executing job with overrides: ['+init_from_nemo_model=/home/ubuntu/Models/tts_fastpitch_speaker_adapter_ipa_vtrainable_v1.0/tts_en_fastpitch_multispeaker_adapter_pretrain_ipa.nemo', 'train_dataset=/home/ubuntu/Data/TTS_1hr_1/output/manifests/runInstance_train_manifest.json', 'validation_datasets=/home/ubuntu/Data/TTS_1hr_1/output/manifests/runInstance_val_manifest.json', "sup_data_types=['align_prior_matrix', 'pitch', 'energy']", 'sup_data_path=/home/ubuntu/ModelTraining/tts/data-1hr-1/NeMoTTS_sup_data', 'pitch_mean=166.378662109375', 'pitch_std=61.548152923583984', 'model.speaker_encoder.precomputed_embedding_dim=384', '~model.speaker_encoder.lookup_module', '~model.speaker_encoder.gst_module', 'model.train_ds.dataloader_params.batch_size=8', 'model.validation_ds.dataloader_params.batch_size=8', '+model.text_tokenizer.add_blank_at=True', 'model.optim.name=adam', 'model.optim.lr=0.001', 'model.optim.sched.warmup_steps=10', '+model.optim.sched.min_lr=0.0001', 'exp_manager.exp_dir=/home/ubuntu/ModelTraining/tts/data-1hr-1/NeMoTTS_logs/unfreeze_aligner', '+exp_manager.create_wandb_logger=False', '+exp_manager.wandb_logger_kwargs.name=tutorial-FastPitch-finetune-adaptation', '+exp_manager.wandb_logger_kwargs.project=NeMo', '+exp_manager.checkpoint_callback_params.save_top_k=-1', 'trainer.max_epochs=200', 'trainer.check_val_every_n_epoch=10', 'trainer.log_every_n_steps=1', 'trainer.devices=1', 'trainer.strategy=ddp', 'trainer.precision=32', 'model.train_ds.dataloader_params.num_workers=15', 'model.validation_ds.dataloader_params.num_workers=15', 'model.unfreeze_aligner=true']
Traceback (most recent call last):
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1133, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/opt/conda/envs/nemo/lib/python3.10/queue.py", line 180, in get
    self.not_empty.wait(remaining)
  File "/opt/conda/envs/nemo/lib/python3.10/threading.py", line 324, in wait
    gotit = waiter.acquire(True, timeout)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 293316) is killed by signal: Aborted. 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ubuntu/ModelTraining/NeMo/examples/tts/fastpitch_finetune_adapters.py", line 139, in main
    trainer.fit(model)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 544, in fit
    call._call_and_handle_interrupt(
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 105, in launch
    return function(*args, **kwargs)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 580, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 987, in _run
    results = self._run_stage()
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1033, in _run_stage
    self.fit_loop.run()
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 205, in run
    self.advance()
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/fit_loop.py", line 363, in advance
    self.epoch_loop.run(self._data_fetcher)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 140, in run
    self.advance(data_fetcher)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 212, in advance
    batch, _, __ = next(data_fetcher)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/fetchers.py", line 133, in __next__
    batch = super().__next__()
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/loops/fetchers.py", line 60, in __next__
    batch = next(self.iterator)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/utilities/combined_loader.py", line 341, in __next__
    out = next(self._iterator)
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/pytorch_lightning/utilities/combined_loader.py", line 78, in __next__
    out[i] = next(self.iterators[i])
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 631, in __next__
    data = self._next_data()
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1329, in _next_data
    idx, data = self._get_data()
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1285, in _get_data
    success, data = self._try_get_data()
  File "/opt/conda/envs/nemo/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 1146, in _try_get_data
    raise RuntimeError(f'DataLoader worker (pid(s) {pids_str}) exited unexpectedly') from e
RuntimeError: DataLoader worker (pid(s) 293316) exited unexpectedly
```

This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

This issue was closed because it has been inactive for 7 days since being marked as stale.