RuntimeError: stft input and window must be on the same device but got self on cuda:1 and window on cuda:0
State: closed | Pull request: no
Labels: name: bug

name: stale

>>> 
>>> trainer = pl.Trainer(strategy='dp', gpus=4, max_epochs=1)
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
>>> trainer.fit(quartznet)
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/overrides/data_parallel.py", line 64, in forward
    output = super().forward(*inputs, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 93, in forward
    return self.module.validation_step(*inputs, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/nemo/collections/asr/models/ctc_models.py", line 544, in validation_step
    log_probs, encoded_len, predictions = self.forward(input_signal=signal, input_signal_length=signal_len)
  File "/usr/local/lib/python3.8/site-packages/nemo/core/classes/common.py", line 963, in __call__
    outputs = wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/nemo/collections/asr/models/ctc_models.py", line 470, in forward
    processed_signal, processed_signal_length = self.preprocessor(
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/nemo/core/classes/common.py", line 963, in __call__
    outputs = wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/nemo/collections/asr/modules/audio_preprocessing.py", line 85, in forward
    processed_signal, processed_length = self.get_features(input_signal, length)
  File "/usr/local/lib/python3.8/site-packages/nemo/collections/asr/modules/audio_preprocessing.py", line 268, in get_features
    return self.featurizer(input_signal, length)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/nemo/collections/asr/parts/preprocessing/features.py", line 327, in forward
    x = self.stft(x)
  File "/usr/local/lib/python3.8/site-packages/nemo/collections/asr/parts/preprocessing/features.py", line 219, in <lambda>
    self.stft = lambda x: torch.stft(
  File "/usr/local/lib/python3.8/site-packages/torch/functional.py", line 606, in stft
    return _VF.stft(input, n_fft, hop_length, win_length, window,  # type: ignore[attr-defined]
RuntimeError: stft input and window must be on the same device but got self on cuda:1 and window on cuda:0