RuntimeError: arguments are located on different GPUs
State: closed | Pull request: no
Labels: name: bug

**Describe the bug**

During speech_to_text.py on my own data set training while using FilterbankFeatures, 
at training step.
I get the error: RuntimeError: arguments are located on different GPUs
at line 358 in /nemo/collections/ast/parts/features.py 
x = torch.matmul(self.fb.to(x.dtype), x)

**Steps/Code to reproduce bug**
python speeh_to_text.py 
model.train_ds.manifest_filepath=<my dataset>
model.validation_ds.manifest_filepath=<my dataset>
trainer.gpus=3
dataset.num_workers=10

**Environment overview (please complete the following information)**

 - Environment location: Nemo Docker container

 - Method of NeMo install: cloning NeMo main branch

**Environment details**

- PyTorch 1.6.0
- Python 3.6.7

**Additional context**

Nvidia V100


I have the same issue.
**Same Environment and GPUs**

- python 3.6.10
- Pytorch: 1.7.0a0+8deb4fe
- pytorch-lightning==1.0.8
- pytorch-lightning-bolts==0.2.5
- pytorch-transformers==1.1.0

> I have the same issue.
> **Same Environment and GPUs**
> 
> * python 3.6.10
> * Pytorch: 1.7.0a0+8deb4fe
> * pytorch-lightning==1.0.8
> * pytorch-lightning-bolts==0.2.5
> * pytorch-transformers==1.1.0

I was able to get it running by running the command from the git repo root directory. 

python examples/asr/speech_to_text.py \
	model.train_ds.manifest_filepath="<path to json>json" \
	model.validation_ds.manifest_filepath="<path to json>json" \
	hydra.run.dir="." \
	trainer.gpus=<> \
	trainer.max_epochs=XX \
	+model.optim.args.betas=[0.8,0.5] \
	+model.optim.args.weight_decay=0.0001 \
	model.train_ds.batch_size=196 \
	model.validation_ds.batch_size=196


> > I have the same issue.
> > **Same Environment and GPUs**
> > 
> > * python 3.6.10
> > * Pytorch: 1.7.0a0+8deb4fe
> > * pytorch-lightning==1.0.8
> > * pytorch-lightning-bolts==0.2.5
> > * pytorch-transformers==1.1.0
> 
> I was able to get it running by running the command from the git repo root directory.
> 
> python examples/asr/speech_to_text.py
> model.train_ds.manifest_filepath="json"
> model.validation_ds.manifest_filepath="json"
> hydra.run.dir="."
> trainer.gpus=<>
> trainer.max_epochs=XX
> +model.optim.args.betas=[0.8,0.5]
> +model.optim.args.weight_decay=0.0001
> model.train_ds.batch_size=196
> model.validation_ds.batch_size=196

Interesting, ill try it as well, thank you ! Although 
How could exactly it might effect something I still cant understand.

It seems like it doesnt work, I have done cd to the repo root dir, from there Ive run the command simillar to yours.
should note that i use PYTHONPATH='....' beforehand to direct the nemo import nemo.
any other suggestions ?


Also i should note that the problem doesnt occurs  when using nemo versions from october. 

I'm not sure at this point. I just ran this from the /NeMo directory  in the container. I also cd to examples/asr/ and then removed the "examples/asr/" from the python command and ran it again, they both worked. I checked my repo and I am 14 commits behind right now. 

```
python examples/asr/speech_to_text.py \
	model.train_ds.manifest_filepath="" \
	model.validation_ds.manifest_filepath="" \
	hydra.run.dir="." \
	trainer.gpus=2 \
	trainer.max_epochs=1 \
	+model.optim.args.betas=[0.8,0.5] \
	+model.optim.args.weight_decay=0.0001 \
	model.train_ds.batch_size=196 \
	model.validation_ds.batch_size=196 \
	+model.train_ds.num_workers=4
```

Since you are running in a container, are you using the nemo version or the nvcr version and pip installed everything? I went with the latter. Essentially, if you use the nvcr contianer and then run a couple of the Jupyter tutorials from he repo, it will install everything you need. Also, what driver do you have? The latest NVIDIA containers were built with the 455 driver and though you will see (nvidia-smi) the GPUs from inside the container, you won't be able to access them with the Cuda library. 

Just stating some observations that in the hopes something helps.


So ive managed to work around the problem via using the ddp instead of dp with correct calibration of train and validation ds num of workers