Repository: pytorch
Issue: Using Dataparallel with multi input error.  Input and parameter tensors are not at the same device, found input tensor at cpu and parameter tensor at cuda:0
Labels: triaged,module: data parallel

File "train_pytorch.py", line 169, in <module>
    output = net([train1[0], train1[1], train1[2], train1[3], train1[4]])
  File "/gpfs/u/home/EXPS/EXPSxyli/scratch/miniconda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gpfs/u/home/EXPS/EXPSxyli/scratch/miniconda/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/gpfs/u/home/EXPS/EXPSxyli/scratch/miniconda/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/gpfs/u/home/EXPS/EXPSxyli/scratch/miniconda/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/gpfs/u/home/EXPS/EXPSxyli/scratch/miniconda/lib/python3.7/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/gpfs/u/home/EXPS/EXPSxyli/scratch/miniconda/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/gpfs/u/home/EXPS/EXPSxyli/scratch/miniconda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gpfs/u/scratch/EXPS/EXPSxyli/AutoAI_Report_Gen/models/codegnngru.py", line 132, in forward
    tencout, tstate_h = self.gru1(tde)
  File "/gpfs/u/home/EXPS/EXPSxyli/scratch/miniconda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/gpfs/u/home/EXPS/EXPSxyli/scratch/miniconda/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 727, in forward
    self.dropout, self.training, self.bidirectional, self.batch_first)
RuntimeError: Input and parameter tensors are not at the same device, found input tensor at cpu and parameter tensor at cuda:0


code in train_pytorch.py:
```python
train1 = train_batch[0]  ##tdatseqs, comseqs, smlnodes, wedge_1
            train2 = train_batch[1]  ##comouts
            train2 = np.array(train2)
            train2 = torch.from_numpy(train2)
            train2 = train2.type(torch.LongTensor)
            train2 = train2.to(device)
            for i in range(5):
                train1[i] = np.array(train1[i])
                train1[i] = torch.from_numpy(train1[i])
                train1[i] = train1[i].type(torch.LongTensor)
                train1[i] = train1[i].to(device)
            # forward propogation
            output = net([train1[0], train1[1], train1[2], train1[3], train1[4]])
```


code in model build and use dataparallel:
```python
def create_model(modeltype, config):
    mdl = None
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    print('cuda' if torch.cuda.is_available() else 'cpu')
    if modeltype == 'codegnngru':
        mdl = CodeGNNGRU(config).to(device)
        mdl = torch.nn.DataParallel(mdl)
```

init gpu
```python
def init_tf(gpu):
    import os
    os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
    os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2,3,4,5"
```