Repository: pytorch
Issue: DataParallel doesn't properly handle kwargs
Labels: module: nn,triaged,module: data parallel

## 🐛 Bug

When DP is given a specified non-tensor kwarg and the batch tensor doesn't have enough elements to scatter to each device, a weird error is thrown:
```py
In [5]: import torch
   ...: import torch.nn as nn
   ...:
   ...: class Net(nn.Module):
   ...:     def forward(self, x, *, delta=2):
   ...:         return x + 2
   ...:
   ...: dpn = nn.DataParallel(Net(), device_ids=(0, 1)).cuda()
   ...: x = torch.randn(1, 3, device='cuda')
   ...: dpn(x)
   ...: dpn(x, delta=3)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-740170232696> in <module>
      9 x = torch.randn(1, 3, device='cuda')
     10 dpn(x)
---> 11 dpn(x, delta=3)

/data/vision/torralba/scratch2/tongzhou/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/data/vision/torralba/scratch2/tongzhou/miniconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
    150             return self.module(*inputs[0], **kwargs[0])
    151         replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
--> 152         outputs = self.parallel_apply(replicas, inputs, kwargs)
    153         return self.gather(outputs, self.output_device)
    154

/data/vision/torralba/scratch2/tongzhou/miniconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in parallel_apply(self, replicas, inputs, kwargs)
    160
    161     def parallel_apply(self, replicas, inputs, kwargs):
--> 162         return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
    163
    164     def gather(self, outputs, output_device):

/data/vision/torralba/scratch2/tongzhou/miniconda3/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py in parallel_apply(modules, inputs, kwargs_tup, devices)
     83         output = results[i]
     84         if isinstance(output, ExceptionWrapper):
---> 85             output.reraise()
     86         outputs.append(output)
     87     return outputs

/data/vision/torralba/scratch2/tongzhou/miniconda3/lib/python3.7/site-packages/torch/_utils.py in reraise(self)
    392             # (https://bugs.python.org/issue2651), so we work around it.
    393             msg = KeyErrorMessage(msg)
--> 394         raise self.exc_type(msg)

TypeError: Caught TypeError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/data/vision/torralba/scratch2/tongzhou/miniconda3/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/data/vision/torralba/scratch2/tongzhou/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: 'x'
```

The key issue is that 
https://github.com/pytorch/pytorch/blob/fb24f7c4adbd3469f552f3aa0db006317889f88f/torch/nn/parallel/scatter_gather.py#L36-L41
does not actually care about whether the scattered thing is a tensor or not, and blindly pads args and kwargs to have the same length.