Repository: pytorch
Issue: Support DataParallel with PackedSequence
Labels: oncall: distributed,module: bootcamp,triaged,enhancement,module: data parallel

## 🚀 Feature
Ability to feed PackedSequence into DataParallel model as input. This can reduce wasteful data transfer when dealing with variable length data.

## Motivation
Dealing with variable length data as input to DataParallel is painful. If you put them into a python list, DataParallel scatter them along wrong axis (instead of splitting the list into equal size, it splits the elements inside the list). You can pad them to the same size, but then you waste memory and transfer bandwidth moving meaningless padding values. 

The ideal solution would be DataParallel supporting PackedSequence input, because PackedSequence is designed for handling variable length data.

## Pitch
I propose that we can do something like the below example.
```python
from torch.nn.utils.rnn import pack_sequence

# batch size = 2 and sequence lengths = (2, 3)
data = pack_sequence([
    torch.tensor([[0.1, 0.1, 0.1], [0.2, 0.2, 0.2]]), 
    torch.tensor([[0.1, 0.1, 0.1], [0.2, 0.2, 0.2], [0.3, 0.3, 0.3]])
  ], 
  enforce_sorted=False
)
model = nn.LSTM(3, 100)
dp_model = nn.DataParallel(model)
dp_model.to('cuda')

output, _ = dp_model(data)
output # PackedSequence
```

## Alternatives
Padding variable length data to the same length and use them as input. This is wasteful in terms of memory and transfer bandwidth.

## Additional context
FYI, the above example produces this error with 1.2.0 version
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/opt/conda/lib/python3.7/site-packages/torch/_utils.py", line 369, in reraise
    raise self.exc_type(msg)
AttributeError: Caught AttributeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 564, in forward
    return self.forward_tensor(input, hx)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 539, in forward_tensor
    max_batch_size = input.size(0) if self.batch_first else input.size(1)
AttributeError: 'tuple' object has no attribute 'size'
```

cc @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @xush6528