Repository: pytorch
Issue: [Feature Request] Additional torch.nn.LSTM functionality
Labels: todo,module: rnn,triaged,enhancement

If you have a question or would like help and support, please ask at our
[forums](https://discuss.pytorch.org/).

If you are submitting a feature request, please preface the title with [feature request].
If you are submitting a bug report, please fill in the following details.

## Issue description

Ultimately, I would like to create a clean, fully functional, stacked LSTM unit out of existing functions, which allows different widths for each layer, allows dropout, batched and packed sequences with masks, and allows passage of a list of layer sizes to the module for automatic construction.  This specific feature request is motivated by the error message from the code below, telling me to ask for a feature request.

## Code example

> class Model1(torch.nn.Module):
> 
>     [etc, etc]
> 
>     def forward(self, inputs, lengths):
>         pack1              = nn.utils.rnn.pack_padded_sequence(inputs, lengths, batch_first=True)
>         out1, self.hidden1 = self.lstm1(pack1, (self.hidden1[0].detach(), self.hidden1[1].detach()))
>         pad1               = nn.utils.rnn.pad_packed_sequence(out1, batch_first=True)        
>         drop1              = self.dropout1(pad1)
>         
>         pack2              = nn.utils.rnn.pack_padded_sequence(drop1, lengths, batch_first=True)
>         out2, self.hidden2 = self.lstm2(pack2,   (self.hidden2[0].detach(), self.hidden2[1].detach()))
>         pad2               = nn.utils.rnn.pad_packed_sequence(out2, batch_first=True)
> 
> return pad2

This results in the error message: "RuntimeError: Returning Variables sharing storage with other Variables that require grad is not supported in Python functions. Please submit a feature request if you hit this error."

That is the direct result of trying to unpack and repack the sequences around the application of the dropout layer, since applying the dropout to a packed sequence does not result in a packed sequence.  And trying to apply the dropout manually is necessary because the LSTM layer is hardwired against applying its own dropout to the "last" layer of the LSTM, which is every layer in this approach because they are all separate.  They are all separate because the multi-layer stacked LSTM units all assume the same layer width.

And even if I got all that to work right, I would still be tinkering with the module definition directly, because trying to construct those layers in a list ends up with the layer weights not registered as parameters for the optimizer. 

I must be missing something obvious, because this seems to be very much more difficult than it should be, to construct a pretty standard configuration with modern features.  

## System Info

Collecting environment information...
PyTorch version: 0.4.0
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.4 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609
CMake version: Could not collect

Python version: 3.5
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration: GPU 0: Quadro M4000
Nvidia driver version: 390.48
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.1.4
/usr/lib/x86_64-linux-gnu/libcudnn_static_v7.a

Versions of relevant libraries:
[pip3] numpy (1.14.2)
[pip3] torch (0.4.0)
[pip3] torchvision (0.2.1)
[conda] Could not collect

cc @zou3519