Repository: pytorch
Issue: Quantizable LSTM has different behavior than LSTM in bidirectional setting
Labels: oncall: quantization,module: rnn,triaged

## Issue description

The quantizable LSTM has a different behavior than the regular LSTM module in the bidirectional setting with multiple layers. In particular, the input parameters of the 2nd to n-th layer of the quantizable LSTM do not take the backward cells into account.

## Code example

The relevant code lines are 
**Quantizable LSTM:** 
https://github.com/pytorch/pytorch/blob/c2ce79f06eb4a8cec2f9cfbdf3a1a4021a0a4cfa/torch/ao/nn/quantizable/modules/rnn.py#L320-L324

**Regular LSTM:**
https://github.com/pytorch/pytorch/blob/c2ce79f06eb4a8cec2f9cfbdf3a1a4021a0a4cfa/torch/nn/modules/rnn.py#L92-L94

The proposed solution would be to use this code in the quantized LSTM model
```
for layer in range(1, num_layers):
    layers.append(_LSTMLayer(self.hidden_size * num_directions, 
                             self.hidden_size,
                             self.bias, batch_first=False,
                             bidirectional=self.bidirectional,
                             **factory_kwargs))
```

## System Info
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Red Hat Enterprise Linux release 8.6 (Ootpa) (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-10)
Clang version: 13.0.1 (Red Hat 13.0.1-2.module+el8.6.0+15487+86bd2a95)
CMake version: version 3.20.2
Libc version: glibc-2.28

Python version: 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:25:59)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-4.18.0-372.26.1.el8_6.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: 
GPU models and configuration: GPU 0: Tesla V100-SXM2-32GB
Nvidia driver version: 520.61.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.11.0
[pip3] torchaudio==0.11.0
[pip3] torchvision==0.12.0
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               h2bc3f7f_2  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] mkl                       2021.4.0           h8d4b97c_729    conda-forge
[conda] mkl-service               2.4.0            py39h7e14d7c_0    conda-forge
[conda] mkl_fft                   1.3.1            py39h0c7bc48_1    conda-forge
[conda] mkl_random                1.2.2            py39hde0f152_0    conda-forge
[conda] numpy                     1.19.5                   pypi_0    pypi
[conda] pytorch                   1.11.0          py3.9_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.11.0               py39_cu113    pytorch
[conda] torchvision               0.12.0               py39_cu113    pytorch


cc @jerryzh168 @jianyuh @raghuramank100 @jamesr66a @vkuzo @jgong5 @Xia-Weiwen @leslie-fang-intel @zou3519 @mikaylagawarecki