Repository: pytorch
Issue: Inconsistencies in LSTM outputs when processing sequence stepwise on CPU
Labels: module: numerical-stability,module: rnn,triaged

## 🐛 Bug

I *believe* I've run into what may be a bug in the PyTorch LSTM module. My understanding of LSTMs is that given the internal state information, it shouldn't matter if you are processing a sequence in one go (so `shape == [seq_len, batch, n]`) or processing the sequence "stepwise"; passing the updated state tensors back in with each iteration. E.g., 
```python

in_tensor = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(1, batch_size, hidden_size)
cell = torch.zeros(1, batch_size, hidden_size)

output_list = []
for x in in_tensor:
    x = x.unsqueeze(0)
    out, (hidden, cell) = lstm(x, hx=(hidden, cell))
    output_list.append(out)
output = torch.cat(output_list, dim=0)
```

Again, *as I understand it* (and maybe my understanding is wrong), the above code should produce the same
output as if `in_tensor` were just passed through the lstm layer all at once. E.g.,

```python
in_tensor = torch.randn(seq_len, batch_size, input_size)
hidden = torch.zeros(1, batch_size, hidden_size)
cell = torch.zeros(1, batch_size, hidden_size)

output, _ = lstm(in_tensor, hx=(hidden, cell))
```

In my use of `LSTM` during inference however, I've found these two methods to produce slightly different results. The difference is not dramatic, but it won't pass an equality check. It should be noted that I have not tested this on GPU. All of
my testing has been done on CPU.

### EDIT:
Looks like `GRU` does the same thing.

## To Reproduce

[Here](https://gist.github.com/mdlockyer/a8a0de85f3c2893c16cf92192e8424cb) is a script that reproduces the problem. You'll probably notice that most of the numbers are the same except for the last few decimal places.

## Environment

PyTorch version: 1.9.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 11.6 (x86_64)
GCC version: Could not collect
Clang version: 13.0.0 (clang-1300.0.29.3)
CMake version: version 3.19.7
Libc version: N/A

Python version: 3.9.2 (v3.9.2:1a79785e3e, Feb 19 2021, 09:06:10)  [Clang 6.0 (clang-600.0.57)] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.2
[pip3] torch==1.9.0
[conda] blas                        1.0                mkl  
[conda] mkl                         2019.4         233  
[conda] mkl-service           2.3.0            py37hfbe908c_0  
[conda] mkl_fft                   1.0.15           py37h5e564d8_0  
[conda] mkl_random          1.1.0             py37ha771720_0  
[conda] mypy                      0.782           pypi_0    pypi
[conda] mypy-extensions  0.4.3            pypi_0    pypi
[conda] numpy                    1.18.1           py37h7241aed_0  
[conda] numpy-base          1.18.1           py37h6575580_1  
[conda] numpydoc              0.9.2           py_0


cc @zou3519