Repository: pytorch
Issue: nn.Transformer out[0:-1] not precisely equal to last_out when predicting in tgt mask
Labels: module: nn,triaged

### 🐛 Describe the bug

Should nn.Transformer output keep the same last values when using tgt mask to predict current value? 

```python
import torch
from torch import nn

transformer = nn.Transformer(nhead=16, num_encoder_layers=12)

src = torch.randn((10, 32, 512))
tgt = torch.randn((20, 32, 512))

out_sequence_list = []
seq_tgt = tgt[0:1, :, :]

# predict
with torch.no_grad():
    for _ in range(src.size(0)):
        tgt_mask = transformer.generate_square_subsequent_mask(seq_tgt.size(0))

        seq_out = transformer(src=src,
                              tgt=seq_tgt,
                              tgt_mask=tgt_mask)
        latest_out = seq_out[-1, :, :].unsqueeze(0)

        out_sequence_list.append(latest_out)

        # AssertError when seq_out.size(0) >= 2

        # In [1]: seq_out - torch.cat(out_sequence_list, dim=0)
        # Out[1]:
        # tensor([[[-0.2359,  1.1037, -0.4994,  ..., -1.6979,  1.0394,  0.7114],
        #          [-0.2334, -0.2180,  2.1017,  ..., -0.6985,  1.3802,  0.6556],
        #          [-0.0134,  2.5504,  1.7151,  ..., -0.2102, -0.3800,  0.3372],
        #          ...,
        #          [ 0.1166, -0.5099,  0.4002,  ...,  0.4912,  0.3495,  0.2175],
        #          [ 1.3356,  0.9382, -0.0159,  ...,  0.1846, -1.5192,  1.2687],
        #          [-0.5507,  0.4309,  0.5729,  ..., -0.5270, -0.8455, -1.1845]],
        #         [[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        #          [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        #          [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        #          ...,
        #          [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        #          [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        #          [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000]]])
        assert torch.all(seq_out == torch.cat(out_sequence_list, dim=0))

        seq_tgt = torch.cat([seq_tgt, latest_out], dim=0)
```

### Versions

```
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.3.1 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.28.1)
CMake version: version 3.26.3
Libc version: N/A

Python version: 3.10.4 (main, Mar 31 2022, 03:37:37) [Clang 12.0.0 ] (64-bit runtime)
Python platform: macOS-13.3.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] pytorch-crf==0.7.2
[pip3] pytorch-ignite==0.5.0.dev20221205
[pip3] pytorch-lightning==1.9.3
[pip3] torch==2.0.0
[pip3] torchaudio==2.0.0
[pip3] torchdata==0.6.0
[pip3] torchmetrics==0.11.4
[pip3] torchtext==0.3.1
[pip3] torchvision==0.15.0
[conda] ignite                    0.5.0.dev20221205            py_0    pytorch-nightly
[conda] numpy                     1.21.5          py310h220015d_3    defaults
[conda] numpy-base                1.21.5          py310h742c864_3    defaults
[conda] pytorch                   2.0.0                  py3.10_0    pytorch
[conda] pytorch-crf               0.7.2                    pypi_0    pypi
[conda] pytorch-lightning         1.9.3                    pypi_0    pypi
[conda] torch                     2.0.0                    pypi_0    pypi
[conda] torchaudio                2.0.0                 py310_cpu    pytorch
[conda] torchdata                 0.6.0                     py310    pytorch
[conda] torchmetrics              0.11.4                   pypi_0    pypi
[conda] torchtext                 0.3.1                    pypi_0    pypi
[conda] torchvision               0.15.0                py310_cpu    pytorch
```

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @bhosmer @cpuhrsch @erichan1 @drisspg