Repository: pytorch
Issue: `bias=False` fails in `Transformer` when `batch_first=True` and in eval mode
Labels: module: nn,triaged

### 🐛 Describe the bug

```
import torch
from torch import nn

transformer_model = nn.Transformer(nhead=16, num_encoder_layers=12, bias=False, batch_first=True)
src = torch.rand((10, 32, 512))
tgt = torch.rand((10, 32, 512))

transformer_model.eval()
out = transformer_model(src, tgt)
```

```
Traceback (most recent call last):
  File "/Users/adam.amster/Library/Application Support/JetBrains/PyCharm2024.3/scratches/scratch_35.py", line 9, in <module>
    out = transformer_model(src, tgt)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adam.amster/PycharmProjects/seq2seq translation/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adam.amster/PycharmProjects/seq2seq translation/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adam.amster/PycharmProjects/seq2seq translation/.venv/lib/python3.11/site-packages/torch/nn/modules/transformer.py", line 206, in forward
    memory = self.encoder(src, mask=src_mask, src_key_padding_mask=src_key_padding_mask,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adam.amster/PycharmProjects/seq2seq translation/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adam.amster/PycharmProjects/seq2seq translation/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adam.amster/PycharmProjects/seq2seq translation/.venv/lib/python3.11/site-packages/torch/nn/modules/transformer.py", line 391, in forward
    output = mod(output, src_mask=mask, is_causal=is_causal, src_key_padding_mask=src_key_padding_mask_for_layers)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adam.amster/PycharmProjects/seq2seq translation/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adam.amster/PycharmProjects/seq2seq translation/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adam.amster/PycharmProjects/seq2seq translation/.venv/lib/python3.11/site-packages/torch/nn/modules/transformer.py", line 676, in forward
    elif not all((x.device.type in _supported_device_type) for x in tensor_args):
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/adam.amster/PycharmProjects/seq2seq translation/.venv/lib/python3.11/site-packages/torch/nn/modules/transformer.py", line 676, in <genexpr>
    elif not all((x.device.type in _supported_device_type) for x in tensor_args):
                  ^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'device'
```

note that it works fine in train mode

### Versions

```
PyTorch version: 2.2.2
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.5 (x86_64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: version 3.24.0
Libc version: N/A

Python version: 3.11.9 (main, Apr  2 2024, 08:25:04) [Clang 15.0.0 (clang-1500.1.0.2.5)] (64-bit runtime)
Python platform: macOS-14.5-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M3 Pro

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] torch==2.2.2
[pip3] torchinfo==1.8.0
[pip3] torchmetrics==1.4.0.post0
[pip3] torchtext==0.17.2
```

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki