Repository: pytorch
Issue: batch size unexpectedly affects model inference on Mac M1
Labels: triaged,module: correctness (silent),module: mps

### 🐛 Describe the bug

Batch size appears to unexpectedly affect the output of a model in inference mode when running on a Macbook Pro M1. This issue does not seem to present elsewhere (e.g. older Macbook Pro with intel cpu).
Haven't found the exact crossover point that triggers this.

code to reproduce:

```python
import torch
from torchvision import models

# set up inceptionv3 model for feature extraction
model = models.inception_v3(weights=models.Inception_V3_Weights.DEFAULT)
model.fc = torch.nn.Identity()
model.eval()

# create random data
input_data = torch.randn(128, 3, 299, 299)

# create large batch - all items
batch_128 = input_data
# create a couple of small batches
batch_1 = input_data[0:1]
batch_4 = input_data[0:4]

# just take first output of each and compare
output_128_0 = model(batch_128)[0]
output_1_0 = model(batch_1)[0]
output_4_0 = model(batch_4)[0]

# sanity check
assert torch.equal(batch_128[0], batch_1[0]), "data not equal... scenario set up badly"


# small batches should be the same
assert torch.equal(output_1_0, output_4_0), "output not equal between small batches"

# should produce same output? this breaks on macbook pro m1
assert torch.equal(
    output_128_0, output_1_0
), "output not equal between small and large batches"

print("all fine")
```

### Versions

```
Collecting environment information...
PyTorch version: 2.0.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.6.5 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.202)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.4 (v3.10.4:9d38120e33, Mar 23 2022, 17:29:05) [Clang 13.0.0 (clang-1300.0.29.30)] (64-bit runtime)
Python platform: macOS-12.6.5-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.25.0
[pip3] torch==2.0.1
[pip3] torchvision==0.15.2
[conda] nomkl                     3.0                           0  
[conda] numpy                     1.21.5           py39h42add53_3  
[conda] numpy-base                1.21.5           py39hadd41eb_3  
[conda] numpydoc                  1.2                pyhd3eb1b0_0  
```
and 

```
Collecting environment information...
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 12.6.5 (arm64)
GCC version: Could not collect
Clang version: 14.0.0 (clang-1400.0.29.202)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.4 (v3.10.4:9d38120e33, Mar 23 2022, 17:29:05) [Clang 13.0.0 (clang-1300.0.29.30)] (64-bit runtime)
Python platform: macOS-12.6.5-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Pro

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] pytorch-lightning==1.8.3.post1
[pip3] torch==1.13.1
[pip3] torchmetrics==0.11.4
[pip3] torchtext==0.14.1
[pip3] torchvision==0.14.1
[conda] nomkl                     3.0                           0  
[conda] numpy                     1.21.5           py39h42add53_3  
[conda] numpy-base                1.21.5           py39hadd41eb_3  
[conda] numpydoc                  1.2                pyhd3eb1b0_0  
```

cc @kulinseth @albanD @malfet @DenisVieriu97 @jhavukainen @razarmehr @abhudev