Repository: pytorch
Issue: MiniCPM-V 2.6 transformer model returns nonsense tokens when running on MPS backend vs CPU backend.
Labels: triaged,module: correctness (silent),module: mps

### 🐛 Describe the bug

When I run MiniCPM-v2.6 model on my MacBook, the outputs look fine when using CPU backend, but they tend to contain nonsense English tokens or foreign language tokens when running on MPS backend. Here's the code to reproduce the issue:

```python
import requests
import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True,
    attn_implementation='sdpa', torch_dtype=torch.bfloat16)
model = model.eval().to("mps")

tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6', trust_remote_code=True)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg?download=true"
image = Image.open(requests.get(url, stream=True).raw).convert('RGB')

question = 'What is in the image?'
msgs = [{'role': 'user', 'content': [image, question]}]

res = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(res)
```

Here are some example generations with MPS backend:
```
SRúite, aá.
우리 이미지에는 자동차와 건물이 있습니다.
可能です。
ῶς ἐγὼ τὴν εἰκώνα προσεβλέψω.
OTOS
```

Here are example generations with CPU backend for the exact same input.
```
The image features a vintage car, specifically a turquoise-colored Volkswagen Beetle, parked in front of a yellow building with wooden doors.
The image features a vintage turquoise-colored car parked on the side of a street. The car has classic design elements, such as round headlights and chrome accents. Behind the car, there's a yellow wall with two wooden doors.
The image features a vintage turquoise-colored car parked on a paved street in front of an old building with yellow walls and wooden doors.
```

### Versions

```
Collecting environment information...
PyTorch version: 2.5.0.dev20240826
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.5 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.3.9.4)
CMake version: Could not collect
Libc version: N/A

Python version: 3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 10:07:17) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-14.5-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M3 Max

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.4.0
[pip3] torchaudio==2.4.0.dev20240826
[pip3] torchvision==0.20.0.dev20240826
[conda] numpy                     1.26.4          py312h7f4fdc5_0
[conda] numpy-base                1.26.4          py312he047099_0
[conda] pytorch                   2.5.0.dev20240826        py3.12_0    pytorch-nightly
[conda] torch                     2.4.0                    pypi_0    pypi
[conda] torchaudio                2.4.0.dev20240826       py312_cpu    pytorch-nightly
[conda] torchvision               0.20.0.dev20240826       py312_cpu    pytorch-nightly
```

cc @kulinseth @albanD @malfet @DenisVieriu97 @jhavukainen