Repository: pytorch
Issue: Run ChatRWKV on MBP(intel CPU)+eGPU[rx6800 16G], returna a very big number -9223372036854775808, looks like overflow
Labels: triaged,module: mps

### 🐛 Describe the bug

Run [ChatRWKV](https://github.com/BlinkDL/ChatRWKV) using 'mps', returna a very big number, looks like overflow.

MBP(intel CPU, not M1/M2), with eGPU[rx6800 16G]
pytorch==2.0.0

It can  load model, but when calculate the 1st token, it gets a very big number `-9223372036854775808` and return error.

error log:

```
Bob: hi

 len(tokens), tokens[]: 8 [26845, 27, 14260, 187, 187, 2422, 547, 27]
Alice:
 token, out: -9223372036854775808 tensor([ -5.4473, -25.4831,  -6.7508,  ...,  -7.4079,  -6.1975,  -4.1842],
       device='mps:0')

 len(tokens), tokens[]: 1 [-9223372036854775808]
Traceback (most recent call last):
  File "/Users/ppt/Github/ChatRWKV/v2/chat.py", line 474, in <module>
    on_message(msg)
  File "/Users/ppt/Github/ChatRWKV/v2/chat.py", line 387, in on_message
    out = run_rnn([token], newline_adj=newline_adj)
  File "/Users/ppt/Github/ChatRWKV/v2/chat.py", line 163, in run_rnn
    out, model_state = model.forward(tokens[:CHUNK_LEN], model_state)
  File "/Users/ppt/Github/ChatRWKV/v2/../rwkv_pip_package/src/rwkv/model.py", line 563, in forward
    x = w['emb.weight'][tokens if seq_mode else tokens[0]]
RuntimeError: Expected !is_symbolic() to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)
(ChatRWKV) ppt@pptdeMacBook-Pro v2 % 
```

I tried to rollback to the previous version of pytorch(1.12.1/1.13.1), it will send a warning msg and fallback to cpu, but it can  return a correct number:

```
Bob: 你好

 len(tokens), tokens[]: 10 [26845, 27, 209, 24553, 34439, 187, 187, 2422, 547, 27]
Alice:/Users/ppt/Github/ChatRWKV/v2/../rwkv_pip_package/src/rwkv/utils.py:68: UserWarning: The operator 'aten::sort.values_stable' is not currently supported on the MPS backend and will fall back to run on the CPU. This may have performance implications. (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/mps/MPSFallback.mm:11.)
  sorted_ids = torch.argsort(probs)

 token, out: 27 tensor([ -5.4373, -25.4622,  -6.7405,  ...,  -7.3974,  -6.1903,  -4.1751],
       device='mps:0')

 len(tokens), tokens[]: 1 [27]
:
 token, out: 24553 tensor([-1.0000e+09, -2.1039e+01, -1.5952e+00,  ..., -1.3116e+00,
        -6.4459e-01, -1.3537e-01], device='mps:0')

 len(tokens), tokens[]: 1 [24553]
你
 token, out: 34439 tensor([-1.0000e+09, -2.2532e+01, -6.4443e+00,  ..., -7.8328e+00,
        -7.3615e+00, -5.4704e+00], device='mps:0')

 len(tokens), tokens[]: 1 [34439]
好
 token, out: 30676 tensor([-1.0000e+09, -1.7130e+01,  1.8807e+00,  ..., -9.7331e-01,
        -2.4166e-01,  1.1678e-01], device='mps:0')

 len(tokens), tokens[]: 1 [30676]

 token, out: 221 tensor([-1.0000e+09, -3.8395e+01, -1.6493e+01,  ..., -2.1061e+01,
        -2.0698e+01, -1.8379e+01], device='mps:0')

 len(tokens), tokens[]: 1 [221]
啊
 token, out: 26680 tensor([-1.0000e+09, -1.5628e+01,  3.8878e+00,  ...,  5.0439e-01,
         1.1729e+00,  2.6680e+00], device='mps:0')

 len(tokens), tokens[]: 1 [26680]
！
 token, out: 15367 tensor([-1.0000e+09, -1.8554e+01,  4.5207e-01,  ..., -1.0022e+00,
        -3.3723e-01,  1.5153e+00], device='mps:0')
```

https://github.com/BlinkDL/ChatRWKV/issues/93

### Versions

(ChatRWKV) ppt@pptdeMacBook-Pro ChatRWKV % python ./collect_env.py 
Collecting environment information...
PyTorch version: 2.0.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 13.2.1 (x86_64)
GCC version: Could not collect
Clang version: 14.0.3 (clang-1403.0.22.14.1)
CMake version: Could not collect
Libc version: N/A

Python version: 3.10.10 (main, Mar 21 2023, 13:41:39) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-10.16-x86_64-i386-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Intel(R) Core(TM) i5-1038NG7 CPU @ 2.00GHz

Versions of relevant libraries:
[pip3] numpy==1.24.2
[pip3] torch==2.0.0
[pip3] torchaudio==2.0.1
[pip3] torchvision==0.15.1
[conda] numpy                     1.24.2                   pypi_0    pypi
[conda] torch                     2.0.0                    pypi_0    pypi
[conda] torchaudio                2.0.1                    pypi_0    pypi
[conda] torchvision               0.15.1                   pypi_0    pypi
(ChatRWKV) ppt@pptdeMacBook-Pro ChatRWKV % 

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev