Repository: pytorch
Issue: MPS Performance regressions on Sonoma 14.0 
Labels: high priority,triaged,module: mps

### 🐛 Describe the bug

This issue is related to #77799. tldr: speed 50% slower, big memory leaks. 

Basically, since upgrading to Sonoma, performance of the MPS device on sentence transformers models has taken a big nosedive. I don't have an apples-to-apples (🥁) comparison exactly, but an M1 ultra on Sonoma is 50% slower than an M1 Pro on Ventura. [Here](https://github.com/pytorch/pytorch/issues/77799#issuecomment-1304882735) are some numbers I collected on Ventura with an M1 ultra, not sure that data can be exactly compared, but it looks like the ratio between inference time on M1 Ultra/Ventura and M1 Ultra / Sonoma is about 1:2. 

On Sonoma (M1 Ultra):
```
In [1]: from sentence_transformers import SentenceTransformer

In [2]: import torch

In [5]: %timeit model.encode(["hi"], device=torch.device("cpu"))
85.3 ms ± 12 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [6]: %timeit model.encode(["hi"], device=torch.device("mps"))
23 ms ± 616 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
```


The MPS device is clearly functioning and present, but 23 ms is about 50% slower than an M1 Pro on Ventura. Here is the Ventura data:

```
In [2]: import torch

In [3]: model = SentenceTransformer("all-mpnet-base-v2")

In [4]: %timeit model.encode(["hi"], device=torch.device("mps"))
14.7 ms ± 854 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [5]: %timeit model.encode(["hi"], device=torch.device("cpu"))
40.1 ms ± 408 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

Both cases with with today's nightly PyTorch, and sentence-transformers==2.2.2, transformers==4.34.1. 


Additionally, let me know if I should file another ticket for this, but I observe huge memory leaks when using the MPS device. I often conduct a lot of bulk embedding using this model, and after ~32k calls to `model.forward`, I see memory usage exceeding 100GB and increasing. It's compressed, so it looks leaked. I observe this problem when running torch in an Flask wrapper.

I am wondering if something about Apple's Metal Performance Shaders implementation changed recently. 

### Versions

Collecting environment information...
PyTorch version: 2.2.0.dev20231018
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.0 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.0.40.1)
CMake version: version 3.27.6
Libc version: N/A

Python version: 3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)] (64-bit runtime)
Python platform: macOS-14.0-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Ultra

Versions of relevant libraries:
[pip3] mypy==1.4.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.25.2
[pip3] torch==2.2.0.dev20231018
[pip3] torchsort==0.1.9
[pip3] torchvision==0.17.0.dev20231018
[conda] Could not collect

cc @ezyang @gchanan @zou3519 @kadeng @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr @abhudev