Repository: pytorch
Issue: [MPS] Severe Performance Degradation with Deterministic Algorithm on macOS 14.4
Labels: module: performance,triaged,module: regression,module: mps

### 🐛 Describe the bug

## Issue Description
After updating to macOS 14.4, I have encountered significant performance degradation when using MPS with the deterministic algorithm **enabled**.

## Reproduce the Issue
Below is a minimal piece of code to reproduce the issue:
```python
import torch
import torchvision.models as models
from tqdm import tqdm

torch.manual_seed(0)
device = 'mps'

data = torch.rand(256, 3, 32, 32, device=device)  # Simulating a batch of 256, 32x32 RGB images
model = models.resnet152().to(device)  # Other models can also reproduce the issue (e.g., DenseNet, EfficientNet)

# Default, use_deterministic_algorithms == False
for i in tqdm(range(20)):
    model(data)

# use_deterministic_algorithms == True
torch.use_deterministic_algorithms(True)
for i in tqdm(range(20)):
    model(data)
```

## Current Behavior
Based on the result from my device (M1 Max 24-core GPU), the time used for the above forward pass with the **_deterministic algorithm disabled_** (default of PyTorch) and **_enabled_** are **_3s and 22s respectively_**. In other words, the degradation is roughly 8x. The result may be a bit different with different models, but still a severe degradation.

## Expected Behavior
It should be only around 2x degradation.

Since I identified this issue after upgrading to macOS 14.4, I didn't test the above code on macOS 14.3. However, I have some records of the performance on macOS 14.3 from a personal project. It shows that, for that project, training one epoch took around 85s without enabling the deterministic algorithm; and around 144s with deterministic algorithm enabled. Thus, the record suggested the degradation should be around 2x normally.

## Notes
From the "Activity Monitor", we can observe that **the GPU utilization caps at around 50% when the deterministic algorithm is enabled**. This cap persists regardless of the number of code instances executed concurrently, as long as they all have the deterministic algorithm enabled. (Total GPU utilization of all instances caps at around 50%) At the same time, the CPU utilization for each instance hits 100%. It appears that this behavior is the direct cause of this issue.

##
Thank you for your attention to this matter.

### Versions

PyTorch version: 2.2.1
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.4 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.3.9.4)
CMake version: Could not collect
Libc version: N/A

Python version: 3.12.2 | packaged by Anaconda, Inc. | (main, Feb 27 2024, 12:57:28) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-14.4-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Max

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.2.1
[pip3] torchaudio==2.2.1
[pip3] torchvision==0.17.1
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] torch                     2.2.1                    pypi_0    pypi
[conda] torchaudio                2.2.1                    pypi_0    pypi
[conda] torchvision               0.17.1                   pypi_0    pypi

cc @kulinseth @albanD @malfet @DenisVieriu97 @razarmehr