Repository: pytorch
Issue: Conv vs. Scipy tests fail on MacOS
Labels: triaged,module: correctness (silent),module: mps

### 🐛 Describe the bug

`vs_scipy` tests in `test/nn/test_convolution.py` fail on Mac

```
__ TestConvolutionNNDeviceTypeMPS.test_conv3d_vs_scipy_mode_same_mps_float32 ___
Traceback (most recent call last):
  File "/Users/ec2-user/runner/_work/pytorch/pytorch/test/nn/test_convolution.py", line 2076, in test_conv3d_vs_scipy
    _test(t, weight_odd, mode)
  File "/Users/ec2-user/runner/_work/pytorch/pytorch/test/nn/test_convolution.py", line 2069, in _test
    self.assertEqual(actual, expected, rtol=2e-5, atol=5e-6)
  File "/Users/ec2-user/runner/_work/_temp/conda_environment_10845817884/lib/python3.9/site-packages/torch/testing/_internal/common_utils.py", line 3885, in assertEqual
    raise error_metas.pop()[0].to_error(
AssertionError: Tensor-likes are not close!

Mismatched elements: 1 / 250 (0.4%)
Greatest absolute difference: 8.262693881988525e-06 at index (2, 0, 4) (up to 5e-06 allowed)
Greatest relative difference: 8.1622478319332e-05 at index (2, 0, 4) (up to 2e-05 allowed)

To execute this test, run the following from the base repo dir:
    python test/nn/test_convolution.py TestConvolutionNNDeviceTypeMPS.test_conv3d_vs_scipy_mode_same_mps_float32
```





### Versions

(torch) sidk@mac-907256 pytorch % python collect_env.py 
Collecting environment information...
PyTorch version: 2.5.0a0+git65bab4e
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 15.1 (arm64)
GCC version: Could not collect
Clang version: 16.0.0 (clang-1600.0.21.4)
CMake version: version 3.26.4
Libc version: N/A

Python version: 3.10.14 (main, May  6 2024, 14:42:37) [Clang 14.0.6 ] (64-bit runtime)
Python platform: macOS-15.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M2 Max

Versions of relevant libraries:
[pip3] executorch==0.3.0a0+9b7a6d9
[pip3] flake8==6.1.0
[pip3] flake8-bugbear==23.3.23
[pip3] flake8-comprehensions==3.15.0
[pip3] flake8-executable==2.1.3
[pip3] flake8-logging-format==0.9.0
[pip3] flake8-pyi==23.3.1
[pip3] flake8-simplify==0.19.3
[pip3] functorch==2.0.0
[pip3] mypy==1.10.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.0
[pip3] optree==0.12.1
[pip3] torch==2.5.0.dev20240806
[pip3] torchao==0.1
[pip3] torchaudio==2.5.0.dev20240902
[pip3] torchsr==1.0.4
[pip3] torchutils==0.0.4
[pip3] torchvision==0.20.0.dev20240902
[conda] executorch                0.3.0a0+9b7a6d9          pypi_0    pypi
[conda] functorch                 2.0.0                    pypi_0    pypi
[conda] numpy                     1.26.0                   pypi_0    pypi
[conda] optree                    0.12.1                   pypi_0    pypi
[conda] torch                     2.5.0a0+git8a166ae          pypi_0    pypi
[conda] torchao                   0.1                      pypi_0    pypi
[conda] torchaudio                2.5.0.dev20240902          pypi_0    pypi
[conda] torchfix                  0.4.0                    pypi_0    pypi
[conda] torchsr                   1.0.4                    pypi_0    pypi
[conda] torchutils                0.0.4                    pypi_0    pypi
[conda] torchvision               0.20.0.dev20240902          pypi_0    pypi

cc @kulinseth @albanD @malfet @DenisVieriu97 @jhavukainen