Repository: pytorch
Issue: torch.compile mode="max-autotune" precision appears to be lower
Labels: high priority,triaged,oncall: pt2,module: inductor

### 🐛 Describe the bug

I'm sorry, this is going to be a terrible bug report, but I wasn't able to simplify the problem. 

I'm running a training run with a model that is compiled with `torch.compile`. The blue line below is loss with the `default` mode, the red line below is loss with `max-autotune`, the training runs are otherwise identical:

![spiky_comparison](https://user-images.githubusercontent.com/22680696/224837823-788ccd48-6fb0-4099-9c87-46926dca134f.png)


It appears as if precision/stability is noticeably lower on the max-autotune training run. This is running on an RTXA600 GPU. The forward pass is wrapped in 
 `torch.autocast`, there are no graph breaks or warning messages. Max-autotune code is also noticeably a bit faster, so it is definitely doing something that the default mode is not.
 
 Feel free to close this issue if this is not enough info. I could also test more specific things or try to make a mini version to get to the bottom of this, but I have no idea what to look for, or which part would be relevant.

### Error logs

_No response_

### Minified repro

_No response_

### Versions

```
PyTorch version: 2.1.0.dev20230307
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Pop!_OS 22.04 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.9 (main, Mar  1 2023, 18:23:06) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.2.0-76060200-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.5.119
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3050 Ti Laptop GPU
Nvidia driver version: 525.85.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True


Versions of relevant libraries:
[pip3] lion-pytorch==0.0.7
[pip3] numpy==1.23.5
[pip3] torch==2.1.0.dev20230307
[pip3] torchaudio==2.0.0.dev20230307
[pip3] torchvision==0.15.0.dev20230307
[conda] blas                      1.0                         mkl  
[conda] lion-pytorch              0.0.7                    pypi_0    pypi
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0           py310h7f8727e_0  
[conda] mkl_fft                   1.3.1           py310hd6ae3a3_0  
[conda] mkl_random                1.2.2           py310h00e6091_0  
[conda] numpy                     1.23.5          py310hd5efca6_0  
[conda] numpy-base                1.23.5          py310h8e6c178_0  
[conda] pytorch                   2.1.0.dev20230307 py3.10_cuda11.8_cudnn8.7.0_0    pytorch-nightly
[conda] pytorch-cuda              11.8                 h7e8668a_3    pytorch-nightly
[conda] pytorch-mutex             1.0                        cuda    pytorch-nightly
[conda] torchaudio                2.0.0.dev20230307     py310_cu118    pytorch-nightly
[conda] torchtriton               2.0.0+b8b470bc59           py310    pytorch-nightly
[conda] torchvision               0.15.0.dev20230307     py310_cu118    pytorch-nightly

```

cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @bdhirsh @anijain2305 @chauhang @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @muchulee8 @aakhundov @ColinPeppler @amjames @desertfire @wconstab @soumith @ngimel