Repository: pytorch
Issue: [AMD] [ROCm] Numerical difference between Pytorch 2.6.0.dev of ROCm 6.2 and ROCm 6.3
Labels: high priority,module: rocm,triaged

### 🐛 Describe the bug

# Description
I am getting different numerical output results between Pytorch 2.6.0.dev of ROCm 6.2 and ROCm 6.3.

All the tests in the https://github.com/linkedin/Liger-Kernel/pull/506 pass with PyTorch 2.6.0.dev of ROCm 6.2

However, one of the tests fail in the environment with PyTorch 2.6.0.dev of ROCm 6.3.
- Only one failed test in ROCm 6.3 
  - FAILED CONVERGENCE TEST
  ```
  ====================================================== short test summary info =======================================================
  FAILED test/convergence/test_mini_models.py::test_mini_model[mini_mllama-32-0.0001-dtype2-1e-08-1e-05-0.005-1e-05-0.005-1e-05] - AssertionError: Number of mismatched elements: 2
  Mismatch at index (0, 7): tensor1[(0, 7)] = 3.0651497840881348, tensor2[(0, 7)] = 3.0652356147766113
  Mismatch at index (0, 9): tensor1[(0, 9)] = 1.470238447189331, tensor2[(0, 9)] = 1.4702625274658203
  ======================================== 1 failed, 16 passed, 2 warnings in 94.82s (0:01:34) ==================
  ```

# Steps to reproduce:
1. Launch docker container
  ```
  #!/bin/bash
  sudo docker run -it \
     --network=host \
     --group-add=video \
     --ipc=host \
     --cap-add=SYS_PTRACE \
     --security-opt seccomp=unconfined \
     --device /dev/kfd \
     --device /dev/dri \
     -v <path-to-Liger-Kernel>:/liger-kernel-workspace \
     rocm/pytorch:rocm6.3_ubuntu22.04_py3.10_pytorch_release_2.5.0_preview \
     /bin/bash
  ```

2. Setup
   ```
   python -m pip uninstall -y torch torchvision triton
   python -m pip install -e .[dev] --extra-index-url https://download.pytorch.org/whl/nightly/rocm6.3
   python -m pip install triton==3.1.0
   ```
   
 3. Run tests
 ``` make test-convergence ```


https://github.com/linkedin/Liger-Kernel/pull/506

### Versions

```bash
root@root:/liger-kernel-workspace# python collect_env.py                                                                                  /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_pytree.py:185: FutureWarning: optree is installed but the version is too ol
d to support PyTorch Dynamo in C++ pytree. C++ pytree support is disabled. Please consider upgrading optree using `python3 -m pip install --u
pgrade 'optree>=0.13.0'`.                                                                                                                    
  warnings.warn(                                                                                                                             
Collecting environment information...                                                                                                        
PyTorch version: 2.6.0.dev20241231+rocm6.3 
Is debug build: False                                                                                                             
CUDA used to build PyTorch: N/A                                                                                                              
ROCM used to build PyTorch: 6.3.42131-fa1d09cbd                                                                                              
                                                                                                                                             
OS: Ubuntu 22.04.5 LTS (x86_64)                                                                                                              
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0                                                                                           
Clang version: 18.0.0git (https://github.com/RadeonOpenCompute/llvm-project roc-6.3.1 24463 5d263cec09eef95b1ed5f3e7f6a578c616efa0a4)        
CMake version: version 3.26.4                                                                                                                
Libc version: glibc-2.35                                                                                                                     
                                                                                                                                             
Python version: 3.10.15 (main, Oct  3 2024, 07:27:34) [GCC 11.2.0] (64-bit runtime)                                                          
Python platform: Linux-5.15.0-84-generic-x86_64-with-glibc2.35                                                                               
Is CUDA available: True                                                                                                                      
CUDA runtime version: Could not collect                                                                                                      
CUDA_MODULE_LOADING set to: LAZY                                                                                                             
GPU models and configuration: AMD Instinct MI210 (gfx90a:sramecc+:xnack-)                                                                    
Nvidia driver version: Could not collect                                                                                                     
cuDNN version: Could not collect                                                                                                             
HIP runtime version: 6.3.42131                                                                                                               
MIOpen runtime version: 3.3.0                                                                                                                
Is XNNPACK available: True                                                                                                                   
                                                                                                                                             
CPU:                                                                                                                                         
Architecture:                       x86_64                                                                                                   
CPU op-mode(s):                     32-bit, 64-bit                                                                                           
Address sizes:                      48 bits physical, 48 bits virtual                                                                        
Byte Order:                         Little Endian                                                                                            
CPU(s):                             32                                                                                                       
On-line CPU(s) list:                0-31                                                                                                     
Vendor ID:                          AuthenticAMD                                                                                             
Model name:                         AMD Ryzen 9 7950X 16-Core Processor                                                                      
CPU family:                         25                                                                                                       
Model:                              97                                                                                                       
Thread(s) per core:                 2                                                                                                        
Core(s) per socket:                 16                                                                                                       
Socket(s):                          1                                                                                                        
Stepping:                           2                                                                                                        
Frequency boost:                    enabled                                                                                                  
CPU max MHz:                        5879.8818
CPU min MHz:                        3000.0000
BogoMIPS:                           8983.59
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx512_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overflow_recov succor smca fsrm flush_l1d
Virtualization:                     AMD-V
L1d cache:                          512 KiB (16 instances)
L1i cache:                          512 KiB (16 instances)
L2 cache:                           16 MiB (16 instances)
L3 cache:                           64 MiB (2 instances)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-31
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] flake8==7.1.1
[pip3] mypy==1.10.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] onnx==1.16.1
[pip3] onnxscript==0.1.0.dev20240817
[pip3] optree==0.12.1
[pip3] pytorch-triton-rocm==3.2.0+git0d4682f0
[pip3] torch==2.6.0.dev20241231+rocm6.3
[pip3] triton==3.1.0
[conda] No relevant packages

```

cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd