Repository: pytorch
Issue: Undocumented fast pass behavior in nn.TransformerEncoderLayer causes failures in test_transformerencoderlayer_cuda_float32 (__main__.TestNNDeviceTypeCUDA)
Labels: module: nn,triaged

### 🐛 Describe the bug

This issue is related to https://github.com/pytorch/pytorch/issues/134687

Unit test test_transformerencoderlayer_cuda_float32 (in `test_nn.py`) performs the following tests:
https://github.com/pytorch/pytorch/blob/a8611da86f42a442c3ab891a038af55440ccd8d0/test/test_nn.py#L12426-L12450

This test expects all NaN output tensor if the `TransformerEncoderLayer` not using fast path. However, the determining factor of all NaN output is if the `SDPBackend.FLASH_ATTENTION` or `SDPBackend.EFFICIENT_ATTENTION` is used by the underlying SDPA operator within TransformerEncoderLayer

However, the non-fast path of `TransformerEncoderLayer` does not explicitly disables the FA/MEFF attention:
https://github.com/pytorch/pytorch/blob/a8611da86f42a442c3ab891a038af55440ccd8d0/torch/nn/modules/transformer.py#L896-L927

Therefore, the actual backend selection logic depends on three parts of the whole codebase
1. The fast path/non-fast path logic in `TransformerEncoderLayer`
2. [bool can_use_flash_attention(sdp_params const& params, bool debug)](https://github.com/pytorch/pytorch/blob/a8611da86f42a442c3ab891a038af55440ccd8d0/aten/src/ATen/native/transformers/cuda/sdp_utils.cpp#L583)
3. [bool can_use_mem_efficient_attention(sdp_params const& params, bool debug)](https://github.com/pytorch/pytorch/blob/a8611da86f42a442c3ab891a038af55440ccd8d0/aten/src/ATen/native/transformers/cuda/sdp_utils.cpp#L634)

2 and 3 are undocumented, and FA may be used even if `TransformerEncoderLayer` should use non-fast path according to its document.

## Suggested solution

When calling `TransformerEncoderLayer._sa_block`, explicitly disables FA/MEFF backend by enclosing `x = self.self_attn` with `with sdpa_kernel([SDPBackend.MATH]):`

### Versions

```
PyTorch version: 2.5.0a0+git735162e             
Is debug build: False                           
CUDA used to build PyTorch: N/A                 
ROCM used to build PyTorch: 6.2.41133-dd7f95766 
                                                           
OS: Ubuntu 20.04.6 LTS (x86_64)                                                                                        
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0      
Clang version: Could not collect                                                                                       
CMake version: version 3.26.4                                                                                          
Libc version: glibc-2.31                                                                                                                                                                                                                      
                                                           
Python version: 3.9.19 (main, May  6 2024, 19:43:03)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.5.0-35-generic-x86_64-with-glibc2.31                                                                                                                                                                                 Is CUDA available: True                                                                                                                                                                                                                       CUDA runtime version: Could not collect                                                                                                                                                                                                       CUDA_MODULE_LOADING set to: LAZY                                                                                                                                                                                                              GPU models and configuration: AMD Instinct MI210 (gfx90a:sramecc+:xnack-)
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 6.2.41133
MIOpen runtime version: 3.2.0 
Is XNNPACK available: True          
                                                           
CPU:                               
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      43 bits physical, 48 bits virtual
CPU(s):                             128
On-line CPU(s) list:                0-127
Thread(s) per core:                 2
Core(s) per socket:                 32
Socket(s):                          2
NUMA node(s):                       2
Vendor ID:                          AuthenticAMD
CPU family:                         23                                                                                 
Model:                              49                                                                                 
Model name:                         AMD EPYC 7542 32-Core Processor      
Stepping:                           0                                                                                  
Frequency boost:                    enabled                                                                            
CPU MHz:                            1500.000                                                                           
CPU max MHz:                        2900.0000                                                                          
CPU min MHz:                        1500.0000
BogoMIPS:                           5800.24
Virtualization:                     AMD-V
L1d cache:                          2 MiB
L1i cache:                          2 MiB
L2 cache:                           32 MiB
L3 cache:                           256 MiB
NUMA node0 CPU(s):                  0-31,64-95
NUMA node1 CPU(s):                  32-63,96-127
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow: Mitigation; Safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid
 aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_
nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_loc
al clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev se
v_es

Versions of relevant libraries:
[pip3] flake8==6.1.0
[pip3] flake8-bugbear==23.3.23
[pip3] flake8-comprehensions==3.15.0
[pip3] flake8-executable==2.1.3
[pip3] flake8-logging-format==0.9.0
[pip3] flake8-pyi==23.3.1
[pip3] flake8-simplify==0.19.3
[pip3] mypy==1.10.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.0
[pip3] optree==0.12.1
[pip3] pytorch-triton-rocm==3.0.0
[pip3] torch==2.5.0a0+git2aca0ae
[pip3] torchaudio==2.4.0+rocm6.1
[pip3] torchvision==0.19.0a0
[pip3] triton==3.0.0
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-include               2021.4.0           h06a4308_640  
[conda] numpy                     1.20.3                   pypi_0    pypi
[conda] optree                    0.11.0                   pypi_0    pypi
[conda] torch                     2.4.0a0+git6b37b45          pypi_0    pypi
[conda] torchvision               0.19.0a0+69e03db          pypi_0    pypi
[conda] triton                    3.0.0                    pypi_0    pypi
```

cc @albanD @mruberry @jbschlosser @walterddr @mikaylagawarecki @bhosmer @cpuhrsch @erichan1 @drisspg