Repository: pytorch
Issue: TestSDPAPrivateUse1Only tests causes irreversible changes to PyTorch internal state
Labels: module: tests,triaged,module: PrivateUse1

### 🐛 Describe the bug

Related Issue: https://github.com/pytorch/pytorch/issues/134602

To reproduce:
```
# PYTORCH_TEST_WITH_ROCM=1 is need on ROCM platform
PYTORCH_TESTING_DEVICE_ONLY_FOR="cuda" python test/test_transformers.py -v --use-pytest
```

The `--use-pytest` option will run `TestSDPAPrivateUse1Only.*` at the beginning of the whole `test/test_transformers.py` test suite, and causes massive failures due to synchronization problems because `TestSDPAPrivateUse1Only` will irreversibly change the internal state of PyTorch by registering the `privateuse1` backend:
https://github.com/pytorch/pytorch/blob/a8611da86f42a442c3ab891a038af55440ccd8d0/test/test_transformers.py#L3766-L3780

After registering the privateuse1 backend, `at::getAccelerator` will always return `kPrivateUse1`
https://github.com/pytorch/pytorch/blob/a8611da86f42a442c3ab891a038af55440ccd8d0/aten/src/ATen/DeviceAccelerator.cpp#L17-L22

Normally this is not a problem, since for common applications, `privateuse1` and other devices are mutually exclusive.
However, this causes synchronization problem in PyTorch's test suites, if `PrivateUse1` is tested before any other cuda/rocm/other accelerator tests.

Suppose we are testing some **_CUDA backward tests_** after running `privateuse1` related test.
The registered `privateuse1` backend will the default device during autograd:
https://github.com/pytorch/pytorch/blob/fb1c58089290f982be1cd06e95b690308fb8af78/torch/csrc/autograd/function.h#L242-L253

And the `opt_device_type` certainly will not match any of the input's device because they are all CUDA.
Consequently the function `Node::stream()` will always return `std::nullopt`, disregard any stream supplied by `input_metadata_`.

## Suggested Solution 1

Add prefix `z` to privateuse1 related tests to ensure it is always the last one (even if `--use-pytest` is supplied)

## Suggested Solution 2

Add a context manager to unregister the `privateuse1` backend when the test is complete. Note there is no C++ API to unregister the `privateuse1` backend at the moment.

### Versions

```
PyTorch version: 2.5.0a0+git735162e             
Is debug build: False                           
CUDA used to build PyTorch: N/A                 
ROCM used to build PyTorch: 6.2.41133-dd7f95766 
                                                           
OS: Ubuntu 20.04.6 LTS (x86_64)                                                                                        
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0      
Clang version: Could not collect                                                                                       
CMake version: version 3.26.4                                                                                          
Libc version: glibc-2.31                                                                                                                                                                                                                      
                                                           
Python version: 3.9.19 (main, May  6 2024, 19:43:03)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.5.0-35-generic-x86_64-with-glibc2.31                                                                                                                                                                                 Is CUDA available: True                                                                                                                                                                                                                       CUDA runtime version: Could not collect                                                                                                                                                                                                       CUDA_MODULE_LOADING set to: LAZY                                                                                                                                                                                                              GPU models and configuration: AMD Instinct MI210 (gfx90a:sramecc+:xnack-)
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 6.2.41133
MIOpen runtime version: 3.2.0 
Is XNNPACK available: True          
                                                           
CPU:                               
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Byte Order:                         Little Endian
Address sizes:                      43 bits physical, 48 bits virtual
CPU(s):                             128
On-line CPU(s) list:                0-127
Thread(s) per core:                 2
Core(s) per socket:                 32
Socket(s):                          2
NUMA node(s):                       2
Vendor ID:                          AuthenticAMD
CPU family:                         23                                                                                 
Model:                              49                                                                                 
Model name:                         AMD EPYC 7542 32-Core Processor      
Stepping:                           0                                                                                  
Frequency boost:                    enabled                                                                            
CPU MHz:                            1500.000                                                                           
CPU max MHz:                        2900.0000                                                                          
CPU min MHz:                        1500.0000
BogoMIPS:                           5800.24
Virtualization:                     AMD-V
L1d cache:                          2 MiB
L1i cache:                          2 MiB
L2 cache:                           32 MiB
L3 cache:                           256 MiB
NUMA node0 CPU(s):                  0-31,64-95
NUMA node1 CPU(s):                  32-63,96-127
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Mitigation; untrained return thunk; SMT enabled with STIBP protection
Vulnerability Spec rstack overflow: Mitigation; Safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines; IBPB conditional; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid
 aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_
nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_loc
al clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sev se
v_es

Versions of relevant libraries:
[pip3] flake8==6.1.0
[pip3] flake8-bugbear==23.3.23
[pip3] flake8-comprehensions==3.15.0
[pip3] flake8-executable==2.1.3
[pip3] flake8-logging-format==0.9.0
[pip3] flake8-pyi==23.3.1
[pip3] flake8-simplify==0.19.3
[pip3] mypy==1.10.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.0
[pip3] optree==0.12.1
[pip3] pytorch-triton-rocm==3.0.0
[pip3] torch==2.5.0a0+git2aca0ae
[pip3] torchaudio==2.4.0+rocm6.1
[pip3] torchvision==0.19.0a0
[pip3] triton==3.0.0
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-include               2021.4.0           h06a4308_640  
[conda] numpy                     1.20.3                   pypi_0    pypi
[conda] optree                    0.11.0                   pypi_0    pypi
[conda] torch                     2.4.0a0+git6b37b45          pypi_0    pypi
[conda] torchvision               0.19.0a0+69e03db          pypi_0    pypi
[conda] triton                    3.0.0                    pypi_0    pypi
```

cc @mruberry @ZainRizvi @NmomoN @mengpenghui @fwenguang @cdzhan @1274085042 @PHLens