Repository: pytorch
Issue: compiling for rocm gfx1010, getting cuda errors
Labels: module: build,module: rocm,triaged

### 🐛 Describe the bug

I am compiling for rocm, but getting a cuda error.
The same error has been discussed in https://github.com/pytorch/pytorch/issues/108344, but the user said it was solved by using `USE_ROCM` and did not elaborate further.

```
export VERBOSE=1
export PYTORCH_ROCM_ARCH="gfx1010"
export USE_ROCM=1
export ROCM_PATH=/usr
export USE_CUDA=0
export USE_XPU=0
```


```
[6372/7445] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/jit/ir/ir.cpp.o
FAILED: caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/jit/ir/ir.cpp.o 
/usr/bin/ccache /usr/bin/c++ -DAT_PER_OPERATOR_HEADERS -DBUILD_ONEDNN_GRAPH -DCAFFE2_BUILD_MAIN_LIB -DCPUINFO_SUPPORTED_PLATFORM=1 -DFLASHATTENTION_DISABLE_ALIBI -DFMT_HEADER_ONLY=1 -DFXDIV_USE_INLINE_ASSEMBLY=0 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DNNP_CONVOLUTION_ONLY=0 -DNNP_INFERENCE_ONLY=0 -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DUSE_C10D_GLOO -DUSE_DISTRIBUTED -DUSE_EXTERNAL_MZCRC -DUSE_RPC -DUSE_TENSORPIPE -DXNN_LOG_LEVEL=0 -D_FILE_OFFSET_BITS=64 -Dtorch_cpu_EXPORTS -I/home/user/git/pytorch/build/aten/src -I/home/user/git/pytorch/aten/src -I/home/user/git/pytorch/build -I/home/user/git/pytorch -I/home/user/git/pytorch/cmake/../third_party/benchmark/include -I/home/user/git/pytorch/third_party/onnx -I/home/user/git/pytorch/build/third_party/onnx -I/home/user/git/pytorch/nlohmann -I/home/user/git/pytorch/torch/csrc/api -I/home/user/git/pytorch/torch/csrc/api/include -I/home/user/git/pytorch/caffe2/aten/src/TH -I/home/user/git/pytorch/build/caffe2/aten/src/TH -I/home/user/git/pytorch/build/caffe2/aten/src -I/home/user/git/pytorch/build/caffe2/../aten/src -I/home/user/git/pytorch/torch/csrc -I/home/user/git/pytorch/third_party/miniz-3.0.2 -I/home/user/git/pytorch/third_party/kineto/libkineto/include -I/home/user/git/pytorch/third_party/kineto/libkineto/src -I/home/user/git/pytorch/third_party/cpp-httplib -I/home/user/git/pytorch/aten/src/ATen/.. -I/home/user/git/pytorch/third_party/FXdiv/include -I/home/user/git/pytorch/c10/.. -I/home/user/git/pytorch/third_party/pthreadpool/include -I/home/user/git/pytorch/third_party/cpuinfo/include -I/home/user/git/pytorch/aten/src/ATen/native/quantized/cpu/qnnpack/include -I/home/user/git/pytorch/aten/src/ATen/native/quantized/cpu/qnnpack/src -I/home/user/git/pytorch/aten/src/ATen/native/quantized/cpu/qnnpack/deps/clog/include -I/home/user/git/pytorch/third_party/NNPACK/include -I/home/user/git/pytorch/third_party/fbgemm/include -I/home/user/git/pytorch/third_party/fbgemm -I/home/user/git/pytorch/third_party/fbgemm/third_party/asmjit/src -I/home/user/git/pytorch/third_party/ittapi/src/ittnotify -I/home/user/git/pytorch/third_party/FP16/include -I/home/user/git/pytorch/third_party/tensorpipe -I/home/user/git/pytorch/build/third_party/tensorpipe -I/home/user/git/pytorch/third_party/tensorpipe/third_party/libnop/include -I/home/user/git/pytorch/third_party/fmt/include -I/home/user/git/pytorch/build/third_party/ideep/mkl-dnn/include -I/home/user/git/pytorch/third_party/ideep/mkl-dnn/src/../include -I/home/user/git/pytorch/third_party/flatbuffers/include -isystem /home/user/git/pytorch/build/third_party/gloo -isystem /home/user/git/pytorch/cmake/../third_party/gloo -isystem /home/user/git/pytorch/cmake/../third_party/tensorpipe/third_party/libuv/include -isystem /home/user/git/pytorch/cmake/../third_party/googletest/googlemock/include -isystem /home/user/git/pytorch/cmake/../third_party/googletest/googletest/include -isystem /home/user/git/pytorch/third_party/protobuf/src -isystem /home/user/git/pytorch/third_party/XNNPACK/include -isystem /home/user/git/pytorch/third_party/ittapi/include -isystem /home/user/git/pytorch/cmake/../third_party/eigen -isystem /home/user/git/pytorch/third_party/ideep/mkl-dnn/include/oneapi/dnnl -isystem /home/user/git/pytorch/third_party/ideep/include -isystem /home/user/git/pytorch/INTERFACE -isystem /home/user/git/pytorch/third_party/nlohmann/include -isystem /home/user/git/pytorch/build/include -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-dangling-reference -Wno-error=dangling-reference -Wno-error=redundant-move -Wno-stringop-overflow -DHAVE_AVX512_CPU_DEFINITION -DHAVE_AVX2_CPU_DEFINITION -O3 -DNDEBUG -DNDEBUG -std=gnu++17 -fPIC -DTORCH_USE_LIBUV -DCAFFE2_USE_GLOO -Wall -Wextra -Wdeprecated -Wno-unused-parameter -Wno-missing-field-initializers -Wno-array-bounds -Wno-unknown-pragmas -Wno-strict-overflow -Wno-strict-aliasing -Wunused-function -Wunused-variable -Wunused-but-set-variable -Wno-maybe-uninitialized -fvisibility=hidden -O2 -pthread -DASMJIT_STATIC -fopenmp -fopenmp -MD -MT caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/jit/ir/ir.cpp.o -MF caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/jit/ir/ir.cpp.o.d -o caffe2/CMakeFiles/torch_cpu.dir/__/torch/csrc/jit/ir/ir.cpp.o -c /home/user/git/pytorch/torch/csrc/jit/ir/ir.cpp
/home/user/git/pytorch/torch/csrc/jit/ir/ir.cpp: In member function 'bool torch::jit::Node::hasSideEffects() const':
/home/user/git/pytorch/torch/csrc/jit/ir/ir.cpp:1181:16: error: 'set_stream' is not a member of 'torch::jit::cuda'; did you mean 'c10::cuda::set_stream'?
 1181 |     case cuda::set_stream:
      |                ^~~~~~~~~~
In file included from /home/user/git/pytorch/torch/csrc/jit/ir/ir.h:18,
                 from /home/user/git/pytorch/torch/csrc/jit/ir/ir.cpp:1:
/home/user/git/pytorch/aten/src/ATen/core/interned_strings.h:223:11: note: 'c10::cuda::set_stream' declared here
  223 |   _(cuda, set_stream)                \
      |           ^~~~~~~~~~
/home/user/git/pytorch/aten/src/ATen/core/interned_strings.h:351:35: note: in definition of macro 'DEFINE_SYMBOL'
  351 |   namespace ns { constexpr Symbol s(static_cast<unique_t>(_keys::ns##_##s)); }
      |                                   ^
/home/user/git/pytorch/aten/src/ATen/core/interned_strings.h:352:1: note: in expansion of macro 'FORALL_NS_SYMBOLS'
  352 | FORALL_NS_SYMBOLS(DEFINE_SYMBOL)
      | ^~~~~~~~~~~~~~~~~
/home/user/git/pytorch/torch/csrc/jit/ir/ir.cpp:1182:16: error: '_set_device' is not a member of 'torch::jit::cuda'; did you mean 'c10::cuda::_set_device'?
 1182 |     case cuda::_set_device:
      |                ^~~~~~~~~~~
/home/user/git/pytorch/aten/src/ATen/core/interned_strings.h:222:11: note: 'c10::cuda::_set_device' declared here
  222 |   _(cuda, _set_device)               \
      |           ^~~~~~~~~~~
/home/user/git/pytorch/aten/src/ATen/core/interned_strings.h:351:35: note: in definition of macro 'DEFINE_SYMBOL'
  351 |   namespace ns { constexpr Symbol s(static_cast<unique_t>(_keys::ns##_##s)); }
      |                                   ^
/home/user/git/pytorch/aten/src/ATen/core/interned_strings.h:352:1: note: in expansion of macro 'FORALL_NS_SYMBOLS'
  352 | FORALL_NS_SYMBOLS(DEFINE_SYMBOL)
      | ^~~~~~~~~~~~~~~~~
/home/user/git/pytorch/torch/csrc/jit/ir/ir.cpp:1183:16: error: '_current_device' is not a member of 'torch::jit::cuda'; did you mean 'c10::cuda::_current_device'?
 1183 |     case cuda::_current_device:
      |                ^~~~~~~~~~~~~~~
/home/user/git/pytorch/aten/src/ATen/core/interned_strings.h:224:11: note: 'c10::cuda::_current_device' declared here
  224 |   _(cuda, _current_device)           \
      |           ^~~~~~~~~~~~~~~
/home/user/git/pytorch/aten/src/ATen/core/interned_strings.h:351:35: note: in definition of macro 'DEFINE_SYMBOL'
  351 |   namespace ns { constexpr Symbol s(static_cast<unique_t>(_keys::ns##_##s)); }
      |                                   ^
/home/user/git/pytorch/aten/src/ATen/core/interned_strings.h:352:1: note: in expansion of macro 'FORALL_NS_SYMBOLS'
  352 | FORALL_NS_SYMBOLS(DEFINE_SYMBOL)
      | ^~~~~~~~~~~~~~~~~
/home/user/git/pytorch/torch/csrc/jit/ir/ir.cpp:1184:16: error: 'synchronize' is not a member of 'torch::jit::cuda'; did you mean 'c10::cuda::synchronize'?
 1184 |     case cuda::synchronize:
      |                ^~~~~~~~~~~
/home/user/git/pytorch/aten/src/ATen/core/interned_strings.h:225:11: note: 'c10::cuda::synchronize' declared here
  225 |   _(cuda, synchronize)               \
      |           ^~~~~~~~~~~
/home/user/git/pytorch/aten/src/ATen/core/interned_strings.h:351:35: note: in definition of macro 'DEFINE_SYMBOL'
  351 |   namespace ns { constexpr Symbol s(static_cast<unique_t>(_keys::ns##_##s)); }
      |                                   ^
/home/user/git/pytorch/aten/src/ATen/core/interned_strings.h:352:1: note: in expansion of macro 'FORALL_NS_SYMBOLS'
  352 | FORALL_NS_SYMBOLS(DEFINE_SYMBOL)
      | ^~~~~~~~~~~~~~~~~

```

<details>
  <summary>output of `rocminfo`</summary>

```
ROCk module is loaded
=====================
HSA System Attributes
=====================
Runtime Version:         1.1
Runtime Ext Version:     1.6
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE
System Endianness:       LITTLE
Mwaitx:                  DISABLED
DMAbuf Support:          YES

==========
HSA Agents
==========
*******
Agent 1
*******
  Name:                    AMD Ryzen 9 5950X 16-Core Processor
  Uuid:                    CPU-XX
  Marketing Name:          AMD Ryzen 9 5950X 16-Core Processor
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    0
  Device Type:             CPU
  Cache Info:
    L1:                      32768(0x8000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   3400
  BDFID:                   0
  Internal Node ID:        0
  Compute Unit:            32
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Memory Properties:
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    65759348(0x3eb6874) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    65759348(0x3eb6874) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    65759348(0x3eb6874) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 4
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    65759348(0x3eb6874) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*******
Agent 2
*******
  Name:                    gfx1010
  Uuid:                    GPU-XX
  Marketing Name:          AMD Radeon RX 5700 XT
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    1
  Device Type:             GPU
  Cache Info:
    L1:                      16(0x10) KB
    L2:                      4096(0x1000) KB
  Chip ID:                 29471(0x731f)
  ASIC Revision:           2(0x2)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   2100
  BDFID:                   2048
  Internal Node ID:        1
  Compute Unit:            40
  SIMDs per CU:            2
  Shader Engines:          2
  Shader Arrs. per Eng.:   2
  WatchPts on Addr. Ranges:4
  Coherent Host Access:    FALSE
  Memory Properties:
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          32(0x20)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        40(0x28)
  Max Work-item Per CU:    1280(0x500)
  Grid Max Size:           4294967295(0xffffffff)
  Grid Max Size per Dimension:
    x                        4294967295(0xffffffff)
    y                        4294967295(0xffffffff)
    z                        4294967295(0xffffffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 151
  SDMA engine uCode::      35
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    8372224(0x7fc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    8372224(0x7fc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Recommended Granule:0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx1010:xnack-
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        4294967295(0xffffffff)
        y                        4294967295(0xffffffff)
        z                        4294967295(0xffffffff)
      FBarrier Max Size:       32
*** Done ***
```
</details>

[full log](https://github.com/user-attachments/files/18544411/log.txt)

please do not tell me to use docker

### Versions



<details>
  <summary>output of `python torch/utils/collect_env.py`</summary>

```
Collecting environment information...                                                                                                                                                                                                                       PyTorch version: 2.7.0.dev20250123+rocm6.3
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 6.3.42131-fa1d09cbd

OS: Gentoo Linux (x86_64)
GCC version: (Gentoo 13.2.1_p20240210 p14) 13.2.1 20240210
Clang version: 19.1.5
CMake version: version 3.31.4
Libc version: glibc-2.40

Python version: 3.12.8 (main, Jan 12 2025, 23:50:05) [GCC 14.2.1 20241116] (64-bit runtime)
Python platform: Linux-6.7.7-gentoo-dist-x86_64-AMD_Ryzen_9_5950X_16-Core_Processor-with-glibc2.40
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: AMD Radeon RX 5700 XT (gfx1010:xnack-)
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 6.3.42131
MIOpen runtime version: 3.3.0
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             32
On-line CPU(s) list:                0-31
Vendor ID:                          AuthenticAMD
Model name:                         AMD Ryzen 9 5950X 16-Core Processor
CPU family:                         25
Model:                              33
Thread(s) per core:                 2
Core(s) per socket:                 16
Socket(s):                          1
Stepping:                           0
Frequency boost:                    enabled
CPU(s) scaling MHz:                 73%
CPU max MHz:                        5083.3979
CPU min MHz:                        2200.0000
BogoMIPS:                           6789.80
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm debug_swap
L1d cache:                          512 KiB (16 instances)
L1i cache:                          512 KiB (16 instances)
L2 cache:                           8 MiB (16 instances)
L3 cache:                           64 MiB (2 instances)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-31
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; Safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==2.0.2
[pip3] onnx==1.16.2
[pip3] optree==0.14.0
[pip3] pytorch-triton-rocm==3.2.0+git0d4682f0
[pip3] pytorch-triton-xpu==3.2.0+gite98b6fcb
[pip3] torch==2.7.0.dev20250123+rocm6.3
[pip3] torchaudio==2.6.0.dev20250124+rocm6.3
[pip3] torchsde==0.2.6
[pip3] torchvision==0.22.0.dev20250124+rocm6.3
[conda] Could not collect
```


cc @malfet @seemethere @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd