Repository: pytorch
Issue:  USE_NATIVE_ARCH flag causes nvcc build failure due to "'arch=native': expected a number"
Labels: module: build,triaged

### 🐛 Describe the bug

I'm attempting to build pytorch wheels on a Jetson AGX Xavier DevKit running Ubuntu 18.04.

The script below works fine when the `USE_NATIVE_ARCH` option is not set, and fails when `USE_NATIVE_ARCH=1`.

<details>
<summary>
Script (click to expand)
</summary>

```bash
#!/bin/bash
set -e
set -o pipefail

python3.10 -m venv wheel_builder
source wheel_builder/bin/activate

PYTORCH_VERSION="1.11.0"

cd "${HOME}"

git clone --depth 1 --recursive --branch "v$PYTORCH_VERSION" https://github.com/pytorch/pytorch.git
cd pytorch

export BUILD_TEST=0
export USE_NCCL=0
export USE_DISTRIBUTED=0
export USE_QNNPACK=0
export USE_PYTORCH_QNNPACK=0
export TORCH_CUDA_ARCH_LIST="5.3;6.2;7.2"
export USE_NUMPY=1
export USE_CUDNN=1
export USE_NATIVE_ARCH=1
export PYTORCH_BUILD_VERSION="$PYTORCH_VERSION"
export PYTORCH_BUILD_NUMBER=1

export CC=$(which gcc-8)
export CXX=$(which g++-8)

pip install numpy==1.22.3

pip install wheel
pip install -r requirements.txt
pip install scikit-build
pip install ninja

python setup.py bdist_wheel
```
</details>

The configuration and CMake outputs can be found here: [Summaries.txt](https://github.com/pytorch/pytorch/files/8768732/Summaries.txt)

The build target that fails is
`[2888/3297] Building CUDA object caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/CrossKernel.cu.o`. It fails because `nvcc` is provided the `-march=native` option, yet doesn't seem to understand or expect that option. `-march` is a known gcc flag and works fine for native binaries but may not be a supported option for nvcc.

It does seem that nvcc has an `arch` option in some contexts, probably referring to CUDA compute capabilities. But `native` isn't an accepted value.


Error output:

```
FAILED: caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/CrossKernel.cu.o/usr/local/cuda-10.2/bin/nvcc  -DAT_PER_OPERATOR_HEADERS -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DTORCH_CUDA_BUILD_MAIN_LIB -DUSE_CUDA -DUSE_EXTERNAL_MZCRC -D_FILE_OFFSET_BITS=64 -Dtorch_cuda_EXPORTS -Iaten/src -I../aten/src -I. -I../ -I../cmake/../third_party/cudnn_frontend/include -I../third_party/onnx -Ithird_party/onnx -I../third_party/foxi -Ithird_party/foxi -Iinclude -I../torch/csrc/distributed -I../aten/src/THC -I../aten/src/ATen/cuda -Icaffe2/aten/src -I../aten/../third_party/catch/single_include -I../aten/src/ATen/.. -I../c10/cuda/../.. -I../c10/.. -I../torch/csrc/api -I../torch/csrc/api/include -isystem=../third_party/protobuf/src -isystem=../third_party/XNNPACK/include -isystem=../cmake/../third_party/eigen -isystem=/usr/include/python3.10 -isystem=/home/aruw/jetson-setup/asset-builds/wheel_builder/lib/python3.10/site-packages/numpy/core/include -isystem=../cmake/../third_party/pybind11/include -isystem=../cmake/../third_party/cub -isystem=/usr/local/cuda-10.2/include -Xfatbin -compress-all -DONNX_NAMESPACE=onnx_torch -gencode arch=compute_53,code=sm_53 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_72,code=sm_72 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda  -Wno-deprecated-gpu-targets --expt-extended-lambda -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -O3 -DNDEBUG -Xcompiler=-fPIC   -march=native -D__NEON__ -DTH_HAVE_THREAD -Xcompiler=-Wall,-Wextra,-Wno-unused-parameter,-Wno-unused-variable,-Wno-unused-function,-Wno-unused-result,-Wno-unused-local-typedefs,-Wno-missing-field-initializers,-Wno-write-strings,-Wno-unknown-pragmas,-Wno-type-limits,-Wno-array-bounds,-Wno-unknown-pragmas,-Wno-sign-compare,-Wno-strict-overflow,-Wno-strict-aliasing,-Wno-error=deprecated-declarations,-Wno-missing-braces,-Wno-maybe-uninitialized -DTORCH_CUDA_BUILD_MAIN_LIB -Xcompiler -pthread -std=c++14 -x cu -c ../aten/src/ATen/native/cuda/CrossKernel.cu -o caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/CrossKernel.cu.o && /usr/local/cuda-10.2/bin/nvcc  -DAT_PER_OPERATOR_HEADERS -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DTORCH_CUDA_BUILD_MAIN_LIB -DUSE_CUDA -DUSE_EXTERNAL_MZCRC-D_FILE_OFFSET_BITS=64 -Dtorch_cuda_EXPORTS -Iaten/src -I../aten/src -I. -I../ -I../cmake/../third_party/cudnn_frontend/include -I../third_party/onnx -Ithird_party/onnx -I../third_party/foxi -Ithird_party/foxi -Iinclude -I../torch/csrc/distributed -I../aten/src/THC -I../aten/src/ATen/cuda -Icaffe2/aten/src -I../aten/../third_party/catch/single_include -I../aten/src/ATen/.. -I../c10/cuda/../.. -I../c10/.. -I../torch/csrc/api -I../torch/csrc/api/include -isystem=../third_party/protobuf/src -isystem=../third_party/XNNPACK/include -isystem=../cmake/../third_party/eigen -isystem=/usr/include/python3.10 -isystem=/home/aruw/jetson-setup/asset-builds/wheel_builder/lib/python3.10/site-packages/numpy/core/include -isystem=../cmake/../third_party/pybind11/include -isystem=../cmake/../third_party/cub -isystem=/usr/local/cuda-10.2/include -Xfatbin -compress-all -DONNX_NAMESPACE=onnx_torch -gencode arch=compute_53,code=sm_53 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_72,code=sm_72 -Xcudafe --diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl --expt-relaxed-constexpr --expt-extended-lambda -Wno-deprecated-gpu-targets --expt-extended-lambda -DCUDA_HAS_FP16=1 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -O3 -DNDEBUG -Xcompiler=-fPIC -march=native -D__NEON__ -DTH_HAVE_THREAD -Xcompiler=-Wall,-Wextra,-Wno-unused-parameter,-Wno-unused-variable,-Wno-unused-function,-Wno-unused-result,-Wno-unused-local-typedefs,-Wno-missing-field-initializers,-Wno-write-strings,-Wno-unknown-pragmas,-Wno-type-limits,-Wno-array-bounds,-Wno-unknown-pragmas,-Wno-sign-compare,-Wno-strict-overflow,-Wno-strict-aliasing,-Wno-error=deprecated-declarations,-Wno-missing-braces,-Wno-maybe-uninitialized -DTORCH_CUDA_BUILD_MAIN_LIB -Xcompiler -pthread -std=c++14 -x cu -M ../aten/src/ATen/native/cuda/CrossKernel.cu -MT caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/CrossKernel.cu.o -o caffe2/CMakeFiles/torch_cuda.dir/__/aten/src/ATen/native/cuda/CrossKernel.cu.o.d

nvcc fatal   : 'arch=native': expected a number
```

I would expect the `USE_NATIVE_ARCH` option to work for CUDA-enabled builds.

### Versions

Some of the below data is unhelpful because it attempts to use `nvidia-smi`, which isn't available on the Jetson line. However, CUDA 10.2 is available and known to work with other wheels we have built.

```
Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (aarch64)
GCC version: (Ubuntu/Linaro 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.13.2
Libc version: glibc-2.27

Python version: 3.10.4 (main, Apr  9 2022, 21:27:52) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.9.253-tegra-aarch64-with-glibc2.27
Is CUDA available: N/A
CUDA runtime version: 10.2.300
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: Probably one of the following:
/usr/lib/aarch64-linux-gnu/libcudnn.so.8.2.1
/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8.2.1
/usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8.2.1
/usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8.2.1
/usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8.2.1
/usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8.2.1
/usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8.2.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

Versions of relevant libraries:
[pip3] numpy==1.22.3
[conda] Could not collect
```

```
aruw@jetson-aruw-lambda:~/jetson-setup/asset-builds$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Feb_28_22:34:44_PST_2021
Cuda compilation tools, release 10.2, V10.2.300
Build cuda_10.2_r440.TC440_70.29663091_0
```

cc @malfet @seemethere