Repository: pytorch
Issue: [BUG] torch/extension.h: undefined symbol
Labels: module: cpp-extensions,triaged,module: pybind

### 🐛 Describe the bug

`ImportError: /path/to/lib/my_pytorch_extension.so: undefined symbol: _ZN8pybind116detail11type_casterIN2at6TensorEvE4castERKS3_NS_19return_value_policyENS_6handleE`

This error occurs when using a C++ extension made with torch/extension.h and CMake. It doesn't occurs when using setuptool.
Here is a simple example to reproduce it:

my_torch_extension.cpp:
```cpp
#include <torch/extension.h>  // Cause of the error
#include <pybind11/pybind11.h>

torch::Tensor create_tensor() {
    return torch::rand({2, 3});
}

PYBIND11_MODULE(my_pytorch_extension, m) {
    m.def("create_tensor", &create_tensor, "Create and return a random tensor");
}
```

CMakeLists.txt:
```cmake
cmake_minimum_required(VERSION 3.10)
project(my_pytorch_extension LANGUAGES CXX)

# Specify the path to the Torch package
set(CMAKE_PREFIX_PATH "/home/user/anaconda3/envs/start/lib/python3.12/site-packages/torch")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

# Find the required packages
find_package(Torch REQUIRED)
find_package(pybind11 REQUIRED)

# Create the shared library
add_library(my_pytorch_extension SHARED my_pytorch_extension.cpp)

# Link the Torch and pybind11 libraries
target_link_libraries(my_pytorch_extension PRIVATE "${TORCH_LIBRARIES}" pybind11::module)

# Set the C++ standard
set_property(TARGET my_pytorch_extension PROPERTY CXX_STANDARD 17)
```

test.py:
```py
import torch
from build import my_pytorch_extension # error here

# Try to use the extension
x = my_pytorch_extension.create_tensor()
```

### Versions

PyTorch version: 2.3.0+rocm5.7
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 5.7.31921-d1770ee1b

OS: Linux Mint 21.3 (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.12.1 | packaged by Anaconda, Inc. | (main, Jan 19 2024, 15:51:05) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-121-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: AMD Radeon RX 6800 XT (gfx1030)
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 5.7.31921
MIOpen runtime version: 2.20.0
Is XNNPACK available: True

CPU:
Architecture :                          x86_64
Mode(s) opératoire(s) des processeurs : 32-bit, 64-bit
Address sizes:                          48 bits physical, 48 bits virtual
Boutisme :                              Little Endian
Processeur(s) :                         16
Liste de processeur(s) en ligne :       0-15
Identifiant constructeur :              AuthenticAMD
Nom de modèle :                         AMD Ryzen 7 5700X3D 8-Core Processor
Famille de processeur :                 25
Modèle :                                33
Thread(s) par cœur :                    2
Cœur(s) par socket :                    8
Socket(s) :                             1
Révision :                              2
Frequency boost:                        disabled
Vitesse maximale du processeur en MHz : 3200,0000
Vitesse minimale du processeur en MHz : 2200,0000
BogoMIPS :                              6388.20
Drapaux :                               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm
Virtualisation :                        AMD-V
Cache L1d :                             256 KiB (8 instances)
Cache L1i :                             256 KiB (8 instances)
Cache L2 :                              4 MiB (8 instances)
Cache L3 :                              96 MiB (1 instance)
Nœud(s) NUMA :                          1
Nœud NUMA 0 de processeur(s) :          0-15
Vulnerability Gather data sampling:     Not affected
Vulnerability Itlb multihit:            Not affected
Vulnerability L1tf:                     Not affected
Vulnerability Mds:                      Not affected
Vulnerability Meltdown:                 Not affected
Vulnerability Mmio stale data:          Not affected
Vulnerability Reg file data sampling:   Not affected
Vulnerability Retbleed:                 Not affected
Vulnerability Spec rstack overflow:     Mitigation; safe RET, no microcode
Vulnerability Spec store bypass:        Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:               Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:               Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                    Not affected
Vulnerability Tsx async abort:          Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.3.0+rocm5.7
[pip3] torchaudio==2.3.0+rocm5.7
[pip3] torchvision==0.18.0+rocm5.7
[pip3] triton==2.2.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] torch                     2.3.0+rocm5.7            pypi_0    pypi
[conda] torchaudio                2.3.0+rocm5.7            pypi_0    pypi
[conda] torchvision               0.18.0+rocm5.7           pypi_0    pypi
[conda] triton                    2.2.0                    pypi_0    pypi

cc @malfet @zou3519 @xmfan @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd