Repository: pytorch
Issue: Unable to build and use libtorch function via pybind11: undefined symbol error upon import
Labels: module: cpp,triaged,module: mkldnn

### 🐛 Describe the bug

I am attempting to write a C++ function in libtorch, compile it with pybind11 and then import the .so file in a python file.

The build seems to work fine (no errors are emitted) but, when I try to import it into my python file, execution fails with the error message:
**ImportError: /home/cppNpython/MyProj/cpp/build/srcfile.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2_jS2_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE**

This is my directory structure:
```bash
|-- MyProj
|   |-- cpp
|   |   |-- build
|   |   |   `-- __init__.py
|   |   |-- include
|   |   |   `-- srcfile.h
|   |   |-- lib
|   |   |   |-- libtorch
|   |   |   `-- pybind11
|   |   |-- source
|   |   |   `-- srcfile.cpp
|   |   `-- CMakeLists.txt
|   `-- __init__.py
`-- test.py
```

Here are the contents of each file:-
**srcfile.cpp**

```c++
#include <srcfile.h>

torch::Tensor test(torch::Tensor t1, torch::Tensor t2) {
  return t1;
}

PYBIND11_MODULE(srcfile, m) {
    m.doc() = "demo function"; // optional module docstring
    m.def("test", &test, "A function giving torch tensor");
}
```

**srcfile.h**
```c++
#ifndef __SRCFILE_H__
#define __SRCFILE_H__

#include <torch/torch.h>
#include <pybind11/pybind11.h>

torch::Tensor test(torch::Tensor, torch::Tensor);

#endif
```

**CMakeLists.txt**
```bash
project(srcfile)
cmake_minimum_required(VERSION 3.10)
set(CMAKE_CXX_STANDARD 14)

find_package(PythonLibs REQUIRED)
set(CMAKE_PREFIX_PATH "lib/libtorch/share/cmake/Torch")
find_package(Torch REQUIRED)

include_directories(lib/libtorch)

include_directories(${PYTHON_INCLUDE_DIRS})

add_subdirectory(lib/pybind11)
pybind11_add_module(srcfile src/srcfile.cpp)

target_include_directories(srcfile PRIVATE include/)
target_link_libraries(srcfile PUBLIC ${TORCH_LIBRARIES})

set_property(TARGET srcfile PROPERTY CXX_STANDARD 14)
```

and at last **test.py**

```python
import torch
from MyProj.cpp.build import srcfile as j

def main():
    t1 = torch.rand(2,2)
    t2 = torch.rand(2,2)
    print(j.test(t1, t2))


if __name__ == '__main__':
    main()
```

---------------------------------------------------------------------

Inside the `build` directory I am executing the following command at the terminal:
```bash
cmake .. \
-DPYTHON_INCLUDE_DIR=$(python -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())")  \
-DPYTHON_LIBRARY=$(python -c "import distutils.sysconfig as sysconfig; print(sysconfig.get_config_var('LIBDIR'))") \
-DTORCH_LIBRARIES=lib/libtorch && make
```
which does not emit any error.
I get the error only when I am attempting to import.

I am not sure what I am doing wrong here. Any help/suggestion is welcome.

### Versions

```bash
PyTorch version: 1.10.2
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.9

Python version: 3.7.0 (default, Oct  9 2018, 10:31:47)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-4.15.0-162-generic-x86_64-with-debian-buster-sid
Is CUDA available: False
CUDA runtime version: No CUDA
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.21.5
[pip3] torch==1.10.2
[conda] blas                      1.0                         mkl  
[conda] cpuonly                   2.0                           0    pytorch
[conda] mkl                       2022.0.1           h06a4308_117  
[conda] numpy                     1.21.5                   pypi_0    pypi
[conda] pytorch                   1.10.2              py3.7_cpu_0    pytorch
[conda] pytorch-mutex             1.0                         cpu    pytorch
```

cc @jbschlosser @gujinghui @PenghuiCheng @XiaobingSuper @jianyuh @VitalyFedyunin