Repository: pytorch
Issue: libtorch static linking results in undefined references to onnx_torch and caffe2::EmbeddingLookup
Labels: triaged,module: undefined reference

## 🐛 Bug

Many `undefined reference` errors appear when statically linking libtorch.

We build libtorch (pytorch version 1.7.1) by `python setup.py install --cmake-only`, then setting `BUILD_SHARED_LIBS=OFF`, and then `python setup.py install`.

We link libtorch by:
```
target_link_libraries(program PUBLIC
        -Wl,--whole-archive ${TORCH_LIBRARIES}
        -Wl,--no-whole-archive
        /usr/local/libtorch/lib/libtorch_cuda.a
        /usr/local/libtorch/lib/libsleef.a
        /usr/local/libtorch/lib/libprotobuf.a
        /usr/local/libtorch/lib/libcaffe2_protos.a
        /usr/local/libtorch/lib/libcaffe2_observers.a
        /usr/local/libtorch/lib/libdnnl.a
        ...
)
# TORCH_LIBRARIES: torch;/usr/local/libtorch/lib/libc10.a;/usr/local/libtorch/lib/libtorch_cpu.a;/usr/local/libtorch/lib/libnnpack.a;/usr/local/libtorch/lib/libpthreadpool.a;/usr/local/libtorch/lib/libcpuinfo.a;/usr/local/libtorch/lib/libclog.a;/usr/local/cuda/lib64/libnvToolsExt.so;/usr/local/libtorch/lib/libc10_cuda.a
```

We get hundreds of errors such as:
```
/usr/bin/ld: /usr/local/libtorch/lib/libtorch_cpu.a(embedding_lookup.cc.o): in function `caffe2::EmbeddingLookup_int64_t_float_float_false(long, long, long, long, float const*, long const*, int const*, float const*, float const*, bool, float*)':
embedding_lookup.cc:(.text+0x1c3e): undefined reference to `caffe2::EmbeddingLookup_int64_t_float_float_false__avx2_fma(long, long, long, long, float const*, long const*, int const*, float const*, float const*, bool, float*)'
/usr/bin/ld: /usr/local/libtorch/lib/libtorch_cpu.a(fused_8bit_rowwise_embedding_lookup_idx.cc.o): in function `caffe2::Fused8BitRowwiseEmbeddingLookupIdx_int32_t_uint8_t_float(long, long, long, long, unsigned char const*, int const*, long const*, float const*, bool, float*)':
fused_8bit_rowwise_embedding_lookup_idx.cc:(.text+0x4e8): undefined reference to `caffe2::Fused8BitRowwiseEmbeddingLookupIdx_int32_t_uint8_t_float_false__avx2_fma(long, long, long, long, unsigned char const*, int const*, long const*, float const*, bool, float*)'

/usr/bin/ld: export.cpp:(.text+0x83c4): undefined reference to `onnx_torch::ModelProto::ByteSizeLong() const'
/usr/bin/ld: export.cpp:(.text+0x840e): undefined reference to `onnx_torch::ModelProto::ModelProto(onnx_torch::ModelProto const&)'
/usr/bin/ld: export.cpp:(.text+0x84d8): undefined reference to `onnx_torch::ModelProto::~ModelProto()'
/usr/bin/ld: export.cpp:(.text+0x7993): undefined reference to `onnx_torch::OperatorSetIdProto* google::protobuf::Arena::CreateMaybeMessage<onnx_torch::OperatorSetIdProto>(google::protobuf::Arena*)'
/usr/bin/ld: /usr/local/libtorch/lib/libtorch_cpu.a(export.cpp.o): in function `torch::jit::check_onnx_proto(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) [clone .cold]':
export.cpp:(.text.unlikely+0x439d): undefined reference to `onnx_torch::ModelProto::~ModelProto()'
/usr/bin/ld: /usr/local/libtorch/lib/libtorch_cpu.a(export.cpp.o):export.cpp:(.text.unlikely+0x43cb): more undefined references to `onnx_torch::ModelProto::~ModelProto()' follow
/usr/bin/ld: my_tests: hidden symbol `onnxifi_load' isn't defined
```

Note that when we do not explicitly link `libsleef.a`, `libdnnl.a`, etc. we get many more `undefined reference` errors such as:
```
/usr/bin/ld: Activation.cpp.AVX2.cpp:(.text+0x16e48): undefined reference to `Sleef_expf8_u10'
/usr/bin/ld: momentum_sgd_op.cc:(.text+0x27b): undefined reference to `caffe2::TensorShape::TensorShape()'
/usr/bin/ld: blobs_queue_db.cc:(.text._ZN6caffe215IDEEPFallbackOpINS_2db20CreateBlobsQueueDBOpINS_10CPUContextEEENS_11SkipIndicesIJLi0EEEEE11RunOnDeviceEv[_ZN6caffe215IDEEPFallbackOpINS_2db20CreateBlobsQueueDBOpINS_10CPUContextEEENS_11SkipIndicesIJLi0EEEEE11RunOnDeviceEv]+0xc59): undefined reference to `dnnl_engine_create'
```

If we link without `--whole-archive` (which is suggested by https://github.com/pytorch/pytorch/issues/21737) we get further errors such as:
```
/usr/bin/ld: engine.cpp:(.text+0x3a33): undefined reference to `c10::operator<<(std::ostream&, c10::TensorOptions const&)'
/usr/bin/ld: /usr/local/libtorch/lib/libtorch_cpu.a(engine.cpp.o): in function `c10::impl::getDeviceGuardImpl(c10::DeviceType)':
engine.cpp:(.text._ZN3c104impl18getDeviceGuardImplENS_10DeviceTypeE[_ZN3c104impl18getDeviceGuardImplENS_10DeviceTypeE]+0x2e): undefined reference to `c10::impl::device_guard_impl_registry'
```

## Environment

PyTorch Version: 1.7.1
OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
CMake version: version 3.17.5
CUDA runtime version: 11.0.221
GPU models and configuration: GPU 0: GeForce GTX 1050 Ti with Max-Q Design
Run in Docker `nvidia/cuda:11.0-devel-ubuntu20.04` with cudnn version 8.0.5.39

Thank you for comments.