Repository: pytorch
Issue: (LLD 8.0.0) ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::utils::computation_cache
Labels: module: build,triaged,module: static linking,module: mkldnn

There's something wrong with `TORCH_STATIC` when building with MKL DNN.

Steps to reproduce:
1. Apply this diff to turn on static build:

```
diff --git a/caffe2/CMakeLists.txt b/caffe2/CMakeLists.txt
index 4b45b7bc8..b6e59c509 100644
--- a/caffe2/CMakeLists.txt
+++ b/caffe2/CMakeLists.txt
@@ -554,6 +554,7 @@ if (TORCH_STATIC)
 else()
   add_library(torch SHARED ${DUMMY_EMPTY_FILE})
 endif()
+add_library(torch_static STATIC ${DUMMY_EMPTY_FILE})
 
 
 target_link_libraries(torch caffe2)
```

Run build with `USE_MKLDNN=1`:

```
[1/8] Linking CXX executable bin/FileStoreTest
[2/8] Linking CXX executable bin/ProcessGroupMPITest
[3/8] Linking CXX executable bin/ProcessGroupGlooTest
[4/8] Linking CXX executable bin/torch_shm_manager
[5/8] Linking CXX shared library lib/libcaffe2_detectron_ops.so
FAILED: lib/libcaffe2_detectron_ops.so 
: && /private/home/ezyang/ccache/lib/c++ -fPIC -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow -O3  -rdynamic -shared -Wl,-soname,libcaffe2_detectron_ops.so -o lib/libcaffe2_detectron_ops.so modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/group_spatial_softmax_op.cc.o modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/ps_roi_pool_op.cc.o modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/roi_pool_f_op.cc.o modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/sample_as_op.cc.o modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/select_smooth_l1_loss_op.cc.o modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/sigmoid_cross_entropy_loss_op.cc.o modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/sigmoid_focal_loss_op.cc.o modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/smooth_l1_loss_op.cc.o modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/softmax_focal_loss_op.cc.o modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/spatial_narrow_as_op.cc.o modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/upsample_nearest_op.cc.o -L/scratch/ezyang/pytorch-tmp-env/lib -Wl,-rpath,/scratch/ezyang/pytorch-tmp/build/lib:/scratch/ezyang/pytorch-tmp-env/lib: lib/libcaffe2.so /usr/lib/gcc/x86_64-linux-gnu/7/libgomp.so -lpthread lib/libprotobuf.a lib/libc10.so -lpthread -lmkl_intel_lp64 -lmkl_gnu_thread -lmkl_core -fopenmp -lm /usr/lib/x86_64-linux-gnu/libdl.so lib/libmkldnn.a -lpthread && :
ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store()::t_store_ in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store() (.part.760))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store()::t_store_ in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store() (.part.760))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store()::t_store_ in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store() (.part.760))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store()::t_store_ in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store() (.part.760))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store()::t_store_ in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store() (.part.760))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store()::t_store_ in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store() (.part.760))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store()::t_store_ in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store() (.part.760))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store()::t_store_ in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store() (.part.760))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store()::t_store_ in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store() (.part.760))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store()::t_store_ in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store() (.part.760))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store()::t_store_ in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store() (.part.760))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store()::t_store_ in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store() (.part.760))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: guard variable for ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store()::t_store_ in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::utils::computation_cache<ideep::tensor::reorder, 1024ul, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::t_store() (.part.760))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: guard variable for ideep::stream::default_stream()::s in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::tensor::reorder::operator()(ideep::tensor const&, ideep::tensor const&))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::stream::default_stream()::s in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::tensor::reorder::operator()(ideep::tensor const&, ideep::tensor const&))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::stream::default_stream()::s in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::tensor::reorder::operator()(ideep::tensor const&, ideep::tensor const&))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::stream::default_stream()::s in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::tensor::reorder::operator()(ideep::tensor const&, ideep::tensor const&))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::stream::default_stream()::s in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::tensor::reorder::operator()(ideep::tensor const&, ideep::tensor const&))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::stream::default_stream()::s in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::tensor::reorder::operator()(ideep::tensor const&, ideep::tensor const&))

ld: error: can't create dynamic relocation R_X86_64_DTPOFF32 against symbol: ideep::stream::default_stream()::s in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o
>>> referenced by batch_permutation_op.cc
>>>               modules/detectron/CMakeFiles/caffe2_detectron_ops.dir/batch_permutation_op.cc.o:(ideep::tensor::reorder::operator()(ideep::tensor const&, ideep::tensor const&))

ld: error: too many errors emitted, stopping now (use -error-limit=0 to see all errors)
collect2: error: ld returned 1 exit status
[6/8] Linking CXX executable bin/TCPStoreTest
[7/8] Linking CXX shared library lib/libtorch_python.so
ninja: build stopped: subcommand failed.
Building wheel torch-1.2.0a0+75faa72
-- Building version 1.2.0a0+75faa72
cmake --build . --target install --config Release -- -j 48
Traceback (most recent call last):
  File "setup.py", line 752, in <module>
    build_deps()
  File "setup.py", line 320, in build_deps
    build_dir='build')
  File "/scratch/ezyang/pytorch-tmp/tools/build_pytorch_libs.py", line 70, in build_caffe2
    cmake.build(my_env)
  File "/scratch/ezyang/pytorch-tmp/tools/setup_helpers/cmake.py", line 287, in build
    self.run(build_args, my_env)
  File "/scratch/ezyang/pytorch-tmp/tools/setup_helpers/cmake.py", line 100, in run
    check_call(command, cwd=self._build_dir, env=env)
  File "/scratch/ezyang/pytorch-tmp-env/lib/python3.7/subprocess.py", line 341, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '48']' returned non-zero exit status 1.
```

If you turn off MKLDNN the problem goes away. I don't have time to investigate, so filing a bug here.

cc @gujinghui @PenghuiCheng @XiaobingSuper @jianyuh