Repository: pytorch
Issue: TORCH_CUDA_API export failure on torch::cuda::nccl::detail::throw_nccl_error(ncclResult_t)
Labels: high priority,module: binaries,oncall: releng,triaged,module: nccl,quansight-nack

## 🐛 Bug

When compiling from source `pytorch 1.4.0` the resulting library `libtorch_python.so` is left with an undefined symbol for `torch::cuda::nccl::detail::throw_nccl_error(ncclResult_t)` (`ImportError: /usr/lib/python3.8/site-packages/torch/lib/libtorch_python.so: undefined symbol: _ZN5torch4cuda4nccl6detail16throw_nccl_errorE12ncclResult_t`)

## To Reproduce

Steps to reproduce the behavior:

1. compile pytorch from source with CUDA and NCCL
2. try to import torch in python
3. get following error
```python
import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/site-packages/torch/__init__.py", line 81, in <module>
    from torch._C import *
ImportError: /usr/lib/python3.8/site-packages/torch/lib/libtorch_python.so: undefined symbol: _ZN5torch4cuda4nccl6detail16throw_nccl_errorE12ncclResult_t
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior

symbol should not be missing from shared library and python should be able to `import torch` successfully

## Environment

 - PyTorch Version (e.g., 1.0): 1.4.0
 - OS (e.g., Linux): ArchLinux x86_64
 - How you installed PyTorch (`conda`, `pip`, source): source
 - Build command you used (if compiling from source):
```bash
  git submodule update --init --recursive
  export VERBOSE=1
  export PYTORCH_BUILD_VERSION=1.4.0
  export PYTORCH_BUILD_NUMBER=1

  export USE_MKLDNN=OFF
  export USE_FFMPEG=ON
  export USE_GFLAGS=ON
  export USE_GLOG=ON
  export BUILD_BINARY=ON
  export USE_OPENCV=ON
  export USE_SYSTEM_NCCL=ON
  export NCCL_ROOT=/usr
  export NCCL_LIB_DIR=/usr/lib
  export NCCL_INCLUDE_DIR=/usr/include
  export NCCL_VERSION=$(pkg-config nccl --modversion)
  export NCCL_VER_CODE=$(sed -n 's/^#define NCCL_VERSION_CODE\s*\(.*\).*/\1/p' /usr/include/nccl.h)
  export CUDAHOSTCXX=g++-8
  export CUDA_HOME=/opt/cuda
  export CUDNN_LIB_DIR=/usr/lib
  export CUDNN_INCLUDE_DIR=/usr/include
  export TORCH_NVCC_FLAGS="-Xfatbin -compress-all"
  export TORCH_CUDA_ARCH_LIST="3.2;3.5;3.7;5.0;5.2;5.3;6.0;6.0+PTX;6.1;6.1+PTX;6.2;6.2+PTX;7.0;7.0+PTX;7.2;7.2+PTX;7.5;7.5+PTX"
  echo "Building with cuda and with non-x86-64 optimizations"
  export USE_CUDA=1
  export USE_CUDNN=1
  echo "add_definitions(-march=haswell)" >> cmake/MiscCheck.cmake
  python setup.py build
```
 - Python version: 3.8.1
 - CUDA/cuDNN version: 10.2.89 / 7.6.5.32
 - GPU models and configuration: NVIDIA GTX 980
 - Any other relevant information:

## Additional context

```bash
$ LD_PRELOAD=/usr/lib/libnccl.so.2.5.7:/usr/lib/libpython3.8.so.1.0 ldd -d /usr/lib/python3.8/site-packages/torch/lib/libtorch_python.so
	linux-vdso.so.1 (0x00007ffc73fac000)
	/usr/lib/libnccl.so.2.5.7 (0x00007f808dd5c000)
	/usr/lib/libpython3.8.so.1.0 (0x00007f808d99d000)
	libshm.so => /usr/lib/python3.8/site-packages/torch/lib/libshm.so (0x00007f808d993000)
	libnvToolsExt.so.1 => /opt/cuda/lib/libnvToolsExt.so.1 (0x00007f808d78a000)
	libtorch.so => /usr/lib/python3.8/site-packages/torch/lib/libtorch.so (0x00007f80485c2000)
	libc10_cuda.so => /usr/lib/python3.8/site-packages/torch/lib/libc10_cuda.so (0x00007f8048599000)
	libc10.so => /usr/lib/python3.8/site-packages/torch/lib/libc10.so (0x00007f8048561000)
	libglog.so.0 => /usr/lib/libglog.so.0 (0x00007f8048500000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f80484de000)
	libcudart.so.10.2 => /opt/cuda/lib/libcudart.so.10.2 (0x00007f8048260000)
	libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f8048076000)
	libm.so.6 => /usr/lib/libm.so.6 (0x00007f8047f30000)
	libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f8047f14000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f8047d4d000)
	/usr/lib64/ld-linux-x86-64.so.2 (0x00007f80980f0000)
	librt.so.1 => /usr/lib/librt.so.1 (0x00007f8047d42000)
	libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f8047d3d000)
	libutil.so.1 => /usr/lib/libutil.so.1 (0x00007f8047d38000)
	libgflags.so.2.2 => /usr/lib/libgflags.so.2.2 (0x00007f8047d11000)
	libcufft.so.10 => /opt/cuda/lib/libcufft.so.10 (0x00007f803e86f000)
	libcurand.so.10 => /opt/cuda/lib/libcurand.so.10 (0x00007f803a7cc000)
	libcublas.so.10 => /opt/cuda/lib/libcublas.so.10 (0x00007f8036516000)
	libcudnn.so.7 => /usr/lib/libcudnn.so.7 (0x00007f801c9a9000)
	libgomp.so.1 => /usr/lib/libgomp.so.1 (0x00007f801c970000)
	libopencv_highgui.so.4.2 => /usr/lib/libopencv_highgui.so.4.2 (0x00007f801c926000)
	libopencv_optflow.so.4.2 => /usr/lib/libopencv_optflow.so.4.2 (0x00007f801c8a3000)
	libopencv_videoio.so.4.2 => /usr/lib/libopencv_videoio.so.4.2 (0x00007f801c820000)
	libavcodec.so.58 => /usr/lib/libavcodec.so.58 (0x00007f801b2b9000)
	libavformat.so.58 => /usr/lib/libavformat.so.58 (0x00007f801b055000)
	libavutil.so.56 => /usr/lib/libavutil.so.56 (0x00007f801af37000)
	libswscale.so.5 => /usr/lib/libswscale.so.5 (0x00007f801aeaa000)
	libblas.so.3 => /usr/lib/libblas.so.3 (0x00007f801ae51000)
	liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00007f801a763000)
	libcusparse.so.10 => /opt/cuda/lib64/libcusparse.so.10 (0x00007f8012afd000)
	libmagma.so => /usr/lib/libmagma.so (0x00007f7ffd2c0000)
	libopencv_ximgproc.so.4.2 => /usr/lib/libopencv_ximgproc.so.4.2 (0x00007f7ffd052000)
	libopencv_imgcodecs.so.4.2 => /usr/lib/libopencv_imgcodecs.so.4.2 (0x00007f7ffcac1000)
	libopencv_video.so.4.2 => /usr/lib/libopencv_video.so.4.2 (0x00007f7ffca44000)
	libopencv_calib3d.so.4.2 => /usr/lib/libopencv_calib3d.so.4.2 (0x00007f7ffc86d000)
	libopencv_features2d.so.4.2 => /usr/lib/libopencv_features2d.so.4.2 (0x00007f7ffc26a000)
	libopencv_imgproc.so.4.2 => /usr/lib/libopencv_imgproc.so.4.2 (0x00007f7ffab64000)
	libopencv_flann.so.4.2 => /usr/lib/libopencv_flann.so.4.2 (0x00007f7ffaaf7000)
	libopencv_core.so.4.2 => /usr/lib/libopencv_core.so.4.2 (0x00007f7ff99f4000)
	libcublasLt.so.10 => /opt/cuda/lib/libcublasLt.so.10 (0x00007f7ff7b5f000)
	libGL.so.1 => /usr/lib/libGL.so.1 (0x00007f7ff7ad9000)
	libQt5Test.so.5 => /usr/lib/libQt5Test.so.5 (0x00007f7ff7a84000)
	libQt5OpenGL.so.5 => /usr/lib/libQt5OpenGL.so.5 (0x00007f7ff7a2c000)
	libQt5Widgets.so.5 => /usr/lib/libQt5Widgets.so.5 (0x00007f7ff738a000)
	libQt5Gui.so.5 => /usr/lib/libQt5Gui.so.5 (0x00007f7ff6ce2000)
	libQt5Core.so.5 => /usr/lib/libQt5Core.so.5 (0x00007f7ff67aa000)
	libdc1394.so.25 => /usr/lib/libdc1394.so.25 (0x00007f7ff6731000)
	libgstreamer-1.0.so.0 => /usr/lib/libgstreamer-1.0.so.0 (0x00007f7ff6623000)
	libgobject-2.0.so.0 => /usr/lib/libgobject-2.0.so.0 (0x00007f7ff65c7000)
	libglib-2.0.so.0 => /usr/lib/libglib-2.0.so.0 (0x00007f7ff64a0000)
	libgstapp-1.0.so.0 => /usr/lib/libgstapp-1.0.so.0 (0x00007f7ff6490000)
	libgstriff-1.0.so.0 => /usr/lib/libgstriff-1.0.so.0 (0x00007f7ff6480000)
	libgstpbutils-1.0.so.0 => /usr/lib/libgstpbutils-1.0.so.0 (0x00007f7ff644a000)
	libswresample.so.3 => /usr/lib/libswresample.so.3 (0x00007f7ff642a000)
	libvpx.so.6 => /usr/lib/libvpx.so.6 (0x00007f7ff6108000)
	libwebpmux.so.3 => /usr/lib/libwebpmux.so.3 (0x00007f7ff60fa000)
	libwebp.so.7 => /usr/lib/libwebp.so.7 (0x00007f7ff608a000)
	liblzma.so.5 => /usr/lib/liblzma.so.5 (0x00007f7ff6062000)
	libdav1d.so.3 => /usr/lib/libdav1d.so.3 (0x00007f7ff5f42000)
	libopencore-amrwb.so.0 => /usr/lib/libopencore-amrwb.so.0 (0x00007f7ff5f2c000)
	libz.so.1 => /usr/lib/libz.so.1 (0x00007f7ff5f12000)
	libaom.so.0 => /usr/lib/libaom.so.0 (0x00007f7ff5abb000)
	libgsm.so.1 => /usr/lib/libgsm.so.1 (0x00007f7ff58af000)
	libmp3lame.so.0 => /usr/lib/libmp3lame.so.0 (0x00007f7ff5636000)
	libopencore-amrnb.so.0 => /usr/lib/libopencore-amrnb.so.0 (0x00007f7ff560b000)
	libopenjp2.so.7 => /usr/lib/libopenjp2.so.7 (0x00007f7ff55ae000)
	libopus.so.0 => /usr/lib/libopus.so.0 (0x00007f7ff5550000)
	libspeex.so.1 => /usr/lib/libspeex.so.1 (0x00007f7ff5533000)
	libtheoraenc.so.1 => /usr/lib/libtheoraenc.so.1 (0x00007f7ff52f4000)
	libtheoradec.so.1 => /usr/lib/libtheoradec.so.1 (0x00007f7ff50da000)
	libvorbis.so.0 => /usr/lib/libvorbis.so.0 (0x00007f7ff4eae000)
	libvorbisenc.so.2 => /usr/lib/libvorbisenc.so.2 (0x00007f7ff4c05000)
	libx264.so.159 => /usr/lib/libx264.so.159 (0x00007f7ff4949000)
	libx265.so.179 => /usr/lib/libx265.so.179 (0x00007f7ff36be000)
	libxvidcore.so.4 => /usr/lib/libxvidcore.so.4 (0x00007f7ff35ab000)
	libva.so.2 => /usr/lib/libva.so.2 (0x00007f7ff3586000)
	libmfx.so.1 => /usr/lib/libmfx.so.1 (0x00007f7ff3578000)
	libxml2.so.2 => /usr/lib/libxml2.so.2 (0x00007f7ff3412000)
	libbz2.so.1.0 => /usr/lib/libbz2.so.1.0 (0x00007f7ff33ff000)
	libmodplug.so.1 => /usr/lib/libmodplug.so.1 (0x00007f7ff3272000)
	libbluray.so.2 => /usr/lib/libbluray.so.2 (0x00007f7ff3222000)
	libgmp.so.10 => /usr/lib/libgmp.so.10 (0x00007f7ff318f000)
	libgnutls.so.30 => /usr/lib/libgnutls.so.30 (0x00007f7ff2fc5000)
	libssh.so.4 => /usr/lib/libssh.so.4 (0x00007f7ff2f5b000)
	libva-drm.so.2 => /usr/lib/libva-drm.so.2 (0x00007f7ff2f56000)
	libva-x11.so.2 => /usr/lib/libva-x11.so.2 (0x00007f7ff2f4c000)
	libvdpau.so.1 => /usr/lib/libvdpau.so.1 (0x00007f7ff2f47000)
	libX11.so.6 => /usr/lib/libX11.so.6 (0x00007f7ff2e07000)
	libdrm.so.2 => /usr/lib/libdrm.so.2 (0x00007f7ff2df2000)
	libgfortran.so.5 => /usr/lib/libgfortran.so.5 (0x00007f7ff2b52000)
	libjpeg.so.8 => /usr/lib/libjpeg.so.8 (0x00007f7ff2ab9000)
	libpng16.so.16 => /usr/lib/libpng16.so.16 (0x00007f7ff2a81000)
	libtiff.so.5 => /usr/lib/libtiff.so.5 (0x00007f7ff29f2000)
	libjasper.so.4 => /usr/lib/libjasper.so.4 (0x00007f7ff2980000)
	libIlmImf-2_4.so.24 => /usr/lib/libIlmImf-2_4.so.24 (0x00007f7ff268f000)
	libtbb.so.2 => /usr/lib/libtbb.so.2 (0x00007f7ff2647000)
	libcblas.so.3 => /usr/lib/libcblas.so.3 (0x00007f7ff262b000)
	libGLdispatch.so.0 => /usr/lib/libGLdispatch.so.0 (0x00007f7ff2574000)
	libGLX.so.0 => /usr/lib/libGLX.so.0 (0x00007f7ff2541000)
	libmd4c.so.0 => /usr/lib/libmd4c.so.0 (0x00007f7ff252b000)
	libharfbuzz.so.0 => /usr/lib/libharfbuzz.so.0 (0x00007f7ff2425000)
	libsystemd.so.0 => /usr/lib/libsystemd.so.0 (0x00007f7ff237e000)
	libdouble-conversion.so.3 => /usr/lib/libdouble-conversion.so.3 (0x00007f7ff2367000)
	libicui18n.so.65 => /usr/lib/libicui18n.so.65 (0x00007f7ff2073000)
	libicuuc.so.65 => /usr/lib/libicuuc.so.65 (0x00007f7ff1e91000)
	libpcre2-16.so.0 => /usr/lib/libpcre2-16.so.0 (0x00007f7ff1e0d000)
	libzstd.so.1 => /usr/lib/libzstd.so.1 (0x00007f7ff1d61000)
	libraw1394.so.11 => /usr/lib/libraw1394.so.11 (0x00007f7ff1d4f000)
	libusb-1.0.so.0 => /usr/lib/libusb-1.0.so.0 (0x00007f7ff1d33000)
	libgmodule-2.0.so.0 => /usr/lib/libgmodule-2.0.so.0 (0x00007f7ff1d2e000)
	libunwind.so.8 => /usr/lib/libunwind.so.8 (0x00007f7ff1d14000)
	libdw.so.1 => /usr/lib/libdw.so.1 (0x00007f7ff1cc0000)
	libffi.so.6 => /usr/lib/libffi.so.6 (0x00007f7ff1cb3000)
	libpcre.so.1 => /usr/lib/libpcre.so.1 (0x00007f7ff1c40000)
	libgstbase-1.0.so.0 => /usr/lib/libgstbase-1.0.so.0 (0x00007f7ff1bd0000)
	libgstaudio-1.0.so.0 => /usr/lib/libgstaudio-1.0.so.0 (0x00007f7ff1b5e000)
	libgsttag-1.0.so.0 => /usr/lib/libgsttag-1.0.so.0 (0x00007f7ff1b24000)
	libgstvideo-1.0.so.0 => /usr/lib/libgstvideo-1.0.so.0 (0x00007f7ff1a83000)
	libsoxr.so.0 => /usr/lib/libsoxr.so.0 (0x00007f7ff180d000)
	libogg.so.0 => /usr/lib/libogg.so.0 (0x00007f7ff1802000)
	libmvec.so.1 => /usr/lib/libmvec.so.1 (0x00007f7ff17d6000)
	libfontconfig.so.1 => /usr/lib/libfontconfig.so.1 (0x00007f7ff178e000)
	libfreetype.so.6 => /usr/lib/libfreetype.so.6 (0x00007f7ff16be000)
	libp11-kit.so.0 => /usr/lib/libp11-kit.so.0 (0x00007f7ff1595000)
	libidn2.so.0 => /usr/lib/libidn2.so.0 (0x00007f7ff1574000)
	libunistring.so.2 => /usr/lib/libunistring.so.2 (0x00007f7ff13f2000)
	libtasn1.so.6 => /usr/lib/libtasn1.so.6 (0x00007f7ff13dc000)
	libnettle.so.7 => /usr/lib/libnettle.so.7 (0x00007f7ff13a2000)
	libhogweed.so.5 => /usr/lib/libhogweed.so.5 (0x00007f7ff1367000)
	libcrypto.so.1.1 => /usr/lib/libcrypto.so.1.1 (0x00007f7ff1096000)
	libXext.so.6 => /usr/lib/libXext.so.6 (0x00007f7ff1081000)
	libXfixes.so.3 => /usr/lib/libXfixes.so.3 (0x00007f7ff1078000)
	libxcb.so.1 => /usr/lib/libxcb.so.1 (0x00007f7ff104e000)
	libquadmath.so.0 => /usr/lib/../lib/libquadmath.so.0 (0x00007f7ff1001000)
	libImath-2_4.so.24 => /usr/lib/libImath-2_4.so.24 (0x00007f7ff0fe8000)
	libIlmThread-2_4.so.24 => /usr/lib/libIlmThread-2_4.so.24 (0x00007f7ff0fde000)
	libHalf-2_4.so.24 => /usr/lib/libHalf-2_4.so.24 (0x00007f7ff0f99000)
	libIex-2_4.so.24 => /usr/lib/libIex-2_4.so.24 (0x00007f7ff0f44000)
	libgraphite2.so.3 => /usr/lib/libgraphite2.so.3 (0x00007f7ff0f1d000)
	liblz4.so.1 => /usr/lib/liblz4.so.1 (0x00007f7ff0efb000)
	libgcrypt.so.20 => /usr/lib/libgcrypt.so.20 (0x00007f7ff0ddd000)
	libicudata.so.65 => /usr/lib/libicudata.so.65 (0x00007f7fef32c000)
	libudev.so.1 => /usr/lib/libudev.so.1 (0x00007f7fef301000)
	libelf.so.1 => /usr/lib/libelf.so.1 (0x00007f7fef2e5000)
	liborc-0.4.so.0 => /usr/lib/liborc-0.4.so.0 (0x00007f7fef263000)
	libexpat.so.1 => /usr/lib/libexpat.so.1 (0x00007f7fef233000)
	libXau.so.6 => /usr/lib/libXau.so.6 (0x00007f7fef22e000)
	libXdmcp.so.6 => /usr/lib/libXdmcp.so.6 (0x00007f7fef226000)
	libgpg-error.so.0 => /usr/lib/libgpg-error.so.0 (0x00007f7fef201000)
  undefined symbol: _ZN5torch4cuda4nccl6detail16throw_nccl_errorE12ncclResult_t	(/usr/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
```
```bash
$ nm -gDC /usr/lib/python3.8/site-packages/torch/lib/libtorch_python.so | grep -i  torch::cuda::nccl::detail::throw
     U torch::cuda::nccl::detail::throw_nccl_error(ncclResult_t)
```


cc @ezyang @gchanan @zou3519 @seemethere @ngimel