Repository: pytorch
Issue: RuntimeError on PyTorch 1.2 under NVIDIA Nsight Systems
Labels: module: cuda,triaged,module: third_party

## 🐛 Bug

Creating a pinned tensor in PyTorch 1.2 fails with `RuntimeError: Error in dlopen or dlsym: libcaffe2_nvrtc.so...` under NVIDIA Nsight Systems.

I tested both PyTorch versions 1.2 and 1.1. This problem happened only in PyTorch 1.2.

## To Reproduce

Steps to reproduce the behavior:

1. Install PyTorch 1.2
1. Install NVIDIA Nsight Systems 2019.3
1. Run `nsys profile -snone python -c 'import torch; torch.rand(100, pin_memory=True)'`.

```console
$ nsys profile -snone python -c 'import torch; torch.rand(100, pin_memory=True)'
WARNING: Backtraces will not be collected because sampling is disabled.

**** collection configuration ****
        force-overwrite = false
        stop-on-exit = true
        export_sqlite = false
        stats = false
        delay = 0 seconds
        duration = 0 seconds
        inherit-environment = true
        show-output = true
        trace-fork-before-exec = false
        sample_cpu = false
        backtrace_method = LBR
        trace_cublas = false
        trace_cuda = true
        trace_cudnn = false
        trace_nvtx = true
        trace_openacc = false
        trace_vulkan = false
        trace_opengl = true
        trace_osrt = true
        osrt-threshold = 0 nanoseconds
        profile_processes = tree
        application command = python
        application arguments = -c import torch; torch.rand(100, pin_memory=True)
        application working directory = /home/sublee
        environment variables:
        Collecting data...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
RuntimeError: Error in dlopen or dlsym: libcaffe2_nvrtc.so: cannot open shared object file: No such file or directory
        Generating the  file.
        Capturing raw events...
        2878 total events collected.                                                                                                                                                                                                                                               
        Saving diagnostics...
        Saving qdstrm file to disk...
        Finished saving file.


Importing the qdstrm file using /opt/nvidia/nsightsystems/nsightsystems-cli-2019.3.6/Host-x86_64/QdstrmImporter.

Importing...

Importing [==================================================100%]
Saving report to file "/home/sublee/nvidia_nsight_systems/report19.qdrep"
Report file saved.
Please discard the qdstrm file and use the qdrep file instead.

Removed /home/sublee/nvidia_nsight_systems/report19.qdstrm as it was successfully imported.
Please use the qdrep file instead.
```

`torch.rand(100, pin_memory=True)` failed with this error:

```python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
RuntimeError: Error in dlopen or dlsym: libcaffe2_nvrtc.so: cannot open shared object file: No such file or directory
```

## Expected behavior

`torch.rand(100, pin_memory=True)` should not fail under NVIDIA Nsight Systems.

- It doesn't fail in *PyTorch 1.1* under NVIDIA Nsight Systems.
- It doesn't fail in PyTorch 1.2 *without* NVIDIA Nsight Systems.

## Environment

```
Collecting environment information...
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.2 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: Tesla P40
GPU 1: Tesla P40
GPU 2: Tesla P40
GPU 3: Tesla P40
GPU 4: Tesla P40
GPU 5: Tesla P40
GPU 6: Tesla P40
GPU 7: Tesla P40

Nvidia driver version: 410.104
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.2

Versions of relevant libraries:
[pip3] numpy==1.16.4
[pip3] torch==1.2.0
[pip3] torchvision==0.4.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.4                      243
[conda] mkl-include               2019.4                      243
[conda] mkl-service               2.0.2            py36h7b6447c_0
[conda] mkl_fft                   1.0.12           py36ha843d7b_0
[conda] mkl_random                1.0.2            py36hd81dba3_0
[conda] torch                     1.2.0                    pypi_0    pypi
[conda] torchvision               0.4.0                    pypi_0    pypi
```

## Additional context

### Environment variables to reproduce

I tried to run the Python code with `LD_PRELOAD` and `QUADD_INJECTION_PROXY` set by NVIDIA Nsight Systems. The problem was simply reproduced.

```console
$ export LD_PRELOAD=/opt/nvidia/nsightsystems/nsightsystems-cli-2019.3.6/Target-x86_64/x86_64/libToolsInjectionProxy64.so
$ export QUADD_INJECTION_PROXY=OSRT
$ python -c 'import torch; torch.rand(100, pin_memory=True)'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
RuntimeError: Error in dlopen or dlsym: libcaffe2_nvrtc.so: cannot open shared object file: No such file or directory
```

<!--
### Specify `libcaffe2_nvrtc.so` path

In my system, `libcaffe2_nvrtc.so` is at `/opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2_nvrtc.so`. I specified the path at `LD_PRELOAD` when running NVIDIA Nsight Systems. Then the problem has been fixed.

```console
$ nsys profile -eLD_PRELOAD=/opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2_nvrtc.so -snone python -c 'import torch; torch.rand(100, pin_memory=True)'
WARNING: Backtraces will not be collected because sampling is disabled.

**** collection configuration ****
        force-overwrite = false
        stop-on-exit = true
        export_sqlite = false
        stats = false
        delay = 0 seconds
        duration = 0 seconds
        inherit-environment = true
        show-output = true
        trace-fork-before-exec = false
        sample_cpu = false
        backtrace_method = LBR
        trace_cublas = false
        trace_cuda = true
        trace_cudnn = false
        trace_nvtx = true
        trace_openacc = false
        trace_vulkan = false
        trace_opengl = true
        trace_osrt = true
        osrt-threshold = 0 nanoseconds
        profile_processes = tree
        application command = python
        application arguments = -c import torch; torch.rand(100, pin_memory=True)
        application working directory = /home/sublee
        environment variables:
                LD_PRELOAD=/opt/conda/lib/python3.6/site-packages/torch/lib/libcaffe2_nvrtc.so
        Collecting data...
        Generating the  file.
        Capturing raw events...
        4597 total events collected.
        Saving diagnostics...
        Saving qdstrm file to disk...
        Finished saving file.


Importing the qdstrm file using /opt/nvidia/nsightsystems/nsightsystems-cli-2019.3.6/Host-x86_64/QdstrmImporter.

Importing...

Importing [==================================================100%]
Saving report to file "/home/sublee/nvidia_nsight_systems/report20.qdrep"
Report file saved.
Please discard the qdstrm file and use the qdrep file instead.

Removed /home/sublee/nvidia_nsight_systems/report20.qdstrm as it was successfully imported.
Please use the qdrep file instead.
```
-->