Repository: pytorch
Issue: Importing tensorboard jams CUDA device selection
Labels: triaged,module: tensorboard

## 🐛 Bug

On a multi-GPU environment, it is common to set `os.environ['CUDA_VISIBLE_DEVICES']` to select one GPU. However I have found that importing `torch.utils.tensorboard` leads to an unexpected behavior in this context.

I have two Titan Xp GPUs on an Ubuntu 16.04.5 server. Setting `os.environ['CUDA_VISIBLE_DEVICES'] = '1'` had all memory allocations and computations happen precisely on the second Titan GPU, until I integrated tensorboard into my code today. Now with the same code, memory allocation and computation happens on the first Titan GPU.

Below are the screenshots I took. The first Titan GPU is running something, which allocates 8742MB is GPU memory. The second Titan GPU has 10MB allocated by default (I have no idea why). Refer to the reproduction steps below. 

(Screenshot 1)
<img width="1440" alt="스크린샷 2019-09-25 오후 3 58 17" src="https://user-images.githubusercontent.com/29395896/65578067-102af680-dfb0-11e9-8ddf-21eb126c60c2.png">

(Screenshot 2)
<img width="1440" alt="스크린샷 2019-09-25 오후 3 58 55" src="https://user-images.githubusercontent.com/29395896/65578069-11f4ba00-dfb0-11e9-9941-1edb5fd3950d.png">


## To Reproduce

Steps to reproduce the behavior:

1. Run `python reproduction_script.py`.
1. Check that memory is allocated on the second GPU. (Screenshot 1)
1. Uncomment the fourth line `import torch.utils.tensorboard`.
1. Run `python reproduction_script.py` again.
1. Check that memory is allocated on the *first* GPU. (Screenshot 2)

Below is the `reproduction_script.py` code:
```python
import os

import torch
# import torch.utils.tensorboard

os.environ['CUDA_VISIBLE_DEVICES'] = '1'
device = torch.cuda.current_device()

gpu_tensor = torch.ones((200, 200, 200), device=device)

input('Enter anything to terminate script.')
```

## Expected behavior

I expect that setting `os.environ['CUDA_VISIBLE_DEVICES']` to the index of a specific GPU and allocating tensors by the `device` name returned calling `torch.cuda.current_device()` have all memory allocations and computations happen on that specific GPU.

## Environment

```
Collecting environment information...
PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: Could not collect

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: TITAN X (Pascal)
GPU 1: TITAN X (Pascal)

Nvidia driver version: 418.56
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.16.4
[pip] torch==1.2.0
[pip] torchvision==0.4.0a0+6b959ee
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.4                      243
[conda] mkl-service               2.3.0            py36he904b0f_0
[conda] mkl_fft                   1.0.14           py36ha843d7b_0
[conda] mkl_random                1.0.2            py36hd81dba3_0
[conda] pytorch                   1.2.0           py3.6_cuda10.0.130_cudnn7.6.2_0    pytorch
[conda] torchvision               0.4.0                py36_cu100    pytorch
```

## Additional context

`conda list tb-nightly` prints
```
# packages in environment at /home/jaywonchung/anaconda3/envs/meta:
#
# Name                    Version                   Build  Channel
tb-nightly                2.1.0a20190924           pypi_0    pypi
```