Repository: pytorch
Issue: torch.cuda.device_count() returns 1 using 4 TitanX setup.
Labels: needs reproduction,module: cuda,triaged

nvidia-smi:

```
+-----------------------------------------------------------------------------+                            │
| NVIDIA-SMI 384.66                 Driver Version: 384.66                    |                            │
|-------------------------------+----------------------+----------------------+                            │
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |                            │
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |                            │ |===============================+======================+======================|                            │
|   0  GeForce GTX 108...  On   | 00000000:02:00.0 Off |                  N/A |                            │
| 20%   27C    P8    16W / 250W |      0MiB / 11172MiB |      0%      Default |                            │
+-------------------------------+----------------------+----------------------+                            │
|   1  GeForce GTX 108...  On   | 00000000:03:00.0 Off |                  N/A |                            │
| 20%   30C    P8    18W / 250W |      0MiB / 11172MiB |      0%      Default |                            │
+-------------------------------+----------------------+----------------------+                            │
|   2  GeForce GTX 108...  On   | 00000000:82:00.0 Off |                  N/A |                            │
| 20%   25C    P8    17W / 250W |      0MiB / 11172MiB |      0%      Default |                            │
+-------------------------------+----------------------+----------------------+                            │
|   3  GeForce GTX 108...  On   | 00000000:83:00.0 Off |                  N/A |                            │
| 20%   26C    P8    16W / 250W |      0MiB / 11172MiB |      0%      Default |                            │
+-------------------------------+----------------------+----------------------+                            │
                                                                                                            │
+-----------------------------------------------------------------------------+                            │
| Processes:                                                       GPU Memory |                            │
|  GPU       PID  Type  Process name                               Usage      |                            │
|=============================================================================|                            │
|  No running processes found                                                 |                            │
+-----------------------------------------------------------------------------+ 
```

When I start 3 python instances where each selects a different GPU's and first print the number of devices using torch.cuda.device_cout() I get:

```
#devices: 1
 
File "/home/xxxxx/code/xxxxx/xxxxx/xxxxx/xxxxx.py", line 38, in main                    
File "/home/xxxxx/env/lib/python3.6/site-packages/torch/cuda/__init__.py", line 223, in set_device    torch._C._cuda_setDevice(device)                                                                        
RuntimeError: cuda runtime error (10) : invalid device ordinal at torch/csrc/cuda/Module.cpp:88             
THCudaCheck FAIL file=torch/csrc/cuda/Module.cpp line=88 error=10 : invalid device ordinal

```
When I run the same code on a node with 4 GTX1080Ti GPU's pytorch correctly detects all graphics cards and the code runs properly.

Is this a setup problem? Or is this a issue related to pytorch?

I would like to add that when I select a node with 4x GTX980Ti pytorch also only detects 1 device.

cc @ngimel