Repository: pytorch
Issue: Unable to use torch.det() inside nn.DataParallel with multiple gpus
Labels: module: cuda,triaged,module: data parallel,module: linear algebra

## 🐛 Bug

Using `torch.det()` inside `nn.DataParallel` in a multi gpu environment (tested 4) is suffering from a race condition. If the execution is done sequentially, the error doesn't trigger. 

## To Reproduce

```python
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader

# Parameters and DataLoaders
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")


class RandomDataset(Dataset):

    def __init__(self, length):
        self.len = length
        self.data = torch.randn(length, 3, 3)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len


class Model(nn.Module):
    # Our model

    def forward(self, data):
        x = data + torch.eye(3, device=data.device)
        for _ in range(20):
          det = torch.det(x)
          x = det[:,None,None] * x.transpose(-2,-1) / det[:,None,None]
        print("\tIn Model: input size", data.size(), "output size", x.size())
        return x



model = Model()
if torch.cuda.device_count() > 1:
  print("Let's use", torch.cuda.device_count(), "GPUs!")
  model = nn.DataParallel(model)

model.to(device)


while True:
  for data in DataLoader(
    dataset=RandomDataset(400000),
    batch_size=10000,
    shuffle=True,
  ):
      data = data.to(device)
      output = model(data)
      print("Outside: input size", data.size(), "output_size", output.size())

```
<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

Stack 

```
$ python mvce.py 
Let's use 4 GPUs!
        In Model: input size torch.Size([2500, 3, 3]) output size torch.Size([2500, 3, 3])
        In Model: input size torch.Size([2500, 3, 3]) output size torch.Size([2500, 3, 3])
        In Model: input size torch.Size([2500, 3, 3]) output size torch.Size([2500, 3, 3])
        In Model: input size torch.Size([2500, 3, 3]) output size torch.Size([2500, 3, 3])
Outside: input size torch.Size([10000, 3, 3]) output_size torch.Size([10000, 3, 3])
CUDA runtime error: an illegal memory access was encountered (77) in magma_queue_destroy_internal at /opt/conda/conda-bld/magma-cuda92_1583546984698/work/interface_cuda/interface.cpp:944
CUDA runtime error: an illegal memory access was encountered (77) in magma_queue_destroy_internal at /opt/conda/conda-bld/magma-cuda92_1583546984698/work/interface_cuda/interface.cpp:945
CUDA runtime error: an illegal memory access was encountered (77) in magma_queue_destroy_internal at /opt/conda/conda-bld/magma-cuda92_1583546984698/work/interface_cuda/interface.cpp:946
CUDA runtime error: an illegal memory access was encountered (77) in magma_queue_destroy_internal at /opt/conda/conda-bld/magma-cuda92_1583546984698/work/interface_cuda/interface.cpp:944
CUDA runtime error: an illegal memory access was encountered (77) in magma_queue_destroy_internal at /opt/conda/conda-bld/magma-cuda92_1583546984698/work/interface_cuda/interface.cpp:945
CUDA runtime error: an illegal memory access was encountered (77) in magma_queue_destroy_internal at /opt/conda/conda-bld/magma-cuda92_1583546984698/work/interface_cuda/interface.cpp:946
python: /opt/conda/conda-bld/magma-cuda92_1583546984698/work/interface_cuda/interface.cpp:901: void magma_queue_create_from_cuda_internal(magma_device_t, cudaStream_t, cublasHandle_t, cusparseHandle_t, magma_queue**, const char*, const char*, int): Assertion `queue->dCarray__ != __null' failed.
Aborted
```
## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->
To be able to use `torch.det()` inside `nn.DataParallel` like any other linalg operation.

## Environment

```
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 9.2

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.10.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 9.2.148
GPU models and configuration: 
GPU 0: Quadro RTX 5000
GPU 1: Quadro RTX 5000
GPU 2: Quadro RTX 5000
GPU 3: Quadro RTX 5000

Nvidia driver version: 440.82
cuDNN version: Probably one of the following:
/<redacted>/cuda/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.6.0.21
/<redacted>/cuda/cuda-9.0/targets/x86_64-linux/lib/libcudnn.so.7.0.3

Versions of relevant libraries:
[pip3] numpy==1.18.3
[pip3] torch-sparse==0.4.3
[pip3] torchfile==0.1.0
[pip3] torchvision==0.4.0+cu92
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               9.2                           0  
[conda] mkl                       2020.0                      166  
[conda] mkl-service               2.3.0            py36he904b0f_0  
[conda] mkl_fft                   1.0.15           py36ha843d7b_0  
[conda] mkl_random                1.1.0            py36hd6b4f25_0  
[conda] numpy                     1.18.1           py36h4f9e942_0  
[conda] numpy-base                1.18.1           py36hde5b4d6_1  
[conda] pytorch                   1.5.0           py3.6_cuda9.2.148_cudnn7.6.3_0    pytorch
[conda] torch-cluster             1.5.4                    pypi_0    pypi
[conda] torch-scatter             2.0.4                    pypi_0    pypi
[conda] torchsummary              1.5.1                    pypi_0    pypi
[conda] torchvision               0.6.0                 py36_cu92    pytorch
```
## Additional context

N.a.


cc @ngimel @vincentqb @vishwakftw @jianyuh