Repository: pytorch
Issue: RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:27 for spectral norm
Labels: triaged,module: cublas

Hi guys,

I have a GPU GeForce GTX TITAN X with python 3.7.4 and CUDA 10.1. I am training my GAN on this GPU and it seems like the following errors occasionally happens during the training process and it always related to the spectral norm which is provided  
by `torch.nn.utils.spectral_norm` [here](https://pytorch.org/docs/stable/generated/torch.nn.utils.spectral_norm.html#torch.nn.utils.spectral_norm)

In this example it happens at iteration 180 and another fail (not list hear) happens at iteration 5800. Is that a bug?

```
Traceback (most recent call last):
  File "/mnt/HDD2/hwan7885/anaconda3/envs/Nepean/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/mnt/HDD2/hwan7885/anaconda3/envs/Nepean/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/mnt/HDD2/hwan7885/proj_hr/gan_ST_map_cnn/sagan/train_cifar10.py", line 15, in <module>
    trainer.train()
  File "/mnt/HDD2/hwan7885/proj_hr/gan_ST_map_cnn/sagan/trainer.py", line 261, in train
    fake_images = self.G(z, real_labels)
  File "/mnt/HDD2/hwan7885/anaconda3/envs/Nepean/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/HDD2/hwan7885/proj_hr/gan_ST_map_cnn/sagan/sagan_models.py", line 188, in forward
    act4 = self.block4(act3, labels)    # n x g_conv_dim*2 x 64 x 64
  File "/mnt/HDD2/hwan7885/anaconda3/envs/Nepean/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/mnt/HDD2/hwan7885/proj_hr/gan_ST_map_cnn/sagan/sagan_models.py", line 151, in forward
    x0 = self.snconv2d0(x0)
  File "/mnt/HDD2/hwan7885/anaconda3/envs/Nepean/lib/python3.7/site-packages/torch/nn/modules/module.py", line 714, in _call_impl
    result = hook(self, input)
  File "/mnt/HDD2/hwan7885/anaconda3/envs/Nepean/lib/python3.7/site-packages/torch/nn/utils/spectral_norm.py", line 99, in __call__
    setattr(module, self.name, self.compute_weight(module, do_power_iteration=module.training))
  File "/mnt/HDD2/hwan7885/anaconda3/envs/Nepean/lib/python3.7/site-packages/torch/nn/utils/spectral_norm.py", line 85, in compute_weight
    sigma = torch.dot(u, torch.mv(weight_mat, v))
RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:27
```

This is the output of collect_env script
```
Collecting environment information...
PyTorch version: 1.6.0+cu101
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Ubuntu 16.04.6 LTS
GCC version: (Ubuntu 4.9.3-13ubuntu2) 4.9.3
CMake version: version 3.5.1

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 7.5.17
GPU models and configuration: 
GPU 0: GeForce GTX TITAN X
GPU 1: GeForce GTX TITAN X

Nvidia driver version: 418.87.00
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.4
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7

Versions of relevant libraries:
[pip3] numpy==1.17.4
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.1.243             h6bb024c_0  
[conda] libblas                   3.8.0                    14_mkl    conda-forge
[conda] libcblas                  3.8.0                    14_mkl    conda-forge
[conda] liblapack                 3.8.0                    14_mkl    conda-forge
[conda] liblapacke                3.8.0                    14_mkl    conda-forge
[conda] mkl                       2019.4                      243  
[conda] mkl-service               2.3.0            py37he904b0f_0  
[conda] mkl_fft                   1.0.15           py37ha843d7b_0  
[conda] mkl_random                1.1.0            py37hd6b4f25_0  
[conda] numpy                     1.17.4           py37hc1035e2_0  
[conda] numpy-base                1.17.4           py37hde5b4d6_0  
[conda] torch                     1.6.0+cu101              pypi_0    pypi
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchtext                 0.6.0                    pypi_0    pypi
[conda] torchvision               0.7.0+cu101              pypi_0    pypi
```

cc @csarofeen @ptrblck