Repository: pytorch
Issue: RuntimeError: cublas runtime error 
Labels: triaged,module: cublas

My env is :
Collecting environment information...
PyTorch version: 0.4.1
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.5 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609
CMake version: version 3.14.0

Python version: 3.5
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce RTX 2080
Nvidia driver version: 410.48
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.5.0

Versions of relevant libraries:
[pip3] numpy==1.16.4
[pip3] torch==0.4.1
[pip3] torch-sparse==0.4.0
[pip3] torchvision==0.2.1
[conda] blas                      1.0                         mkl  
[conda] cuda100                   1.0                           0    pytorch
[conda] cuda90                    1.0                  h6433d27_0    pytorch
[conda] mkl                       2018.0.3                      1  
[conda] mkl-service               1.1.2            py36h90e4bf4_5  
[conda] mkl_fft                   1.0.6            py36h7dd41cf_0  
[conda] mkl_random                1.0.1            py36h4414c95_1  
[conda] pytorch                   1.1.0           py3.6_cuda10.0.130_cudnn7.5.1_0    pytorch
[conda] torch                     1.2.0a0+1252899          pypi_0    pypi
[conda] torchvision               0.3.0           py36_cu10.0.130_1    pytorch


My error is:
/home/lab/.local/lib/python3.5/site-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='none' instead.
  warnings.warn(warning.format(ret))
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
/home/lab/.local/lib/python3.5/site-packages/torch/nn/modules/upsampling.py:225: UserWarning: nn.UpsamplingBilinear2d is deprecated. Use nn.functional.interpolate instead.
  warnings.warn("nn.UpsamplingBilinear2d is deprecated. Use nn.functional.interpolate instead.")
/home/lab/.local/lib/python3.5/site-packages/torch/nn/modules/upsampling.py:122: UserWarning: nn.Upsampling is deprecated. Use nn.functional.interpolate instead.
  warnings.warn("nn.Upsampling is deprecated. Use nn.functional.interpolate instead.")
Traceback (most recent call last):
  File "tools/demo.py", line 189, in <module>
    demo()
  File "tools/demo.py", line 175, in demo
    corner_pred = eval_net(seg_pred, vertex_pred).cpu().detach().numpy()[0]
  File "/home/lab606/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/lab/.local/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 121, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/lab/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "tools/demo.py", line 55, in forward
    return ransac_voting_layer_v3(mask, vertex_pred, 512, inlier_thresh=0.99)
  File "/home/lab/pvnet/lib/ransac_voting_gpu_layer/ransac_voting_gpu.py", line 592, in ransac_voting_layer_v3
    ATA=torch.matmul(normal.permute(0,2,1),normal)              # [vn,2,2]
RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:411