Repository: pytorch
Issue: Vertices=torch.matmul(vertices.unsqueeze(0), rotations_init), RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched in CentOS
Labels: needs reproduction,module: cuda,triaged,module: numpy

How should I fix this?
```
(phosa) [jalal@goku phosa]$ python demo.py --filename input/dark_bat.jpg --class_name bat
2021-03-26 16:55:48,497 INFO     Calling with args: Namespace(class_name='bat', filename='input/dark_bat.jpg', lw_collision=None, lw_depth=None, lw_inter=None, lw_inter_part=None, lw_scale=None, lw_scale_person=None, lw_sil=None, mesh_index=0, output_dir='output')
2021-03-26 16:55:51,993 INFO     Loading checkpoint from detectron2://PointRend/InstanceSegmentation/pointrend_rcnn_R_50_FPN_3x_coco/164955410/model_final_3c3198.pkl
2021-03-26 16:55:51,995 INFO     URL https://dl.fbaipublicfiles.com/detectron2/PointRend/InstanceSegmentation/pointrend_rcnn_R_50_FPN_3x_coco/164955410/model_final_3c3198.pkl cached in /home/grad3/jalal/.torch/fvcore_cache/detectron2/PointRend/InstanceSegmentation/pointrend_rcnn_R_50_FPN_3x_coco/164955410/model_final_3c3198.pkl
2021-03-26 16:55:52,104 INFO     Reading a file from 'Detectron2 Model Zoo'
WARNING: You are using a SMPL model, with only 10 shape coefficients.
Downloading: "https://download.pytorch.org/models/resnet50-19c8e357.pth" to /home/grad3/jalal/.cache/torch/hub/checkpoints/resnet50-19c8e357.pth
100%|██████████████████████████████████████████████████████████████████████████████████████████| 97.8M/97.8M [00:00<00:00, 108MB/s]
class_name:  bat
  0%|                                                                                                  | 0/12500.0 [00:00<?, ?it/s]Traceback (most recent call last):
  File "demo.py", line 145, in <module>
    main(get_args())
  File "demo.py", line 121, in main
    instances=instances, class_name=args.class_name, mesh_index=args.mesh_index
  File "/scratch3/research/code/phosa/phosa/pose_optimization.py", line 406, in find_optimal_poses
    num_initializations=num_initializations,
  File "/scratch3/research/code/phosa/phosa/pose_optimization.py", line 287, in find_optimal_pose
    vertices=torch.matmul(vertices.unsqueeze(0), rotations_init),
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)`
  0%|                                                                                                  | 0/12500.0 [00:00<?, ?it/s]
```


```
(phosa) [jalal@goku ~]$ python
Python 3.6.8 (default, Nov 16 2020, 16:55:22) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.8.1+cu111'
>>> torch.cuda.is_available()
True
```
and


```
$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.9.2009 (Core)
Release:	7.9.2009
Codename:	Core
```

and 

![Screenshot from 2021-03-26 17-08-40|690x446](upload://3TMjHw5nvF6s1zFiQuWBOjkXeXF.png) 


```
(phosa) [jalal@goku ~]$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
```
and

```
(phosa) [jalal@goku ~]$ python collect_env.py 
Collecting environment information...
PyTorch version: 1.8.1+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: CentOS Linux release 7.9.2009 (Core) (x86_64)
GCC version: (GCC) 7.3.0
Clang version: 3.4.2 (tags/RELEASE_34/dot2-final)
CMake version: version 3.10.0-rc5

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.0.130
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti

Nvidia driver version: 450.51.06
cuDNN version: Probably one of the following:
/scratch/system/usr/local/cuda-10.0/targets/x86_64-linux/lib/libcudnn.so.7.4.2
/scratch/system/usr/local/cuda-9.0/targets/x86_64-linux/lib/libcudnn.so.7.0.5
/scratch/system/usr/local/cuda-9.2/targets/x86_64-linux/lib/libcudnn.so.7.6.5
/scratch2/system/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.7.6.5
/scratch2/system/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn.so.8.0.2
/scratch2/system/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.0.2
/scratch2/system/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.0.2
/scratch2/system/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.0.2
/scratch2/system/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.0.2
/scratch2/system/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.0.2
/scratch2/system/usr/local/cuda-10.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.0.2
/scratch2/system/usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5.1.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] neural-renderer-pytorch==1.1.3
[pip3] numpy==1.19.5
[pip3] torch==1.8.1+cu111
[pip3] torchaudio==0.8.1
[pip3] torchgeometry==0.1.2
[pip3] torchvision==0.9.1+cu111
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               10.0.130                      0  
[conda] mkl                       2018.0.3                      1  
[conda] mkl-service               1.1.2            py36h17a0993_4  
[conda] mkl_fft                   1.0.10                   py36_0    conda-forge
[conda] mkl_random                1.0.2                    py36_0    conda-forge
[conda] msgpack-numpy             0.4.4.3                  pypi_0    pypi
[conda] numpy                     1.19.4                   pypi_0    pypi
[conda] numpydoc                  0.9.1                      py_0    conda-forge
[conda] pytorch                   1.1.0           py3.6_cuda10.0.130_cudnn7.5.1_0    pytorch
[conda] pytorch-nightly           1.0.0.dev20190328 py3.6_cuda10.0.130_cudnn7.4.2_0    pytorch
[conda] pytorchviz                0.0.1                    pypi_0    pypi
[conda] torchvision               0.3.0           py36_cu10.0.130_1    pytorch
```


Please let me know if more details is required.

cc @ngimel @mruberry @rgommers @heitorschueroff