Repository: pytorch
Issue: CUDNN_STATUS_EXECUTION_FAILED
Labels: module: cudnn,module: cuda,triaged

When I train the code, the following problem suddenly appears
```
Traceback (most recent call last):
  File "main.py", line 211, in <module>
    train(args,model,train_loader,optimizer,epoch)
  File "/media/nnir712/F264A15264A119FD/zzh/detect/yolo/zzh_yolo/zzh_yolo_mobilenet/train.py", line 22, in train
    loss, outputs = model(imgs, targets)
  File "/home/nnir712/anaconda3/envs/zzhpy36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/nnir712/F264A15264A119FD/zzh/detect/yolo/zzh_yolo/zzh_yolo_mobilenet/models.py", line 297, in forward
    x = module(x)
  File "/home/nnir712/anaconda3/envs/zzhpy36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/nnir712/anaconda3/envs/zzhpy36/lib/python3.6/site-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/home/nnir712/anaconda3/envs/zzhpy36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/nnir712/anaconda3/envs/zzhpy36/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 345, in forward
    return self.conv2d_forward(input, self.weight)
  File "/home/nnir712/anaconda3/envs/zzhpy36/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
```
System information
```
ubuntu18.04

gmp                       6.1.2                h6c8ec71_1    defaults
google-auth               1.10.0                   pypi_0    pypi
google-auth-oauthlib      0.4.1                    pypi_0    pypi
grpcio                    1.26.0                   pypi_0    pypi
gst-plugins-base          1.14.0               hbbd80ab_1    defaults
gstreamer                 1.14.0               hb453b48_1    defaults
icu                       58.2                 h9c2bf20_1    defaults
idna                      2.8                      pypi_0    pypi
imagecorruptions          1.0.0                    pypi_0    pypi
imageio                   2.6.1                    pypi_0    pypi
imgaug                    0.2.6                    pypi_0    pypi
importlib_metadata        1.4.0                    py36_0    defaults
intel-openmp              2019.4                      243    defaults
ipykernel                 5.1.4            py36h39e3cac_0    defaults
ipython                   7.11.1           py36h39e3cac_0    defaults
ipython_genutils          0.2.0                    py36_0    defaults
ipywidgets                7.5.1                      py_0    defaults
jedi                      0.16.0                   py36_0    defaults
jinja2                    2.10.3                     py_0    defaults
jpeg                      9b                   h024ee3a_2    defaults
jsonpatch                 1.24                     pypi_0    pypi
jsonpointer               2.0                      pypi_0    pypi
jsonschema                3.2.0                    py36_0    defaults
jupyter                   1.0.0                    py36_7    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
jupyter_client            5.3.4                    py36_0    defaults
jupyter_console           6.1.0                      py_0    defaults
jupyter_core              4.6.1                    py36_0    defaults
leveldb                   1.20                 hf484d3e_1    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
libedit                   3.1.20181209         hc058e9b_0    defaults
libffi                    3.2.1                hd88cf55_4    defaults
libgcc-ng                 9.1.0                hdf63c60_0    defaults
libgfortran-ng            7.3.0                hdf63c60_0    defaults
libpng                    1.6.37               hbc83047_0    defaults
libprotobuf               3.8.0                hd408876_0    defaults
libsodium    
oauthlib                  3.1.0                    pypi_0    pypi
olefile                   0.46                     py36_0    defaults
opencv-contrib-python     3.4.1.15                 pypi_0    pypi
opencv-python             4.1.2.30                 pypi_0    pypi
openssl                   1.1.1d               h7b6447c_3    defaults
pandoc                    2.2.3.2                       0    defaults
pandocfilters             1.4.2                    py36_1    defaults
parso                     0.6.0                      py_0    defaults
pbr                       5.1.3                    pypi_0    pypi
pcre                      8.43                 he6710b0_0    defaults
pexpect                   4.8.0                    py36_0    defaults
pickleshare               0.7.5                    py36_0    defaults
pillow                    5.0.0                    pypi_0    pypi
pip                       20.0.2                   py36_1    defaults
prometheus_client         0.7.1                      py_0    defaults
prompt_toolkit            3.0.3                      py_0    defaults
protobuf                  3.8.0            py36he6710b0_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
ptyprocess                0.6.0                    py36_0    defaults
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.7                    pypi_0    pypi
pycocotools               2.0.0                    pypi_0    pypi
pygments                  2.5.2                      py_0    defaults
pyqt                      5.9.2            py36h05f1152_2    defaults
pyrsistent                0.15.7           py36h7b6447c_0    defaults
python                    3.6.8                h0371630_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
python-dateutil           2.8.1                      py_0    defaults
pytorch                   1.4.0           py3.6_cuda10.1.243_cudnn7.6.3_0    pytorch
pyzmq                     18.1.0           py36he6710b0_0    defaults
qt                        5.9.7                h5867ecd_1    defaults
qtconsole                 4.6.0                      py_1    defaults
readline                  7.0                  h7b6447c_5    defaults
requests                  2.22.0                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rsa                       4.0                      pypi_0    pypi
scikit-image              0.16.2                   pypi_0    pypi
scipy                     1.4.1                    pypi_0    pypi
send2trash                1.5.0                    py36_0    defaults
setuptools                41.2.0                   pypi_0    pypi
sip                       4.19.8           py36hf484d3e_0    defaults
six                       1.14.0                   py36_0    defaults
snappy                    1.1.7                hbae5bb6_3    defaults
sqlite                    3.30.1               h7b6447c_0    defaults
tb-nightly                2.0.0a20190915           pypi_0    pypi
tensorboard               2.1.0                    pypi_0    pypi
tensorflow                1.13.1                   pypi_0    pypi
tensorflow-estimator      1.13.0                   pypi_0    pypi
terminado                 0.8.3                    py36_0    defaults
terminaltables            3.1.0                    pypi_0    pypi
testpath                  0.4.4                      py_0    defaults
thop                      0.0.31-1912272122          pypi_0    pypi
tk                        8.6.8                hbc83047_0    defaults
torchfile                 0.1.0                    pypi_0    pypi
torchvision               0.5.0                py36_cu101    pytorch
tornado                   6.0.3            py36h7b6447c_0    defaults
tqdm                      4.35.0                   pypi_0    pypi
traitlets                 4.3.3                    py36_0    defaults
urllib3                   1.25.7                   pypi_0    pypi
visdom                    0.1.8.9                  pypi_0    pypi
wcwidth                   0.1.7                    py36_0    defaults
webencodings              0.5.1                    py36_1    defaults
websocket-client          0.57.0                   pypi_0    pypi
wheel                     0.34.1                   py36_0    defaults
widgetsnbextension        3.5.1                    py36_0    defaults
xz                        5.2.4                h14c3975_4    defaults
zeromq                    4.3.1                he6710b0_3    defaults
zipp                      0.6.0                      py_0    defaults
zlib                      1.2.11               h7b6447c_3    defaults
zstd                      1.3.7                h0b5b093_0    defaults
```
AND :
```
` export | grep cuda`
declare -x CUDA_HOME="/usr/local/cuda-10.1"
declare -x LD_LIBRARY_PATH="”/usr/local/cuda-10.1/lib64:/home/nnir712/anaconda3/pkgs/libprotobuf-3.5.2-h6f1eeef_0/lib:/home/nnir712/anaconda3/pkgs/libprotobuf-3.8.0-hd408876_0/lib:BRARY_PATH:/usr/local/cuda-10.1/lib64:/usr/local/cuda-10.1/extras/CUPTI/lib64”:/home/nnir712/anaconda3/lib"
declare -x PATH="/home/nnir712/.local/bin:/home/nnir712/bin:/usr/local/cuda-10.1/bin:/usr/local/cuda-10.1/bin:/home/nnir712/anaconda3/pkgs/libprotobuf-3.5.2-h6f1eeef_0/bin:/home/nnir712/anaconda3/pkgs/libprotobuf-3.8.0-hd408876_0/bin:/home/nnir712/.local/bin:/home/nnir712/bin:/usr/local/cuda-10.1/bin:/usr/local/cuda-10.1/bin:/home/nnir712/anaconda3/pkgs/libprotobuf-3.5.2-h6f1eeef_0/bin:/home/nnir712/anaconda3/pkgs/libprotobuf-3.8.0-hd408876_0/bin:/home/nnir712/anaconda3/envs/zzhpy36/bin:/home/nnir712/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/home/nnir712/anaconda3/lib:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/home/nnir712/anaconda3/lib"

```


cc @ngimel @csarofeen @ptrblck