Repository: pytorch
Issue: CUDNN_STATUS_INTERNAL_ERROR when training with conv3d
Labels: module: cudnn,module: convolution,triaged

i am trying to do a training with inflated 3d network

replacing the model with different one will not casue error. 
setting up torch.backends.cudnn.enabled to False also fixes the problem, which is quite inefficiency.

here are some info about the enviroment cudnn version 7003, pytorch 0.3

i tried with different version of pytorch and cudnn, nothing helps
removing .nv does not help neither.
gpu and drivers are checked, and nothing seems wrong

model codes:
https://gist.github.com/yunkaili/297cfc8765269e4d677f7e4bfc7cc064

belows are some error info

```
Traceback (most recent call last):
  File "main.py", line 316, in <module>
    main()
  File "main.py", line 123, in main
    train(train_loader, model, criterion, optimizer, epoch)
  File "main.py", line 171, in train
    output = model(input_var)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 68, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/data_parallel.py", line 78, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/parallel_apply.py", line 67, in parallel_apply
    raise output
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/parallel/parallel_apply.py", line 42, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/liyunkai/i3d_base_model2/models.py", line 134, in forward
    base_out = self.base_model(input.view((-1, sample_len, cfg.TRAIN.FRAMES_IN_SNIPPET) + input.size()[-2:]))
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/liyunkai/i3d_base_model2/i3d_model.py", line 163, in forward
    x = self.layer4(x)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/container.py", line 67, in forward
    input = module(input)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/liyunkai/i3d_base_model2/i3d_model.py", line 86, in forward
    out = self.conv3(out)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py", line 325, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/conv.py", line 383, in forward
    self.padding, self.dilation, self.groups)
  File "/usr/local/lib/python3.5/dist-packages/torch/nn/functional.py", line 126, in conv3d
    return f(input, weight, bias)
RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

^CException ignored in: <module 'threading' from '/usr/lib/python3.5/threading.py'>
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 1288, in _shutdown
    t.join()
  File "/usr/lib/python3.5/threading.py", line 1054, in join
    self._wait_for_tstate_lock()
  File "/usr/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
    elif lock.acquire(block, timeout):
KeyboardInterrupt
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py", line 3208, in atexit_operations
    self.reset(new_session=False)
  File "/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py", line 1205, in reset
    self.displayhook.flush()
  File "/usr/local/lib/python3.5/dist-packages/IPython/core/displayhook.py", line 306, in flush
    gc.collect()
KeyboardInterrupt
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py", line 3208, in atexit_operations
    self.reset(new_session=False)
  File "/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py", line 1205, in reset
    self.displayhook.flush()
  File "/usr/local/lib/python3.5/dist-packages/IPython/core/displayhook.py", line 306, in flush
    gc.collect()
KeyboardInterrupt
THCudaCheck FAIL file=/pytorch/torch/lib/THC/generic/THCStorage.c line=184 error=77 : an illegal memory access was encountered
terminate called after throwing an instance of 'std::runtime_error'
  what():  cuda runtime error (77) : an illegal memory access was encountered at /pytorch/torch/lib/THC/generic/THCStorage.c:184
```