Repository: pytorch
Issue: Segmentation fault when using CUDA with RNN
Labels: high priority,needs reproduction,module: crash,module: rnn,module: cuda,triaged

## 🐛 Bug

I get a segfault when trying to use cuda with the GRU module (with torch.nn.utils.rnn.pack_padded_sequence), and `torch.cuda.is_available()` returns true. I ran gdb for more info:

```
#0  0x00007ffff61b5762 in __cxa_throw () from /lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007fff324f2b1f in at::native::_cudnn_rnn(at::Tensor const&, c10::ArrayRef<at::Tensor>, long, at::Tensor const&, at::Tensor const&, at::Tensor const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, at::Tensor const&) () from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so
#2  0x00007fff9484a265 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<std::tuple<at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor> (at::Tensor const&, c10::ArrayRef<at::Tensor>, long, c10::optional<at::Tensor> const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, c10::optional<at::Tensor> const&), &c10::impl::detail::with_explicit_optional_tensors_<std::tuple<at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor> (at::Tensor const&, c10::ArrayRef<at::Tensor>, long, c10::optional<at::Tensor> const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, c10::optional<at::Tensor> const&), std::tuple<at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor> (at::Tensor const&, c10::ArrayRef<at::Tensor>, long, at::Tensor const&, at::Tensor const&, at::Tensor const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, at::Tensor const&), c10::CompileTimeFunctionPointer<std::tuple<at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor> (at::Tensor const&, c10::ArrayRef<at::Tensor>, long, at::Tensor const&, at::Tensor const&, at::Tensor const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, at::Tensor const&), &at::(anonymous namespace)::(anonymous namespace)::wrapper__cudnn_rnn> >::wrapper>, std::tuple<at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor>, c10::guts::typelist::typelist<at::Tensor const&, c10::ArrayRef<at::Tensor>, long, c10::optional<at::Tensor> const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, c10::optional<at::Tensor> const&> >, std::tuple<at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor> (at::Tensor const&, c10::ArrayRef<at::Tensor>, long, c10::optional<at::Tensor> const&, at::Tensor c--Type <RET> for more, q to quit, c to continue without paging--
onst&, c10::optional<at::Tensor> const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, c10::optional<at::Tensor> const&)>::call(c10::OperatorKernel*, at::Tensor const&, c10::ArrayRef<at::Tensor>, long, c10::optional<at::Tensor> const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, c10::optional<at::Tensor> const&) ()
   from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#3  0x00007fff82c92dbd in at::_cudnn_rnn(at::Tensor const&, c10::ArrayRef<at::Tensor>, long, c10::optional<at::Tensor> const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, c10::optional<at::Tensor> const&) () from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#4  0x00007fff844331b8 in torch::autograd::VariableType::(anonymous namespace)::_cudnn_rnn(at::Tensor const&, c10::ArrayRef<at::Tensor>, long, c10::optional<at::Tensor> const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, c10::optional<at::Tensor> const&) () from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#5  0x00007fff84433fa0 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<std::tuple<at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor> (at::Tensor const&, c10::ArrayRef<at::Tensor>, long, c10::optional<at::Tensor> const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, c10::optional<at::Tensor> const&), &torch::autograd::VariableType::(anonymous namespace)::_cudnn_rnn>, std::tuple<at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor>, c10::guts::typelist::typelist<at::Tensor const&, c10::ArrayRef<at::Tensor>, long, c10::optional<at::Tensor> const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, c10::optional<at::Tensor> const&> >, std::tuple<at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor> (at::Tensor const&, c10::ArrayRef<at::Tensor>, long, c10::optional<at::Tensor> const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long, long, long, bool--Type <RET> for more, q to quit, c to continue without paging--
, double, bool, bool, c10::ArrayRef<long>, c10::optional<at::Tensor> const&)>::call(c10::OperatorKernel*, at::Tensor const&, c10::ArrayRef<at::Tensor>, long, c10::optional<at::Tensor> const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, c10::optional<at::Tensor> const&) ()
   from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#6  0x00007fff82c92dbd in at::_cudnn_rnn(at::Tensor const&, c10::ArrayRef<at::Tensor>, long, c10::optional<at::Tensor> const&, at::Tensor const&, c10::optional<at::Tensor> const&, long, long, long, long, bool, double, bool, bool, c10::ArrayRef<long>, c10::optional<at::Tensor> const&) () from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#7  0x00007fff324ed43f in std::pair<at::Tensor, at::Tensor> at::native::(anonymous namespace)::_cudnn_impl<at::Tensor>(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, cudnnRNNMode_t, long, double, bool, bool) () from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so
#8  0x00007fff324edaf2 in at::native::(anonymous namespace)::gru_packed_cudnn(at::Tensor&, at::Tensor&, at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool) ()
   from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so
#9  0x00007fff82903cf9 in void at::native::DispatchStub<void (*)(at::Tensor&, at::Tensor&, at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool), at::native::gru_packed_cudnn_stub>::operator()<at::Tensor&, at::Tensor&, at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>&, bool&, long&, double&, bool&, bool&>(c10::DeviceType, at::Tensor&, at::Tensor&, at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>&, bool&, long&, double&, bool&, bool&) () from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
--Type <RET> for more, q to quit, c to continue without paging--
#10 0x00007fff828f97b7 in at::native::gru(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool) ()
   from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#11 0x00007fff83006ab8 in at::(anonymous namespace)::(anonymous namespace)::wrapper_gru_data(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool) ()
   from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#12 0x00007fff83006b32 in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<std::tuple<at::Tensor, at::Tensor> (at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool), &at::(anonymous namespace)::(anonymous namespace)::wrapper_gru_data>, std::tuple<at::Tensor, at::Tensor>, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool> >, std::tuple<at::Tensor, at::Tensor> (at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool)>::call(c10::OperatorKernel*, at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool) ()
   from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#13 0x00007fff82e8101e in std::tuple<at::Tensor, at::Tensor> c10::Dispatcher::call<std::tuple<at::Tensor, at::Tensor>, at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool>(c10::TypedOperatorHandle<std::tuple<at::Tensor, at::Tensor> (at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool)> const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool) const () from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
--Type <RET> for more, q to quit, c to continue without paging--
#14 0x00007fff82d80947 in at::gru(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<at::Tensor>, bool, long, double, bool, bool) ()
   from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#15 0x00007ffff4f3bd40 in torch::autograd::THPVariable_gru(_object*, _object*, _object*) () from /home/ryan/code/overlap_analysis/venv/lib/python3.8/site-packages/torch/lib/libtorch_python.so
#16 0x00000000005f5b29 in PyCFunction_Call ()
#17 0x00000000005f66f6 in _PyObject_MakeTpCall ()
#18 0x000000000057094b in _PyEval_EvalFrameDefault ()
#19 0x000000000056951a in _PyEval_EvalCodeWithName ()
#20 0x00000000005f60b3 in _PyFunction_Vectorcall ()
#21 0x000000000050a5cc in ?? ()
#22 0x00000000005f54e7 in PyObject_Call ()
#23 0x000000000056ca9e in _PyEval_EvalFrameDefault ()
#24 0x000000000056951a in _PyEval_EvalCodeWithName ()
```

## To Reproduce

Steps to reproduce the behavior:

1. Initialize all tensors and models with `device='cuda'`

The full code is here:
https://github.com/rlorigro/overlap_analysis/blob/main/scripts/predict_overlap_from_full_alignment.py

## Expected behavior

GPU go brrr

## Environment

PyTorch version: 1.8.1+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8 (64-bit runtime)
Python platform: Linux-5.8.0-53-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1080
Nvidia driver version: 465.27
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.1
[pip3] torch==1.8.1+cu111
[pip3] torchaudio==0.8.1
[pip3] torchvision==0.9.1+cu111
[conda] Could not collect


## Additional context

I installed GPU drivers and Cuda with the following procedure:
https://docs.google.com/document/d/1DDHYCdQzvLup44-k57ANOb6lpvdpOYA7Xj-9WAvRD-4/edit?usp=sharing

I should perhaps clarify that I actually installed Cuda 11.3 despite all the output in your environment collection script saying 11.1

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411 @Varal7 @ngimel