Repository: pytorch
Issue: Segmentation fault need help
Labels: needs reproduction,module: cuda,triaged

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

the codes works pretty well in CPU mode (tested in my macbook pro), and get Segmentation fault (core dumped) in GPU mode, which is my pc at home.

## To Reproduce

```
==============Model Training===========
{'batch_size': 64, 'buffer_size': 100000, 'learning_rate': 0.0001, 'learning_starts': 100, 'ent_coef': 'auto_0.1'}
[Thread 0x7ffe371be700 (LWP 5137) exited]
[Thread 0x7ffdea759700 (LWP 5140) exited]
[Thread 0x7ffdb3fff700 (LWP 5143) exited]
[Thread 0x7ffdcbe99700 (LWP 5142) exited]
[Thread 0x7ffddad99700 (LWP 5141) exited]
[Thread 0x7ffe01c99700 (LWP 5139) exited]
[Thread 0x7ffe097d9700 (LWP 5138) exited]
[Thread 0x7ffe379bf700 (LWP 5136) exited]
[Thread 0x7ffe381c0700 (LWP 5135) exited]
[Thread 0x7ffe389c1700 (LWP 5134) exited]
[Thread 0x7ffe39adc700 (LWP 5133) exited]
[Thread 0x7ffe3a2dd700 (LWP 5132) exited]
[Thread 0x7fff4c9ee700 (LWP 5131) exited]
[Thread 0x7fff5d9f0700 (LWP 5130) exited]
[Thread 0x7fff551ef700 (LWP 5129) exited]
[Detaching after fork from child process 5252]
[Detaching after fork from child process 5266]
[New Thread 0x7ffdb3fff700 (LWP 5269)]
Using cuda device
[New Thread 0x7ffdcbe99700 (LWP 5270)]
[New Thread 0x7ffddad99700 (LWP 5271)]
[New Thread 0x7ffdea759700 (LWP 5272)]
[New Thread 0x7fff5d9f0700 (LWP 5273)]
[New Thread 0x7fff551ef700 (LWP 5274)]
[New Thread 0x7fff4c9ee700 (LWP 5275)]
[New Thread 0x7ffe3a2dd700 (LWP 5276)]
[New Thread 0x7ffe39adc700 (LWP 5277)]
[New Thread 0x7ffe389c1700 (LWP 5278)]
[New Thread 0x7ffe381c0700 (LWP 5279)]
Logging to /home/don/findy-home/out/finrl/tensorboard_log/sac/sac_9
[New Thread 0x7ffe379bf700 (LWP 5288)]
[New Thread 0x7ffe371be700 (LWP 5289)]
--Type <RET> for more, q to quit, c to continue without paging--

Thread 47 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffe379bf700 (LWP 5288)]
0x00007ffec8d3748f in at::cuda::getCurrentCUDABlasHandle() ()
   from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
(gdb) where
#0  0x00007ffec8d3748f in at::cuda::getCurrentCUDABlasHandle() ()
   from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#1  0x00007ffec8d257dd in void at::cuda::blas::gemm<float>(char, char, long, long, long, float, float const*, long, float const*, long, float, float*, long) ()
   from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#2  0x00007ffec9f418f0 in at::native::(anonymous namespace)::addmm_out_cuda_impl(at::Tensor&, at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::Scalar, c10::Scalar) ()
   from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#3  0x00007ffec9f43092 in at::native::mm_cuda(at::Tensor const&, at::Tensor const&) ()
   from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#4  0x00007ffec8d7c848 in at::(anonymous namespace)::(anonymous namespace)::wrapper_mm(at::Tensor const&, at::Tensor const&) ()
   from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#5  0x00007ffec8d7c88f in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, at::Tensor const&), &at::(anonymous namespace)::(anonymous namespac--T--T--Type--T--Type--T--T--T--T--T--T--T--T--T--T--T--T--T--T--T--T--T--T---------Type--Type--Type <RET> for mor--Type <RET> f--Type--Type <RET> for more, q to quit, c to continue without paging--
e)::wrapper_mm>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&> >, at::Tensor (at::Tensor const&, at::Tensor const&)>::call(c10::OperatorKernel*, at::Tensor const&, at::Tensor const&) () from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#6  0x00007ffeb71f8596 in at::Tensor c10::Dispatcher::call<at::Tensor, at::Tensor const&, at::Tensor const&>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, at::Tensor const&)> const&, at::Tensor const&, at::Tensor const&) const [clone .constprop.4808] () from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#7  0x00007ffeb71fdbbf in at::mm(at::Tensor const&, at::Tensor const&) () from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#8  0x00007ffeb8ae191c in torch::autograd::VariableType::(anonymous namespace)::mm(at::Tensor const&, at::Tensor const&) ()
   from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#9  0x00007ffeb8ae1e8f in c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, at::Tensor const&), &torch::autograd::VariableType::(anonymous namespace)::mm>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, at::Tensor const&> >, at::Tensor (at::Tensor const&, at::Tensor const&)>::call(c10::OperatorKernel*, at::Tensor const&, at::Tensor const&) () from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#10 0x00007ffeb764a7d6 in at::Tensor c10::Dispatcher::call<at::Tensor, at::Tensor const&, at::Tensor const&>(c10::TypedOperatorHandle<at::Tensor (at::Tensor const&, at::Tensor const&)> const&, at::Tensor const&, at::Tensor const&) const [clone .constprop.1487] () from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#11 0x00007ffeb764e9ff in at::Tensor::mm(at::Tensor const&) const () from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#12 0x00007ffeb9520911 in torch::autograd::generated::details::mm_mat1_backward(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::Scalar const&) ()
   from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#13 0x00007ffeb88f6093 in torch::autograd::generated::AddmmBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) ()
   from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#14 0x00007ffeb8fb3771 in torch::autograd::Node::operator()(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) ()
   from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#15 0x00007ffeb8faf57b in torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) () from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#16 0x00007ffeb8fb019f in torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) ()
   from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#17 0x00007ffeb8fa7979 in torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) ()
   from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#18 0x00007fff2972b293 in torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) ()
   from /home/don/miniconda3/lib/python3.8/site-packages/torch/lib/libtorch_python.so
#19 0x00007fffc6d3319d in std::execute_native_thread_routine (__p=0x5555fd708e30)
    at /home/nwani/m3/conda-bld/compilers_linux-64_1560109574129/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libstdc++-v3/src/c++11/thread.cc:80
#20 0x00007ffff7f97609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#21 0x00007ffff7ebe293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
```
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:05:00.0  On |                  N/A |
| 29%   59C    P2    69W / 250W |   1329MiB / 11175MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:09:00.0 Off |                  N/A |
|  0%   49C    P8    12W / 250W |     13MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1002      G   /usr/lib/xorg/Xorg                 59MiB |
|    0   N/A  N/A      1596      G   /usr/lib/xorg/Xorg                213MiB |
|    0   N/A  N/A      1722      G   /usr/bin/gnome-shell               60MiB |
|    0   N/A  N/A      3932      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      4073      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      4118      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      4225      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      4257      G   /usr/lib/firefox/firefox            2MiB |
|    0   N/A  N/A      4814      C   ...don/miniconda3/bin/python      967MiB |
|    1   N/A  N/A      1002      G   /usr/lib/xorg/Xorg                  4MiB |
|    1   N/A  N/A      1596      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+
```
```
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Feb_14_21:12:58_PST_2021
Cuda compilation tools, release 11.2, V11.2.152
Build cuda_11.2.r11.2/compiler.29618528_0
```

 - PyTorch Version (e.g., 1.0): 1.8.1+cu111
 - OS (e.g., Linux): ubuntu
 - How you installed PyTorch (`conda`, `pip`, source): pip
 - Build command you used (if compiling from source):
 - Python version: 3.8.5
 - CUDA/cuDNN version: 11.2.152
 - GPU models and configuration: nvidia 1080Ti
 - Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


cc @ngimel