Repository: pytorch
Issue:  floating point exception (core dumped) in training 10^4 steps
Labels: module: cuda,triaged

## 🐛 Bug
 floating point exception (core dumped)
## To Reproduce
https://github.com/lukemelas/EfficientNet-PyTorch
 python train.py -b 5 -a "efficientnet-b5" --image_size 224 --gpu 0 --epochs 100


With no change run on the two version of the Pytorch 

On Pytorch 1.5

    20121 floating point exception (core dumped)


On pytorch 1.6
```bash

terminate called after throwing an instance of 'c10::Error'
  what():  invalid device pointer: 0x7ff443ffdc00
Exception raised from free at /opt/conda/conda-bld/pytorch_1595629427478/work/c10/cuda/CUDACachingAllocator.cpp:844 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7ff789d4377d in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x529 (0x7ff789f93769 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7ff789d2fb1d in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0xc8bf69 (0x7ff78ae34f69 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0xc9d394 (0x7ff78ae46394 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0xc9f0b5 (0x7ff78ae480b5 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xc9a30e (0x7ff78ae4330e in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xc9befb (0x7ff78ae44efb in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #8: at::native::cudnn_convolution_backward_input(c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool) + 0xb2 (0x7ff78ae45452 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #9: <unknown function> + 0xd0165b (0x7ff78aeaa65b in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #10: <unknown function> + 0xd31478 (0x7ff78aeda478 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #11: at::cudnn_convolution_backward_input(c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool) + 0x1ad (0x7ff7c13f0ced in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #12: at::native::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x223 (0x7ff78ae43b23 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #13: <unknown function> + 0xd01745 (0x7ff78aeaa745 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #14: <unknown function> + 0xd314d4 (0x7ff78aeda4d4 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)
frame #15: at::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x1e2 (0x7ff7c13ff6a2 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #16: <unknown function> + 0x2c250c2 (0x7ff7c30c30c2 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #17: <unknown function> + 0x2c39684 (0x7ff7c30d7684 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #18: at::cudnn_convolution_backward(at::Tensor const&, at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, c10::ArrayRef<long>, c10::ArrayRef<long>, long, bool, bool, std::array<bool, 2ul>) + 0x1e2 (0x7ff7c13ff6a2 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #19: torch::autograd::generated::CudnnConvolutionBackward::apply(std::vector<at::Tensor, std::allocator<at::Tensor> >&&) + 0x258 (0x7ff7c2f4a098 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #20: <unknown function> + 0x30d1017 (0x7ff7c356f017 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #21: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0x1400 (0x7ff7c356a860 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #22: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x451 (0x7ff7c356b401 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #23: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x89 (0x7ff7c3563579 in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #24: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x4a (0x7ff7c789299a in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #25: <unknown function> + 0xc819d (0x7ff7ca3cc19d in /home/winshare/anaconda3/envs/nightly/lib/python3.7/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #26: <unknown function> + 0x9609 (0x7ff7e43bc609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #27: clone + 0x43 (0x7ff7e42e3103 in /lib/x86_64-linux-gnu/libc.so.6)


```
or
```bash
20121 floating point exception (core dumped)


```
<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

## Expected behavior


code  sample:


i run different code include 
{
Mask RCNN
Faster RCNN
EfficientNet
}
# all code will cause the same error {float point exception}!!!!!!!!!!!!!!!!!!


<!-- A clear and concise description of what you expected to happen. -->

## Environment

Please copy and paste the output from our
[environment collection script](https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py)
(or fill out the checklist below manually).

You can get the script and run it with:
```
PyTorch version: 1.6.0
Is debug build: False
CUDA used to build PyTorch: 10.2

OS: Ubuntu 20.04.1 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-10ubuntu2) 9.3.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.7 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti
Nvidia driver version: 440.100
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] efficientnet-pytorch==0.7.0
[pip3] numpy==1.19.1
[pip3] torch==1.6.0
[pip3] torchvision==0.2.1
[conda] blas                      1.0                         mkl    https://mirrors.ustc.edu.cn/anaconda/pkgs/free
[conda] cudatoolkit               10.2.89              hfd86e86_1    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
[conda] efficientnet-pytorch      0.7.0                     <pip>
[conda] mkl                       2019.4                      243    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
[conda] mkl-service               2.3.0            py37h516909a_0    https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge
[conda] mkl_fft                   1.1.0            py37hc1659b7_1    https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge
[conda] mkl_random                1.1.0            py37hb3f55d8_0    https://mirrors.ustc.edu.cn/anaconda/cloud/conda-forge
[conda] numpy                     1.19.1           py37hbc911f0_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
[conda] numpy-base                1.19.1           py37hfa32c7d_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
[conda] pytorch                   1.6.0           py3.7_cuda10.2.89_cudnn7.6.5_0    pytorch
[conda] torchvision               0.2.1                    py37_0    https://mirrors.ustc.edu.cn/anaconda/pkgs/main
```

 - PyTorch Version (1.3/1.5/1.6):
 - OS (Ubuntu 20):
 - How you installed PyTorch (`conda`):
 - Build command you used (if compiling from source):
 - Python version:
 - CUDA/cuDNN version:10.2
 - GPU models and configurationRTX2080ti
 - Any other relevant information:

<!-- Add any other context about the problem here. -->


cc @ngimel