Repository: pytorch
Issue: "malloc(): memory corruption (fast)", action=3) at malloc.c
Labels: needs reproduction,module: crash,triaged

Training crashes after a number of iterations randomly. I don't know where it happens in my python code. When I run 'py-bt', it says 'unable to locate python frame'. 

## Environment
PyTorch version: 1.3.1
Is debug build: No
CUDA used to build PyTorch: 10.1.243

OS: Ubuntu 16.04.4 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: GeForce RTX 2080 Ti
GPU 1: GeForce RTX 2080 Ti
GPU 2: GeForce RTX 2080 Ti
GPU 3: GeForce RTX 2080 Ti
GPU 4: GeForce RTX 2080 Ti
GPU 5: GeForce RTX 2080 Ti
GPU 6: GeForce RTX 2080 Ti
GPU 7: GeForce RTX 2080 Ti
GPU 8: GeForce RTX 2080 Ti
GPU 9: GeForce RTX 2080 Ti

Nvidia driver version: 418.74
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.1.4

Versions of relevant libraries:
[conda] blas                      1.0                         mkl  
[conda] mkl                       2019.4                      243  
[conda] mkl-service               2.3.0            py37he904b0f_0  
[conda] mkl_fft                   1.0.15           py37ha843d7b_0  
[conda] mkl_random                1.1.0            py37hd6b4f25_0  
[conda] pytorch                   1.3.1           py3.7_cuda10.1.243_cudnn7.6.3_0    pytorch
[conda] torchvision               0.4.2                py37_cu101    pytorch

## Error Message

> Program terminated with signal SIGABRT, Aborted.
#0  0x00007fe80848f428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7fe740ff9700 (LWP 407289))]
(gdb) bt
#0  0x00007fe80848f428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007fe80849102a in __GI_abort () at abort.c:89
#2  0x00007fe8084d17ea in __libc_message (do_abort=do_abort@entry=2, 
    fmt=fmt@entry=0x7fe8085eaed8 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007fe8084dc651 in malloc_printerr (ar_ptr=0x7fe740ff70d0, ptr=0x7ffc4aa30138, 
    str=0x7fe8085eb2e0 "malloc(): memory corruption (fast)", action=3) at malloc.c:5006
#4  _int_malloc (av=av@entry=0x7fe5f0000020, bytes=bytes@entry=16) at malloc.c:3386
#5  0x00007fe8084de184 in __GI___libc_malloc (bytes=bytes@entry=16) at malloc.c:2913
#6  0x00007fe7f85004e5 in operator new (sz=16)
    at /home/nwani/m3/conda-bld/compilers_linux-64_1560109574129/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libstdc++-v3/libsupc++/new_op.cc:50
#7  0x00007fe7f8abfc96 in void std::vector<at::Tensor, std::allocator<at::Tensor> >::_M_realloc_insert<at::Tensor>(__gnu_cxx::__normal_iterator<at::Tensor*, std::vector<at::Tensor, std::allocator<at::Tensor> > >, at::Tensor&&) ()
   from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so
#8  0x00007fe7cb1bf9ea in at::compute_common_type_(c10::ArrayRef<at::OperandInfo>) ()
   from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so
#9  0x00007fe7cb1c084b in at::TensorIterator::compute_types() ()
   from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so
#10 0x00007fe7cb1c1d6c in at::TensorIterator::build() () from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so
#11 0x00007fe7cb1c24b5 in at::TensorIterator::binary_op(at::Tensor&, at::Tensor const&, at::Tensor const&, bool) ()
   from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so
#12 0x00007fe7caf42665 in at::native::add_out(at::Tensor&, at::Tensor const&, at::Tensor const&, c10::Scalar) ()
   from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so
#13 0x00007fe7cd7a6ccd in at::CUDAType::(anonymous namespace)::add_(at::Tensor&, at::Tensor const&, c10::Scalar) ()
   from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so
#14 0x00007fe7cd0c5736 in torch::autograd::VariableType::(anonymous namespace)::add_(at::Tensor&, at::Tensor const&, c10::Scalar)
    () from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so
#15 0x00007fe7cafc78ff in at::Tensor::add_(at::Tensor const&, c10::Scalar) const ()
   from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so
#16 0x00007fe7cd34edda in torch::autograd::AccumulateGrad::apply(std::vector<torch::autograd::Variable, std::allocator<torch::autograd::Variable> >&&) () from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so
#17 0x00007fe7cd34a3f6 in torch::autograd::Node::operator()(std::vector<torch::autograd::Variable, std::allocator<torch::autograd::Variable> >&&) () from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so
#18 0x00007fe7cd343a07 in torch::autograd::Engine::evaluate_function(torch::autograd::NodeTask&) ()
   from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so
#19 0x00007fe7cd345a14 in torch::autograd::Engine::thread_main(torch::autograd::GraphTask*) ()
   from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so
#20 0x00007fe7f8a5a5ca in torch::autograd::python::PythonEngine::thread_init(int) ()
   from /root/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so
#21 0x00007fe7f851c19d in std::execute_native_thread_routine (__p=0x559f1840b6c0)
    at /home/nwani/m3/conda-bld/compilers_linux-64_1560109574129/work/.build/x86_64-conda_cos6-linux-gnu/src/gcc/libstdc++-v3/src/c++11/thread.cc:80
#22 0x00007fe80882b6ba in start_thread (arg=0x7fe740ff9700) at pthread_create.c:333
#23 0x00007fe80856141d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

cc @ezyang @gchanan @zou3519 @jerryzh168