Repository: pytorch
Issue: CUDA error: device-side assert triggered(insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:569)
Labels: module: cuda,triaged

## 🐛 Bug
During the training of image classification model, I met some problem:
    losses.update(loss.item(), input.size(0))
RuntimeError: CUDA error: device-side assert triggered
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:569)
frame #0: std::function<std::string ()>::operator()() const + 0x11 (0x7f401ca385b1 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x2a (0x7f401ca3773a in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #2: <unknown function> + 0x117bd (0x7f401cc627bd in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x15ab3 (0x7f401cc66ab3 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #4: c10::TensorImpl::release_resources() + 0x66 (0x7f401ca24c46 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #5: <unknown function> + 0x199decb (0x7f3f80f64ecb in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #6: <unknown function> + 0x336e090 (0x7f3f82935090 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #7: <unknown function> + 0x392feca (0x7f3f82ef6eca in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #8: torch::autograd::deleteFunction(torch::autograd::Function*) + 0xa2 (0x7f3f82ef6f82 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #9: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x45 (0x7f401d60d985 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #10: torch::autograd::Variable::AutogradMeta::~AutogradMeta() + 0x77 (0x7f3f82f0bcf7 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #11: torch::autograd::Variable::AutogradMeta::~AutogradMeta() + 0x9 (0x7f3f82f0bd39 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch.so)
frame #12: c10::TensorImpl::release_resources() + 0x20 (0x7f401ca24c00 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #13: <unknown function> + 0x1fc1cb (0x7f401d60e1cb in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #14: <unknown function> + 0x4844d4 (0x7f401d8964d4 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #15: <unknown function> + 0x484521 (0x7f401d896521 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #28: __libc_start_main + 0xf0 (0x7f40220ee830 in /lib/x86_64-linux-gnu/libc.so.6) 

## Environment
 - PyTorch Version 1.2 stable
 - docker container ubuntu 16.04
 - conda install 
 - Python version: 3.6
 - CUDA/cuDNN version: cuda 10.0 cudnn 7.6
 - GPU models and configuration: 1080Ti cards
 - Any other relevant information:

I found when I run the training with batchsize 1024 on 8 1080 ti gpu cards, it will report the error. When I run the code with batchsize 512 on 8 gpu cards, the error disappeared. when I run the code with smaller batchsize on a single gpu card, the error can’t be reproduced. So it might be hard to reproduce the error on cpu.