Repository: pytorch
Issue: Check failed: error == cudaSuccess unspecified launch failure [caffe2]
Labels: caffe2

When I tried to resume training the maskrcnn using Detectron. The training progress goes well at the begining, but alone with the training continues, the training time for each iter grows progressly, after handreds or thousands of iterations, the training broke down with the cuda error below:

> *** Check failure stack trace: ***
> F0628 15:21:28.669647  1910 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669657  1911 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669684  1904 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669684  1909 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669703  1914 context_gpu.h:107] C06ck failed: error == cudaSuccess unspecified launch failure
> 28 15:21:28.669694  1906 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669710  1900 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670411  1912 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670415  1903 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failure
> *** Check failure stack trace: ***
> F0628 15:21:28.669647  1910 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669657  1911 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669684  1904 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669684  1909 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669703  1914 context_gpu.h:107] C06ck failed: error == cudaSuccess unspecified launch failure
> 28 15:21:28.669694  1906 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669710  1900 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670411  1912 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670415  1903 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670450  1902 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failure
> *** Check failure stack trace: ***
> F0628 15:21:28.669647  1910 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669657  1911 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669684  1904 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669684  1909 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669703  1914 context_gpu.h:107] C06ck failed: error == cudaSuccess unspecified launch failure
> 28 15:21:28.669694  1906 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669710  1900 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670411  1912 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670415  1903 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670450  1902 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670488  1913 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failure
> E0628 15:21:28.669669  1907 net_dag.cc:195] Secondary exception from operator chain starting at '' (type 'Add'): caffe2::EnforceNotMet: [enforce fail at context_gpu.h:156] . Encountered CUDA error: unspecified launch failure Error from operator:
> input: "gpu_0/mask_fcn_logits_w_grad" input: "gpu_1/mask_fcn_logits_w_grad" output: "gpu_0/mask_fcn_logits_w_grad" name: "" type: "Add" device_option { device_type: 1 cuda_gpu_id: 0 }
> F0628 15:21:28.669647  1910 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669657  1911 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669684  1904 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669684  1909 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669703  1914 context_gpu.h:107] C06ck failed: error == cudaSuccess unspecified launch failure
> 28 15:21:28.669694  1906 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669710  1900 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670411  1912 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670415  1903 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670450  1902 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670488  1913 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failure
> *** Check failure stack trace: ***
> F0628 15:21:28.669647  1910 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669657  1911 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669684  1904 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669684  1909 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669703  1914 context_gpu.h:107] C06ck failed: error == cudaSuccess unspecified launch failure
> 28 15:21:28.669694  1906 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669710  1900 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670411  1912 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670415  1903 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670450  1902 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670488  1913 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureFF0628 15:21:28.669647  1910 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669657  1911 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669684  1904 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669684  1909 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669703  1914 context_gpu.h:107] C06ck failed: error == cudaSuccess unspecified launch failure
> 28 15:21:28.669694  1906 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.669710  1900 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670411  1912 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670415  1903 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670450  1902 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670488  1913 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failureF0628 15:21:28.670533  1907 context_gpu.h:107] Check failed: error == cudaSuccess unspecified launch failure
> *** Check failure stack trace: ***
> *** Check failure stack trace: ***

I googled this error, it seems like something about GPU memory leak, but during training time, GPU memory usage are stable and normal until the progress broke down. I tried to reboot my server but it didn't work, can you help me out with this?

I checked the context_gpu.h at line 107, the code is:

```
~ThreadLocalCUDAObjects() noexcept {
 99     for (int i = 0; i < CAFFE2_COMPILE_TIME_MAX_GPUS; ++i) {
100       for (auto& handle : cublas_handles_[i]) {
101         if (handle) {
102           CUBLAS_CHECK(cublasDestroy(handle));
103         }
104       }
105       for (auto& stream : cuda_streams_[i]) {
106         if (stream) {
107           CUDA_CHECK(cudaStreamDestroy(stream));
108         }
109       }
110       for (auto& handle : cudnn_handles_[i]) {
111         if (handle) {
112           CUDNN_CHECK(cudnnDestroy(handle));
113         }
114       }
115     }
116   }
```