Repository: pytorch
Issue: Pytorch num_worker>0 code worked first time and then it never worked with same setting again
Labels: module: dataloader,triaged

i was trying to run this code on my local machine on rtx 3060ti : https://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-train-amp-aug

with batch size 16 i can run vovnet on my local machine,,,the problem that i am facing is very strange, the code worked fined,,after 2-3 epoch it became very very slow and i saw in spyder that memory consumption is increasing,,i just restarted spyder,removed variable and even shut down my pc several times and reduced num_workers but that same code doesn't work anymore,,,now to run that code i need to set batch size = 2 instead of 16 and if i don't do that then that same code that worked few minutes ago will show me this error : 


```
runfile('C:/Users/Mobassir/.spyder-py3/temp.py', wdir='C:/Users/Mobassir/.spyder-py3')
Using device: cuda
Training with 1 started
17117 4280
model loaded
  0%|          | 0/1070 [00:04<?, ?it/s]
Traceback (most recent call last):

  File "C:\Users\Mobassir\.spyder-py3\temp.py", line 1161, in <module>
    train_one_epoch(epoch, model, optimizer, train_loader, device, scheduler=scheduler, schd_batch_update=False)

  File "C:\Users\Mobassir\.spyder-py3\temp.py", line 1004, in train_one_epoch
    scaler.scale(loss).backward()

  File "C:\Users\Mobassir\anaconda3\envs\kaggle\lib\site-packages\torch\tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)

  File "C:\Users\Mobassir\anaconda3\envs\kaggle\lib\site-packages\torch\autograd\__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(

RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 8.00 GiB total capacity; 1.48 GiB already allocated; 0 bytes free; 2.90 GiB reserved in total by PyTorch)
```

out of frustration i uninstalled cuda,cudnn and reinstalled but it happens again,,,for the first run,i will get no error and after 2-3 epoch training will become super slow and then if i restart and retry means same code won't work again anymore,i can;t understand how to solve this issue,is it related to autograd or what?

i tried to use num_worker = 4 and pin_memory = 1
is it causing issue?
how can i fix this? everything else is exactly same as this kernel : https://www.kaggle.com/khyeh0719/pytorch-efficientnet-baseline-train-amp-aug

my question : 

why the code that worked 5 minutes ago,doesn't work again anymore with that exact setting?(even after restarting computer?)
even with half batch size i get that cuda oom error.
how can i fix this?

sometimes i get this error : 


```
Traceback (most recent call last):

  File "C:\Users\Mobassir\.spyder-py3\temp.py", line 1161, in <module>
    train_one_epoch(epoch, model, optimizer, train_loader, device, scheduler=scheduler, schd_batch_update=False)

  File "C:\Users\Mobassir\.spyder-py3\temp.py", line 1009, in train_one_epoch
    running_loss = running_loss * .99 + loss.item() * .01

RuntimeError: CUDA error: an illegal memory access was encountered
```
when i re-run..

about my environment : 

i used, anaconda virtual environment where i have, 
* python 3.8.5
* pytorch 1.7.0
* cuda 11.0 
* cudnn 8004
* gpu rtx 3060ti 
* Is CUDA available: Yes 
* spyder

cc @SsnL @VitalyFedyunin @ejguan