Repository: pytorch
Issue: Reduce "reserved" memory by PyTorch.
Labels: module: cuda,module: memory usage,triaged

I am using Facebook's Detectron-2 custom training. 

But I am getting error for: **RuntimeError: CUDA out of memory. Tried to allocate 48.00 MiB (GPU 0; 15.90 GiB total capacity; 14.55 GiB already allocated; 33.81 MiB free; 15.08 GiB reserved in total by PyTorch)**

Installing PyTorch:
```
## install dependencies: (use cu101 because colab has CUDA 10.1)
!pip install -U torch==1.5 torchvision==0.6 -f https://download.pytorch.org/whl/cu101/torch_stable.html 
import torch, torchvision
print(torch.__version__, torch.cuda.is_available())
```
Training Code:

```
#We are importing our own Trainer Module here to use the COCO validation evaluation during training. Otherwise no validation eval occurs.
from detectron2.engine import DefaultTrainer
from detectron2.evaluation import COCOEvaluator
class CocoTrainer(DefaultTrainer):
  @classmethod
  def build_evaluator(cls, cfg, dataset_name, output_folder=None):
    if output_folder is None:
        os.makedirs("coco_eval", exist_ok=True)
        output_folder = "coco_eval"
    return COCOEvaluator(dataset_name, cfg, False, output_folder)
```
```
#from .detectron2.tools.train_net import Trainer
#from detectron2.engine import DefaultTrainer
from detectron2.config import get_cfg
#from detectron2.evaluation.coco_evaluation import COCOEvaluator
import os

cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x.yaml"))
cfg.DATASETS.TRAIN = ("custom_train",)
cfg.DATASETS.TEST = ("custom_val",)

cfg.DATALOADER.NUM_WORKERS = 4
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url("COCO-Detection/faster_rcnn_X_101_32x8d_FPN_3x.yaml")  # Let training initialize from model zoo
cfg.SOLVER.IMS_PER_BATCH = 4
cfg.SOLVER.BASE_LR = 0.001

cfg.SOLVER.WARMUP_ITERS = 100
cfg.SOLVER.MAX_ITER = 150 #adjust up if val mAP is still rising, adjust down if overfit
cfg.SOLVER.STEPS = (100, 150)
cfg.SOLVER.GAMMA = 0.05

cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 64
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1 #your number of classes

cfg.TEST.EVAL_PERIOD = 50

os.makedirs(cfg.OUTPUT_DIR, exist_ok=True)
trainer = CocoTrainer(cfg)
trainer.resume_or_load(resume=False)
trainer.train()
```
Error Log:

```
[10/15 06:28:58 d2.data.datasets.coco]: Loaded 13 images in COCO format from /content/Test01/train/train.json
[10/15 06:28:58 d2.data.build]: Removed 0 images with no usable annotations. 13 images left.
[10/15 06:28:58 d2.data.common]: Serializing 13 elements to byte tensors and concatenating them all ...
[10/15 06:28:58 d2.data.common]: Serialized dataset takes 0.01 MiB
[10/15 06:28:58 d2.data.detection_utils]: TransformGens used in training: [ResizeShortestEdge(short_edge_length=(640, 672, 704, 736, 768, 800), max_size=1333, sample_style='choice'), RandomFlip()]
[10/15 06:28:58 d2.data.build]: Using training sampler TrainingSampler

Skip loading parameter 'roi_heads.box_predictor.cls_score.weight' to the model due to incompatible shapes: (81, 1024) in the checkpoint but (2, 1024) in the model! You might want to double check if this is expected.
Skip loading parameter 'roi_heads.box_predictor.cls_score.bias' to the model due to incompatible shapes: (81,) in the checkpoint but (2,) in the model! You might want to double check if this is expected.
Skip loading parameter 'roi_heads.box_predictor.bbox_pred.weight' to the model due to incompatible shapes: (320, 1024) in the checkpoint but (4, 1024) in the model! You might want to double check if this is expected.
Skip loading parameter 'roi_heads.box_predictor.bbox_pred.bias' to the model due to incompatible shapes: (320,) in the checkpoint but (4,) in the model! You might want to double check if this is expected.

[10/15 06:28:58 d2.engine.train_loop]: Starting training from iteration 0
ERROR [10/15 06:28:59 d2.engine.train_loop]: Exception during training:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/detectron2/engine/train_loop.py", line 132, in train
    self.run_step()
  File "/usr/local/lib/python3.6/dist-packages/detectron2/engine/train_loop.py", line 215, in run_step
    loss_dict = self.model(data)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/meta_arch/rcnn.py", line 121, in forward
    features = self.backbone(images.tensor)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/backbone/fpn.py", line 123, in forward
    bottom_up_features = self.bottom_up(x)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/backbone/resnet.py", line 463, in forward
    x = stage(x)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/container.py", line 100, in forward
    input = module(input)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/backbone/resnet.py", line 204, in forward
    out = self.conv2(out)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/detectron2/layers/wrappers.py", line 95, in forward
    x = self.norm(x)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/detectron2/layers/batch_norm.py", line 55, in forward
    return x * scale + bias
RuntimeError: CUDA out of memory. Tried to allocate 48.00 MiB (GPU 0; 15.90 GiB total capacity; 14.55 GiB already allocated; 33.81 MiB free; 15.08 GiB reserved in total by PyTorch)
[10/15 06:28:59 d2.engine.hooks]: Total training time: 0:00:00 (0:00:00 on hooks)

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-19-8118867f3ffb> in <module>()
     28 trainer = CocoTrainer(cfg)
     29 trainer.resume_or_load(resume=False)
---> 30 trainer.train()

16 frames

/usr/local/lib/python3.6/dist-packages/detectron2/layers/batch_norm.py in forward(self, x)
     53             scale = scale.reshape(1, -1, 1, 1)
     54             bias = bias.reshape(1, -1, 1, 1)
---> 55             return x * scale + bias
     56         else:
     57             # When gradients are not needed, F.batch_norm is a single fused op

RuntimeError: CUDA out of memory. Tried to allocate 48.00 MiB (GPU 0; 15.90 GiB total capacity; 14.55 GiB already allocated; 33.81 MiB free; 15.08 GiB reserved in total by PyTorch)
```

cc @ngimel