Repository: pytorch
Issue: RuntimeError: $ Torch: not enough memory: you tried to allocate 72GB. Buy new RAM!
Labels: module: memory usage,triaged

- OS: 

Red Hat Enterprise Linux Server release 7.2 (Maipo)

- PyTorch version: 

pytorch                   0.3.1           py35_cuda8.0.61_cudnn7.0.5_2    pytorch
torchvision               0.2.0            py35heaa392f_1    pytorch

- How you installed PyTorch (conda, pip, source):

$ module load anaconda3/4.3.1
$ source activate pytorchenv
$ conda install pytorch torchvision -c pytorch

- Python version:

$ python -V
Python 3.5.5

- CUDA/cuDNN version:

cuda8.0.61/cudnn7.0.5_2

- GPU models and configuration:

$ nvidia-smi
Tue Feb 27 10:06:13 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.69                 Driver Version: 384.69                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00000000:04:00.0 Off |                    0 |
| N/A   28C    P8    26W / 149W |      1MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           On   | 00000000:05:00.0 Off |                    0 |
| N/A   22C    P8    29W / 149W |      1MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           On   | 00000000:84:00.0 Off |                    0 |
| N/A   32C    P8    26W / 149W |      1MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla K80           On   | 00000000:85:00.0 Off |                    0 |
| N/A   24C    P8    29W / 149W |      1MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla K80           On   | 00000000:8A:00.0 Off |                    0 |
| N/A   24C    P8    26W / 149W |      1MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla K80           On   | 00000000:8B:00.0 Off |                    0 |
| N/A   33C    P8    29W / 149W |      1MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla K80           On   | 00000000:8E:00.0 Off |                    0 |
| N/A   26C    P8    26W / 149W |      1MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla K80           On   | 00000000:8F:00.0 Off |                    0 |
| N/A   35C    P8    31W / 149W |      1MiB / 11439MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

- GCC version (if compiling from source):

N/A

In addition, including the following information will also be very helpful for us to diagnose the problem:
- A script to reproduce the bug. Please try to provide as minimal of a test case as possible.
- Error messages and/or stack traces of the bug
- Context around what you are trying to do

Training a model with:
```
...
os.environ['CUDA_VISIBLE_DEVICES'] = "0"
...
	valid_loader = DataLoader(dataset=valid_dataset, batch_size=32, shuffle=False, num_workers=12)
	# ====== start trianing =======

	cudnn.benchmark = True
	N_CLASSES = 8
	BATCH_SIZE = 32
```
I've tried batch sizes from 128 to 8, and using GPUs from just one to all 8. GPU node has plenty of RAM (124G):

$ free
              total        used        free      shared  buff/cache   available
Mem:      131930696     6299776   124697204       16968      933716   124718092
Swap:      16777212      441996    16335216

Do you have a maximum RAM allocation limit hardcoded in PyTorch (file "THGeneral.c")? Thank you in advance!

Complete error message:
```
$ python3 train.py ./out/ > train.out
Traceback (most recent call last):
  File "train.py", line 120, in <module>
    augment_img = torch.stack(augment_img)[perm_index]
  File "/path-to-anaconda/Anaconda3/4.3.1/envs/pytorchenv/lib/python3.5/site-packages/torch/functional.py", line 64, in stack
    return torch.cat(inputs, dim)
RuntimeError: $ Torch: not enough memory: you tried to allocate 72GB. Buy new RAM! at /opt/conda/conda-bld/pytorch_1518241081361/work/torch/lib/TH/THGeneral.c:253
```