Repository: pykeen/pykeen
Issue: RuntimeError: CUDA out of memory when training with a large file
Root cause: Resource Mishandling

I am facing `CUDA out of memory` when training with a large file. I have changed the batch size to 16 but it does not works. Could you please help? Thanks.

```python
    DATA_PATH = A_FILE_OF_1.18GB

    tfac = TriplesFactory.from_path(DATA_PATH)
    training, testing, validation = tfac.split([.8, .1, .1])
    result = pipeline(
        training=training,
        testing=testing,
        validation=validation,
        model='TransE',
        stopper='early',
        training_kwargs=dict(num_epochs=5, batch_size=16),
    )
```

16 might still be too big depending on your hyperparameters. Before specifying at 16, did you use automatic memory optimization (this is invoked by default if no batch size is given)?

> 16 might still be too big depending on your hyperparameters. Before specifying at 16, did you use automatic memory optimization (this is invoked by default if no batch size is given)?

Thank you for your reply.

Yes, the reason I set 16 is that I find the default value of batch size in the source code is 256. I did not set any batch size before, but there was the same issue.

Hmm, well it's tough to say just from the code you've given. Maybe you can share the output of `pykeen.env()` and the full error stack trace?

> Hmm, well it's tough to say just from the code you've given. Maybe you can share the output of `pykeen.env()` and the full error stack trace?

Here there are.

···
| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 4.4.0-142-generic        |
| User            | XXX                      |
| Time            | Wed Aug 18 21:13:03 2021 |
| Python          | 3.8.10                   |
| PyKEEN          | 1.5.0                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.6.0                    |
| CUDA Available? | true                     |
| CUDA Version    | 10.2                     |
| cuDNN Version   | 7605                     |


···

Stack
```python-traceback
Traceback (most recent call last):
  File "trainkge.py", line 38, in <module>
    mpk = MyPyKeen(temp)
  File "/XXX/my/xmunlp-ndap-common-data/kg//../kg/mypykeen.py", line 20, in __init__
    result = pipeline(
  File "/XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/pykeen/pipeline/api.py", line 1016, in pipeline
    losses = training_loop_instance.train(
  File "/XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/pykeen/training/training_loop.py", line 301, in train
    result = self._train(
  File "/XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/pykeen/training/training_loop.py", line 689, in _train
    raise e
  File "/XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/pykeen/training/training_loop.py", line 598, in _train
    batch_loss = self._forward_pass(
  File "/XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/pykeen/training/training_loop.py", line 760, in _forward_pass
    loss.backward()
  File "/XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/autograd/__init__.py", line 125, in backward
    Variable._execution_engine.run_backward(
RuntimeError: CUDA out of memory. Tried to allocate 1.40 GiB (GPU 0; 10.76 GiB total capacity; 8.59 GiB already allocated; 173.12 MiB free; 9.82 GiB reserved in total by PyTorch)
Exception raised from malloc at /opt/conda/conda-bld/pytorch_1595629395347/work/c10/cuda/CUDACachingAllocator.cpp:272 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x4d (0x7fa1cb6b977d in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x20626 (0x7fa1cb911626 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #2: <unknown function> + 0x214f4 (0x7fa1cb9124f4 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x21b81 (0x7fa1cb912b81 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)
frame #4: at::native::empty_cuda(c10::ArrayRef<long>, c10::TensorOptions const&, c10::optional<c10::MemoryFormat>) + 0x249 (0x7fa1ce813e39 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #5: <unknown function> + 0xd15c49 (0x7fa1cc834c49 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #6: <unknown function> + 0xd2fa77 (0x7fa1cc84ea77 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #7: <unknown function> + 0xe450dd (0x7fa202c590dd in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #8: <unknown function> + 0xe453f7 (0x7fa202c593f7 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::empty(c10::ArrayRef<long>, c10::TensorOptions const&, c10::optional<c10::MemoryFormat>) + 0xfa (0x7fa202d63e7a in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #10: at::TensorIterator::fast_set_up(at::TensorIteratorConfig const&) + 0x56a (0x7fa2029f13ea in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #11: at::TensorIterator::build(at::TensorIteratorConfig&) + 0x76 (0x7fa2029f5456 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #12: at::TensorIterator::TensorIterator(at::TensorIteratorConfig&) + 0xdd (0x7fa2029f5abd in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #13: at::TensorIterator::binary_op(at::Tensor&, at::Tensor const&, at::Tensor const&, bool) + 0x14a (0x7fa2029f5c6a in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #14: at::native::add(at::Tensor const&, at::Tensor const&, c10::Scalar) + 0x47 (0x7fa202732b77 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #15: <unknown function> + 0xd001c0 (0x7fa1cc81f1c0 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cuda.so)
frame #16: <unknown function> + 0x7f66b4 (0x7fa20260a6b4 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #17: at::add(at::Tensor const&, at::Tensor const&, c10::Scalar) + 0x183 (0x7fa202d3a9e3 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #18: <unknown function> + 0x2be54e1 (0x7fa2049f94e1 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #19: <unknown function> + 0x7f66b4 (0x7fa20260a6b4 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #20: at::Tensor::add(at::Tensor const&, c10::Scalar) const + 0x183 (0x7fa202eae3d3 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #21: <unknown function> + 0x30dd790 (0x7fa204ef1790 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #22: <unknown function> + 0x30de4e3 (0x7fa204ef24e3 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #23: torch::autograd::Engine::evaluate_function(std::shared_ptr<torch::autograd::GraphTask>&, torch::autograd::Node*, torch::autograd::InputBuffer&, std::shared_ptr<torch::autograd::ReadyQueue> const&) + 0xb80 (0x7fa204edffe0 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #24: torch::autograd::Engine::thread_main(std::shared_ptr<torch::autograd::GraphTask> const&) + 0x451 (0x7fa204ee1401 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #25: torch::autograd::Engine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x89 (0x7fa204ed9579 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so)
frame #26: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<torch::autograd::ReadyQueue> const&, bool) + 0x4a (0x7fa2092031ba in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #27: <unknown function> + 0xc9067 (0x7fa21ffa0067 in /XXX/miniconda3/envs/py38ndap/lib/python3.8/site-packages/pandas/_libs/window/../../../../../libstdc++.so.6)
frame #28: <unknown function> + 0x76ba (0x7fa23606d6ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #29: clone + 0x6d (0x7fa23548b4dd in /lib/x86_64-linux-gnu/libc.so.6)
```


Are you sharing your GPU with other students?

> Are you sharing your GPU with other students?

I had checked that no one used GPU before I ran the script.

@GabrielLin 

You can also try to reduce the embedding dimension, which defaults to 50.
https://github.com/pykeen/pykeen/blob/8405d82db33e6a6cda2d237ed8d2b719c578483e/src/pykeen/models/unimodal/trans_e.py#L58

To do so, use the `model_kwargs`, e.g.,
```python
result = pipeline(
	...,
	model_kwargs=dict(embedding_dim=16),
)
```
From my experience, L2 norm is more efficient memory-wise, while by default L1 is used for TransE. This can be changed by
```python
result = pipeline(
	...,
	model_kwargs=dict(scoring_fct_norm=2),
)
```


> @GabrielLin
> 
> You can also try to reduce the embedding dimension, which defaults to 50.
> https://github.com/pykeen/pykeen/blob/8405d82db33e6a6cda2d237ed8d2b719c578483e/src/pykeen/models/unimodal/trans_e.py#L58
> 
> To do so, use the `model_kwargs`, e.g.,
> 
> ```python
> result = pipeline(
> 	...,
> 	model_kwargs=dict(embedding_dim=16),
> )
> ```
> 
> From my experience, L2 norm is more efficient memory-wise, while by default L1 is used for TransE. This can be changed by
> 
> ```python
> result = pipeline(
> 	...,
> 	model_kwargs=dict(scoring_fct_norm=2),
> )
> ```

The solution works for me. Thank you @mberr , @cthoyt