Repository: pykeen/pykeen
Issue: HPO pipeline: trial fails because returned value is None
Root cause: Resource Mishandling

**Describe the bug**
I run a hpo_pipeline and it trains but occasionally the evaluation ends with the following message:

```
Trial 29 failed, because the returned value from the objective function cannot be cast to float. Returned value is: None
```

EDIT: I tried to have a test run on CPU (different pytorch installation), however, on CPU it doesnt even train and every single trial fails with this error message. I am using the same code in both the GPU and the CPU run.

**To Reproduce**
```python
from pykeen.hpo import hpo_pipeline

evaluation_relation_whitelist = {'CtD'}

result = hpo_pipeline(
    training=training,
    validation=validation,
    testing=testing,
    n_trials=30,
    model=model,
    stopper='early',
    stopper_kwargs=dict(frequency=5, patience=2, relative_delta=0.002, metric='mean_reciprocal_rank'),
    evaluator="rankbased",
    metric="mean_reciprocal_rank",
    direction="maximize",
    negative_sampler_kwargs=dict(num_negs_per_pos=1),
    evaluator_kwargs=dict(filtered= True,
    ks = {1,3,5,10,20}),
    n_jobs = 5,
    evaluation_relation_whitelist=evaluation_relation_whitelist
)
```

**Expected behavior**
It just returns the trial information.

**Environment (please complete the following information):**
 
    OS: Ubuntu 18.04.5 LTS
    Python version: 3.8.5
    Version of this software: 1.0.6
    Versions of required Python packages:
        click 7.1.2
        click-default-group 1.2.2
        numpy 1.19.2
        torch 1.6.0
        torch-cluster 1.5.7
        torch-geometric 1.6.1
        torch-scatter 2.0.5
        torch-sparse 0.6.7
        torch-spline-conv 1.2.0
        tqdm 4.51.0

**Additional information**
The dataset I am running this on is a prestratified version of Hetionet that I am feeding via the triple factories training, validation, testing.


Hi @mali-git ,

you are right, it says `CUDA out of memory. Tried to allocate 794.00 MiB (GPU 0; 10.76 GiB total capacity; 7.31 GiB already allocated; 558.19 MiB free; 9.44 GiB reserved in total by PyTorch)`.

In some cases my GPU and VRAM is not even at 50% for long stretches of time (with 5 jobs concurrently running). And then, with larger embeddings or a different model it goes out of memory with 5 jobs. How can I improve my pipeline performance with multiple jobs without running the risk of going out of memory?

Hi @fratajcz,

in general, I would only recommend running multiple HPO-jobs on a single GPU if you have enough insights about the memory consumption of the sampled models configurations (i.e., available GPU-RAM vs. memory consumption of the largest model configuration that can be sampled). Please consider that the memory consumption during training and evaluation will differ most likely, too, since, during the evaluation, we maximize the batch-size to accelerate the evaluation-procedure (in case that automatic memory optimization is used).
Therefore, it is safer only to run a single HPO-job on a GPU. Still, you can encounter CUDA out of memory exceptions, but in these cases, PyKEEN will sample a new model configuration. 

We will improve the handling of the OOM exceptions within the HPO-pipeline to avoid the confusing error message mentioned above.


> Hi @fratajcz,
> 
> in general, I would only recommend running multiple HPO-jobs on a single GPU if you have enough insides about the memory consumption of the sampled models configurations (i.e., available GPU-RAM vs. memory consumption of the largest model configuration that can be sampled). Please consider that the memory consumption during training and evaluation will differ most likely, too, since, during the evaluation, we maximize the batch-size to accelerate the evaluation-procedure (in case that automatic memory optimization is used).
> Therefore, it is safer only to run a single HPO-job on a GPU. Still, you can encounter CUDA out of memory exceptions, but in these cases, PyKEEN will sample a new model configuration.
> 
> We will improve the handling of the OOM exceptions within the HPO-pipeline to avoid the confusing error message mentioned above.

If you want to run several training-procedures concurrently, you need to set the training and evaluation batch-size yourself and ensure that they don't exceed the available memory. For running a single experiment, the [automatic memory optimization](https://pykeen.readthedocs.io/en/latest/tutorial/performance.html#automated-memory-optimization) ensures that the memory will not be exceeded (assuming that the current run has sole access to the hardware).