Repository: pykeen/pykeen
Issue: Failures in Hyper-parameter Optimization - Trial 1 failed, because the value None could not be cast to float
Root cause: Validation Gap

**Describe the bug**
When doing a HPO, there were failure messages as below.
```
[W 2021-07-21 12:22:47,380] Trial 1 failed, because the value None could not be cast to float.
INFO:pykeen.pipeline.api:loaded random seed 3043021532 from checkpoint.
INFO:pykeen.training.training_loop:=> loading checkpoint '/root/.data/pykeen/checkpoints/checkpoint_transR_optim.pt'
[W 2021-07-21 12:22:47,408] Trial 2 failed, because the value None could not be cast to float.
```
Full message:
```
[I 2021-07-21 12:21:50,108] A new study created in memory with name: no-name-93e3c6a6-c0c1-4bb1-b2f1-0133b5bd7414
/usr/local/lib/python3.7/dist-packages/optuna/distributions.py:563: UserWarning:

The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].

No random seed is specified. Setting to 3043021532.
INFO:pykeen.training.training_loop:=> no checkpoint found at '/root/.data/pykeen/checkpoints/checkpoint_transR_optim.pt'. Creating a new file.
Training epochs on cuda: 100%
100/100 [00:44<00:00, 2.25epoch/s, loss=0.000649, prev_loss=0.000663]
INFO:pykeen.training.training_loop:=> Saved checkpoint after having finished epoch 100.
INFO:pykeen.evaluation.evaluator:Starting batch_size search for evaluation now...

INFO:pykeen.evaluation.evaluator:Concluded batch_size search with batch_size=512.
Evaluating on cuda: 100%
432/432 [00:00<00:00, 4.59ktriple/s]
INFO:pykeen.evaluation.evaluator:Evaluation took 0.09s seconds
[I 2021-07-21 12:22:47,332] Trial 0 finished with value: 0.8280269936201086 and parameters: {'model.embedding_dim': 256, 'model.relation_dim': 240, 'model.scoring_fct_norm': 2, 'loss.margin': 2, 'loss.margin_activation': 'softplus', 'optimizer.lr': 0.0987811842800941, 'negative_sampler.num_negs_per_pos': 9, 'training.num_epochs': 100, 'training.batch_size': 1432}. Best is trial 0 with value: 0.8280269936201086.
INFO:pykeen.pipeline.api:loaded random seed 3043021532 from checkpoint.
INFO:pykeen.training.training_loop:=> loading checkpoint '/root/.data/pykeen/checkpoints/checkpoint_transR_optim.pt'
[W 2021-07-21 12:22:47,380] Trial 1 failed, because the value None could not be cast to float.
INFO:pykeen.pipeline.api:loaded random seed 3043021532 from checkpoint.
INFO:pykeen.training.training_loop:=> loading checkpoint '/root/.data/pykeen/checkpoints/checkpoint_transR_optim.pt'
[W 2021-07-21 12:22:47,408] Trial 2 failed, because the value None could not be cast to float.
```

**To Reproduce**
Codes:
```
......
results_optim = hpo_pipeline(
    n_trials=3,
    training=training,
    testing=testing,
    validation=validation,
    model='TransR',
    training_kwargs=dict(
        checkpoint_name='checkpoint_transR_optim.pt',
        checkpoint_frequency=2,)
)
```

**Expected behavior**
All of the `n_trials` are conducted and checkpoints saved.

**Environment:**
Output by using `pykeen.env()` in Google Colab:

<html>
<body>
<!--StartFragment-->

Key | Value
-- | --
OS | posix
Platform | Linux
Release | 5.4.104+
User | root
Time | Wed Jul 21 12:20:19 2021
Python | 3.7.11
PyKEEN | 1.5.1-dev
PyKEEN Hash | UNHASHED
PyKEEN Branch |  
PyTorch | 1.9.0+cu102
CUDA Available? | true
CUDA Version | 10.2
cuDNN Version | 7605

<!--EndFragment-->
</body>
</html>

**Additional information**
I am using Google Colab


This is the same issue as #466 - @cthoyt do you know whether there was some progress? (also mentioning @mali-git since he knows well about the hpo code).

There is no progress, I haven't had the time to look into this. It would be nice if there were a minimal example (https://github.com/pykeen/pykeen/issues/466#issuecomment-867973913 suggests a large example with DRKG and CompGCN - I don't think this is appropriate for writing unit tests or quick iteration).



Thanks @mberr @cthoyt . I tried again without checkpoint arguments and it is ok. Would the checkpoints be the reason? 

BTW I can provide my small dataset here for quick unit tests if it is suitable @cthoyt. The dataset has only 166 entities, 5 relations, 2158 triples.

I could reproduce the error with
```python
results_optim = hpo_pipeline(
    n_trials=3,
    dataset="nations",
    model="TransR",
    training_kwargs=dict(
        checkpoint_name="checkpoint.pt",
        checkpoint_frequency=2,
        num_epochs=4,
    ),
)
```

<summary>Full Log</summary>
<details>

```
[I 2021-07-23 15:33:15,914] A new study created in memory with name: no-name-1e0311ef-b612-46b5-b3c9-f56c03482ada
INFO:pykeen.hpo.hpo:Using model: <class 'pykeen.models.unimodal.trans_r.TransR'>
INFO:pykeen.hpo.hpo:Using loss: <class 'pykeen.losses.MarginRankingLoss'>
INFO:pykeen.hpo.hpo:Using optimizer: <class 'torch.optim.adam.Adam'>
INFO:pykeen.hpo.hpo:Using training loop: <class 'pykeen.training.slcwa.SLCWATrainingLoop'>
INFO:pykeen.hpo.hpo:Using negative sampler: <class 'pykeen.sampling.basic_negative_sampler.BasicNegativeSampler'>
INFO:pykeen.hpo.hpo:Using evaluator: <class 'pykeen.evaluation.rank_based_evaluator.RankBasedEvaluator'>
INFO:pykeen.hpo.hpo:Attempting to maximize adjusted_arithmetic_mean_rank_index
INFO:pykeen.hpo.hpo:Filter validation triples when testing: True
~/venv/pykeen/lib/python3.9/site-packages/optuna/distributions.py:562: UserWarning: The distribution is specified by [32, 4000] and step=100, but the range is not divisible by `step`. It will be replaced by [32, 3932].
  warnings.warn(
INFO:pykeen.pipeline.api:=> no training loop checkpoint file found at '~/.data/pykeen/checkpoints/checkpoint.pt'. Creating a new file.
WARNING:pykeen.pipeline.api:No random seed is specified. Setting to 2488991068.
WARNING:pykeen.utils:No cuda devices were available. The model runs on CPU
INFO:pykeen.training.training_loop:=> no checkpoint found at '~/.data/pykeen/checkpoints/checkpoint.pt'. Creating a new file.
Training epochs on cpu: 100%|██████████| 4/4 [00:11<00:00,  2.93s/epoch, loss=0.00468, prev_loss=0.00501]
INFO:pykeen.training.training_loop:=> Saved checkpoint after having finished epoch 4.
INFO:pykeen.evaluation.evaluator:Currently automatic memory optimization only supports GPUs, but you're using a CPU. Therefore, the batch_size will be set to the default value.
INFO:pykeen.evaluation.evaluator:No evaluation batch_size provided. Setting batch_size to '32'.
Evaluating on cpu: 100%|██████████| 199/199 [00:00<00:00, 6.89ktriple/s]
INFO:pykeen.evaluation.evaluator:Evaluation took 0.03s seconds
[I 2021-07-23 15:33:30,417] Trial 0 finished with value: 0.3116339026748308 and parameters: {'model.embedding_dim': 80, 'model.relation_dim': 112, 'model.scoring_fct_norm': 1, 'loss.margin': 3, 'loss.margin_activation': 'relu', 'optimizer.lr': 0.07420980762193746, 'negative_sampler.num_negs_per_pos': 27, 'training.batch_size': 732}. Best is trial 0 with value: 0.3116339026748308.
INFO:pykeen.pipeline.api:loaded random seed 2488991068 from checkpoint.
WARNING:pykeen.utils:No cuda devices were available. The model runs on CPU
INFO:pykeen.training.training_loop:=> loading checkpoint '~/.data/pykeen/checkpoints/checkpoint.pt'
[W 2021-07-23 15:33:30,457] Trial 1 failed, because the value None could not be cast to float.
INFO:pykeen.pipeline.api:loaded random seed 2488991068 from checkpoint.
WARNING:pykeen.utils:No cuda devices were available. The model runs on CPU
INFO:pykeen.training.training_loop:=> loading checkpoint '~/.data/pykeen/checkpoints/checkpoint.pt'
[W 2021-07-23 15:33:30,492] Trial 2 failed, because the value None could not be cast to float.
```

</details>

The problem seems to be that the second trial loads the checkpoint instead of training a new model. It does not happen when you do not specify a checkpoint.

```python
results_optim = hpo_pipeline(
    n_trials=3,
    dataset="nations",
    model="TransR",
    training_kwargs=dict(
        num_epochs=4,
    ),
)
```

I guess, we shouldn't do this anyway, or use a trial-specific path for checkpointing.

Thanks @mberr . What do you mean by "use a trial-specific path for checkpointing"? It seems that the [`hpo_pipeline`](https://pykeen.readthedocs.io/en/stable/reference/hpo.html#:~:text=hpo_pipeline(*%2C%20dataset%3DNone,None%2C%20save_model_directory%3DNone)) doesn't has the checkpoint parameters. If I pass it in `training_kwargs` then it would be the error here, as you mentioned the second trial loads the checkpoint instead of training a new model.

@cthoyt is there any update on this issue?

I started #450 to build a better framework for writing unit tests for code that touches ablation studies and HPO, but I haven't been very motivated to revisit that. Would be happy to accept an external PR to address this issue specifically that includes a reproducible unit test that doesn't take a long time to run.

Just to clear my confusion, @cthoyt you would like a new PR with the possibility of fixing this bug, or would you like me to look into your PR and see if I can make things work there?

My PR #450 does not solve the problem described in this issue, but it lays the groundwork for looking into things like this in the future. I don't need any help on that one.

If you're also experiencing this issue and want to submit a bugfix, that would be great

Ah, I see. If I am able to find a solution to fix this I would send a PR for the same. 👍 

Hi @Hao-666 , @YojanaGadiya ,

with #724 being merged this issue should be resolved. It will also be distributed with upcoming 1.7 release :rocket: