Repository: pykeen/pykeen
Issue: Checkpointing in HPO pipeline
Root cause: Architectural Constraint

### What is your question

I wanted to turn on checkpointing on an hpo_pipeline run to be able to later inspect some of the more promising models by hand. Can this be done and if so how?

My naive attempt at doing it translates to this minimal reproducing example:
```python
from pykeen.hpo import hpo_pipeline


pipeline_result = hpo_pipeline(
    dataset="nations",
    model="TransE",
    n_trials=2,
    training_kwargs=dict(
       checkpoint_name="nations_transh_ckpt.pt",
       checkpoint_frequency=1,
   ),
)
```
**Actual behavior**
First trial passes and produces a checkpoint at `~/.data/pykeen/checkpoint/nations_transh_ckpt.pt`.
Second trial fails with `CheckpointMismatchError("The checkpoint file '/home/jason/.data/pykeen/checkpoints/nations_transh_ckpt.pt' that was provided already exists, but seems to be from a different training loop setup.")`

**Expected**
All trials run through and checkpoints are saved for each one of them; if they cannot be saved to the same file, maybe save to auto-incremented file paths (`nations_transh_ckpt_0.pt`, `nations_transh_ckpt_1.pt`, ..)?
Alternatively: An error that checkpointing is not available for hpo_pipeline.

**What else**
Leaving out the explicit `checkpoint_name` training_kwarg and letting pykeen decide on a suitable default would have been my next best guess. But that does not do it but gives (admirably) `WARNING:pykeen.training.training_loop:A checkpoint frequency was set, but no checkpoint file was given. No checkpoints will be created`

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.13.0-28-generic        |
| Time            | Thu Feb 10 12:06:05 2022 |
| Python          | 3.8.12                   |
| PyKEEN          | 1.7.0                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.10.2                   |
| CUDA Available? | true                     |
| CUDA Version    | 11.3                     |
| cuDNN Version   | 8200                     |


### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Full stack trace for completeness:
```python-traceback
Traceback (most recent call last):
  File "/home/jason/anaconda3/envs/pykeen/lib/python3.8/site-packages/optuna/study/_optimize.py", line 213, in _run_trial
    value_or_values = func(trial)
  File "/home/jason/anaconda3/envs/pykeen/lib/python3.8/site-packages/pykeen/hpo/hpo.py", line 304, in __call__
    raise e
  File "/home/jason/anaconda3/envs/pykeen/lib/python3.8/site-packages/pykeen/hpo/hpo.py", line 254, in __call__
    result = pipeline(
  File "/home/jason/anaconda3/envs/pykeen/lib/python3.8/site-packages/pykeen/pipeline/api.py", line 1200, in pipeline
    losses = training_loop_instance.train(
  File "/home/jason/anaconda3/envs/pykeen/lib/python3.8/site-packages/pykeen/training/training_loop.py", line 290, in train
    best_epoch_model_file_path, last_best_epoch = self._load_state(
  File "/home/jason/anaconda3/envs/pykeen/lib/python3.8/site-packages/pykeen/training/training_loop.py", line 1139, in _load_state
    raise CheckpointMismatchError(
pykeen.training.training_loop.CheckpointMismatchError: The checkpoint file '/home/jason/.data/pykeen/checkpoints/nations_transh_ckpt.pt' that was provided already exists, but seems to be from a different training loop setup.
```

Hi did this get solved? Facing the exact same issue!

Hi @jas-ho  / @MatthewCorney ,

sorry for the delay. 

This might not be exactly what you are looking for, but if you provide a path to the parameter `save_model_directory` of `hpo_pipeline`, the final models of each trial will be saved in sub-directories (cf. https://pykeen.readthedocs.io/en/stable/reference/hpo.html#pykeen.hpo.hpo_pipeline).

Example:
```python
from pykeen.hpo import hpo_pipeline


pipeline_result = hpo_pipeline(
    dataset="nations",
    model="TransE",
    n_trials=2,
    save_model_directory="/tmp/models/",
)
```

EDIT: It seems like `save_model_directory` does not occur in the docstring :sweat_smile: #820 will fix that.

Hi @mberr ,

thanks a lot for your reply (and sorry for not getting back to it until now). 

> This might not be exactly what you are looking for, but if you provide a path to the parameter save_model_directory of hpo_pipeline, the final models of each trial will be saved in sub-directories

Thanks for pointing that out! That's good to know. But as far as I understand it does not solve checkpointing since checkpointing would mean that also unfinished trials could be resumed. 

Note that this has not been a big problem for me in practice. Just wanted to point out that the API seemed a bit inconsistent here. But no big deal really. 

I guess it could be a problem for some people with really big graphs where even individual trials take long enough that resuming on failure would be important. But not sure if in such a case the HPOpipeline would be the right tool in the first place...