Repository: pykeen/pykeen
Issue: Is it possible to resume an Optuna study via the HPO pipline?
Root cause: Documentation Deficiency

Hey,

I know that Optuna allows one to resume a study once it has been completed, I was wondering if this is supported in pykeen?

I have seen that an hpo_result object can be saved to directory - is it possible to reload a saved study and continue to run it? The use case would be if a long hpo run was killed, a user could resume it without having to repeat all the already completed studies.

Optuna explicitly supports this - by default, PyKEEN uses a SQLite database backend, but you can configure it to use other SQL databases or even NoSQL databases just by passing in different kwargs to optuna. You can use the same study name or study ID to look it back up in the database and continue on with new “trials” (this is the optuna terminology) which correspond 1-1 to pykeen pipeline runs. 

However, if what you’re looking for is that you can resume an unfinished trial, then we don’t have an implementation for this and I’m pretty hesitant to create one because of how unintuitive it would be to configure and use 

After the weekend I will try and provide a code example for you! Hopefully the terminology above is enough to get you started. If you haven’t already, check the tutorial on using the HPO pipeline in the docs because it might include something on this 

Hi @cthoyt, thanks so much for your reponse. So I checked the docs and couldn't see an example of how the HPO results are stored in the SQL database. It would be great to see some code examples of how to resume a saved hpo_pipeline either from SQL or from the files created by saving the hpo pipline. 

I was thinking if it would be possible to resume an unfinished trial. This is something I encounter in my workflow as I submit to a HPC system using Slurm, so my jobs are often killed before finishing. I guess one way around this would be to set a time limit rather than a trail limit and keep resuming until the desired trail number has been reached.

It's quite a tricky feature - on the reference page for HPO (https://pykeen.readthedocs.io/en/latest/reference/hpo.html) there are several arguments that don't have explicit documentation. At the bottom there's note that the remaining parameters are passed to optuna.study.create_study() or optuna.study.Study.optimize(). Check https://optuna.readthedocs.io/en/latest/reference/generated/optuna.study.create_study.html#optuna.study.create_study for all the details, but you can use the `storage` parameter to tell Optuna where to store the database. You can use any connection string to a database, including in memory (`:memory:`), SQLite, and most RDBMS's following the the tutorial/rules here: https://docs.sqlalchemy.org/en/13/core/engines.html.

W.r.t. restarting a failed trial within the HPO framework - I don't think this is something we'll be able to tackle within the PyKEEN project. This seems like more of a feature that should be part of Optuna. I totally feel your pain about a job getting cancelled though (I've used slurm before). 

One thing we can do is improve the ability to use checkpoints inside HPO. This feature is new as of last week since #123, and we explicitly punted dealing with HPO until later. At least if this is available, you could manually start up the jobs that got killed to finish. However, it wouldn't contribute back to the database of trials, which is useful for Bayesian optimization techniques like Tree-Parzen Estimator.

It just occur to me that I think you can load a study and add a specific hyper-parameter combination to the queue. If that's done for a combination that caused a job to die, it would also be able to look up the checkpoint if it existed. However, I'm not familiar with doing that. Maybe it's something you can figure out yourself while we're working on the other parts. All of the guts you would need are in the `pykeen.hpo.hpo` module to roll your own `hpo_pipeline` that allows for explicit queuing of an  experiment (just don't use all of the `suggest_*()` functions)

Hi @cthoyt thanks so much for your detailed reply! I'll dig into the links you have sent and see how things work. 

But am I correct then in understanding that calling `save_to_directory` to save the hpo pipeline that had successfully reached a trail number of 100 say to disk can not then be reloaded to continue adding more trails using the history information needed for approaches like Tree-Parzen Estimator etc?

I am going to close this as with the help of the team, I was able to get the functionality I was expecting to work well. 

The trick is to set the `storage` and `load_if_exists` arguments in the `hpo_pipline` object. Then if an experiment was killed part way through, upon restarting the code, the existing completed trails will be loaded and the HPO search will continue from that point. For example: 

``` python
hpo_pipeline_result = hpo_pipeline(
    storage='sqlite:///foo.db',
    load_if_exists=True,
    model=model,
    dataset=dataset,
)
```