Repository: pykeen/pykeen
Issue: Understanding controlling the random seed using the hpo_pipeline
Root cause: Documentation Deficiency

Hi,

I am trying to understand how to control the setting of the random seed using the `hpo_pipeline` object and what exactly it is controlling.

As it stands, I have seen from the documentation that you can set the seed for the datasets (so that train/test/eval splits are consistent across experiments) by passing a dataset_kwargs to the hpo_pipeline object. Similar, it seems it is possible to control the random seed for the model (presumably controlling the parameter initialisation?) by passing a model_kwargs to the hpo_pipeline. 

However having set those parameters, I still get the following message when launching a hpo run: `WARNING:pykeen.pipeline:No random seed is specified. Setting to 3963730628.`

I have seen on the normal `pipeline` object there is a random_seed argument which seems to be not present on the `hpo_pipeline` object. What does this control for the pipeline and is there anyway to set it for the hpo_pipeline (if it is even important for reproducibility)? 

Not yet a solution (@cthoyt might know more), but a tool to investigate:

Within pykeen [`logging`](https://docs.python.org/3/library/logging.html) is used for logging. This module allows you to customize the format of the log message, including options to show which line the statement came from.

Maybe this code already works:

```python
FORMAT="%(asctime) %(filename)s:%(lineno) [%(levelname)s]  d%(message)s"
logging.basicConfig(format=FORMAT)
```

Otherwise you need to explicitly create a handler and formatter
```python
FORMAT="%(asctime) %(filename)s:%(lineno) [%(levelname)s]  d%(message)s"
logger = logging.getLogger("pykeen")
handler = logging.StreamHandler()
formatter = logging.Formatter(FORMAT)
handler.setFormatter(formatter)
logger.addHandler(handler)
```

Thanks @mberr, just to be clear. You are proposing I adjust the logging level to see where the seed message is emanating from? 

> Thanks @mberr, just to be clear. You are proposing I adjust the logging level to see where the seed message is emanating from?

It does not change the level, but rather include the file + linenumber in the formatting of the warning message of unknown origin you see as described here
> However having set those parameters, I still get the following message when launching a hpo run: `WARNING:pykeen.pipeline:No random seed is specified. Setting to 3963730628.`

Is this an minimum working example of the usage you have in mind?
```python
from pykeen.hpo import hpo_pipeline
res = hpo_pipeline(
    dataset="nations",
    model="distmult",
    n_trials=1,
)
```

Running this code with the formatting adjustments mentioned above shows up a single random seed warning
```
2020-12-18 13:25:13,503 pipeline.py:856 [WARNING] No random seed is specified. Setting to 3851534380.
```
originating from here
https://github.com/pykeen/pykeen/blob/b62f5ae33427f41bd8826645e24eb54a03b57281/src/pykeen/pipeline.py#L854-L856

So, if the `pipeline` function does not receive a `random_seed`, it will generate one. This random seed will then also be passed to the model, in case no explicit seed was provided:
https://github.com/pykeen/pykeen/blob/b62f5ae33427f41bd8826645e24eb54a03b57281/src/pykeen/pipeline.py#L899

It is *not* passed to the dataset. Pre-split datasets, like e.g. "nations", already have a deterministic split. For others, you might need to provide a seed for reproducible splits. I am linking @cthoyt here, how might know more about it.

Right now, I do not see a way to pass the `random_seed` parameter of `pipeline` via the `hpo_pipeline`. 
@mali-git / @cthoyt Do you know more about it?

There’s a goofy sentinel value that suppresses this message for the HPO that I think got messed up. I will have to take a look back into this, since the HPO and each pipeline need their own random number generators 

Great thanks for the replies both of you :) So if I explicitly set the model and dataset seed in the hpo_pipeline then I should be good for reproducibility? 

Hi @mberr @cthoyt ,

Just to follow up on this - I just wanted to clarify a few thing now I have a some runs completed.

Say I have explicitly set the dataset and model seed as in the code below - 

```
from pykeen.hpo import hpo_pipeline

hpo_pipeline_result = hpo_pipeline(
    dataset='Hetionet',
    dataset_kwargs=dict(
        random_state=42
    ),
    model='TransE',
    model_kwargs=dict(
        random_seed=42
    ),
)
```

Is it safe to assume that the dataset and model objects are having their seeds set correctly? And thus it is safe to ignore the lack of random seed for the pipeline? 

Interestingly, I can see in the Optuna logs that the seed changes once and then remains consent for the rest of the runs. For example, the first trail had a seed of 1748234673, then a value of 4289512057 for the rest of the runs.