Repository: pykeen/pykeen
Issue: HPO about how to determine range?
GitHub labels: question

🍉 HPO

I read the #116, it seems that perfect. But one thing I am not clear that when I set the range, for example, I want to set embedding dim are in the range of [64, 128, 256, 512], two questions:

1) If I want to declare some certain numbers not range, do I use type='category'? (don't know type values, for now it seems that have 'int', 'float')

2) If I use the para= dict(type='int',
                    low=6,
                    high=8,
                    scale='power_two',)
how to express scale = 'multiply two(*2)' ? And what kind of choices of scale that I could choose(for now, I notice 'log', 'power_two')

1. Almost - use `type='categorical'` and then make a list of values in `choices=[...]`. You can use bool, int, str, or float, since these are supported by Optuna
2. Excellent question, we do have this functionality but it's not documented. I will also add it in #461 but the short answer is there are a few scales built in already.

## Scales

### Linear Scale

By default, you don't need to specify a scale, but you can be explicit by setting `scale='linear'`. I think this behavior is pretty obvious - there is no rescaling and you get back uniform distribution within the bounds specified by the `low` and `high` arguments. This applies to both `type=int` and `type=float` but not the others. The following example uniformly choose from [1,100]:

```python
from pykeen.hpo import hpo_pipeline

result = hpo_pipeline(
    dataset='Nations',
    model='TransE',
    training_loop='sLCWA',
    negative_sampler_kwargs_ranges=dict(
        num_negs_per_positive=dict(type=int, low=1, high=100),
    )
)
```

### Power Scale

We originally implemented this one so you could set `scale='power_two'`. This was useful for ConvE's `output_channels` parameter. However, we realized that using two as a base was a bit limiting, so we also implemented `scale='power'` where you could set `base`. Here's an example to optimize over the number of negatives per positive ratio using `base=10`:

```python
from pykeen.hpo import hpo_pipeline

result = hpo_pipeline(
    dataset='Nations',
    model='TransE',
    training_loop='sLCWA',
    negative_sampler_kwargs_ranges=dict(
        num_negs_per_positive=dict(type=int, scale='power', base=10, low=0, high=2),
    )
)
```

The power scale can only be used with `type=int` and not bool, str, or float. I like this scale because it can quickly discretize a large search space. In this example, you will get `[10**0, 10**1, 10**2]` as choices then uniformly choose from them.

## Modifying the Distribution

### Logarithmic Reweighting

The evil twin to the power scale is logarithmic reweighting on the linear scale. Rather than changing the choices themselves, the log scale uses Optuna's built in `log` functionality to reassign the probabilities uniformly over the log'd distribution. The same example as above could be accomplished with:

```python
from pykeen.hpo import hpo_pipeline

result = hpo_pipeline(
    dataset='Nations',
    model='TransE',
    training_loop='sLCWA',
    negative_sampler_kwargs_ranges=dict(
        num_negs_per_positive=dict(type=int, low=1, high=100, log=True),
    )
)
```

but this time, it's not discretized. However, you're just as likely to pick from [1,10] as [10, 100].

### Stepping

With the linear you can specify the `step` size. This discretizes the distribution in linear space, so if you want to pick from 10, 20, ... 100, you can do:

```python
from pykeen.hpo import hpo_pipeline

result = hpo_pipeline(
    dataset='Nations',
    model='TransE',
    training_loop='sLCWA',
    negative_sampler_kwargs_ranges=dict(
        num_negs_per_positive=dict(type=int, low=10, high=100, step=10),
    )
)
```

This actually also works with logarithmic reweighting, since it is still technically on a linear scale, but with probabilites reweighted logarithmically. So now you'd pick from one of [10] or [20, 30, 40, ...] with the same probability

```python
from pykeen.hpo import hpo_pipeline

result = hpo_pipeline(
    dataset='Nations',
    model='TransE',
    training_loop='sLCWA',
    negative_sampler_kwargs_ranges=dict(
        num_negs_per_positive=dict(type=int, low=10, high=100, step=10, log=True),
    )
)
```

Hi @lzw950905 same as the other issue, I made some big updates in #461 that took some of this explanation and reworked it into the docs. Please check them out on the "latest" branch on Read The Docs at https://pykeen.readthedocs.io/en/latest/tutorial/running_hpo.html. If there's still anything that's unclear, please let me know here or feel free to open a new issue :)