Repository: pykeen/pykeen
Issue: Understanding differences in RotatE performance between version 1.7.0 and 1.8.1
Root cause: Algorithmic Error

### Describe the bug

When I tried to use new version pykeen=1.8.1 to train and get result, same dataset, same parameters, but the result are really lower than 1.7.0 or 1.6.0. Such as in 1.6.0 the result are hit@10=0.30, but at 1.8.0 the hit@10 are reaching 10-5 or even lower.

### How to reproduce

Train a model in same dataset and parameters. Train at 1.8.1 and 1.6.0, the result are really different. I just tried RotatE, other model are not tested.

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 4.15.0-163-generic       |
| Time            | Mon Apr 25 12:33:52 2022 |
| Python          | 3.9.12                   |
| PyKEEN          | 1.8.1                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.11.0+cu113             |
| CUDA Available? | true                     |
| CUDA Version    | 11.3                     |
| cuDNN Version   | 8200                     |

### Additional information

_No response_

@mberr can you remember which things might have changed evaluation? I doubt this is a RotatE - specific.

Might also want to see https://github.com/pykeen/pykeen/pull/877

@lovefxyo , would you mind sharing a (small) code example where you encounter the behaviour?

Of course. Just a simple pipeline and a few parameters. 
```python
import pandas as pd
from pykeen.triples import TriplesFactory
from pykeen.pipeline import pipeline
import torch
from matplotlib import pyplot as plt
from pykeen.models import predict
import pprint

path='./graph_dataset/dblp_1980/'
file_name="RotatE_dblp"

tf = TriplesFactory.from_path(path+"dblp_e.tsv",create_inverse_triples=True)
training, testing, validation = tf.split([.8, .1, .1],random_state=1234)

print("relation_num:",tf.num_relations)
print("entities_num:",tf.num_entities)

result = pipeline(
    device="cuda:0",
    training=training,
    testing=testing,
    validation=validation,
    model='rotate',
    negative_sampler='basic',
    model_kwargs=dict(
       embedding_dim=256
    ),
    training_kwargs=dict(
        num_epochs=40,
        batch_size=256,
    ),
    random_seed=1234,
     negative_sampler_kwargs=dict(
    num_negs_per_pos=30
    )
)

result.save_to_directory("./new_result/test/"+file_name)
result.metric_results.to_df().to_csv("./new_result/test/"+file_name+"/result.csv")
result.plot_losses()
plt.savefig("./new_result/test/"+file_name+"/loss.png", dpi=300)
fo = open("./new_result/test/"+file_name+"/parameter.txt", "w")
fo.write(str(result.configuration))
fo.close()
```


I tried to reduce the script further and use one of the built-in datasets:

```python
import pathlib

from pykeen.datasets import get_dataset
from pykeen.pipeline import pipeline
from pykeen.version import get_version

dataset = get_dataset(dataset="fb15k237")
result = pipeline(
    dataset="fb15k237",
    model="rotate",
    negative_sampler="basic",
    model_kwargs=dict(embedding_dim=256),
    training_kwargs=dict(
        num_epochs=40,
        batch_size=256,
    ),
    random_seed=1234,
    negative_sampler_kwargs=dict(num_negs_per_pos=30),
)
print(result.get_metric("hits_at_10"))
result.save_to_directory(pathlib.Path("results", get_version(with_git_hash=True)))
```

Here are the results for 1.7.0 and 1.8.1.: [results.zip](https://github.com/pykeen/pykeen/files/8561306/results.zip) They differ indeed by quite a large margin, e.g.,

| metric | 1.7.0 | 1.8.1 |
| -- | -- | -- |
| `inverse_harmonic_mean_rank.both.realistic` | 0.2244552597701484 | 0.016628838566273833 |
| `hits_at_10.both.realistic` | 0.41836285350817104 | 0.028623152950386536 |

I suspect, this comes from an incorrect initialization, since we changed the internal shape of the stored embedding to be compliant with https://pytorch.org/docs/stable/complex_numbers.html#transition-from-the-old-representation, switching from `tuple(shape[:-1]) + (2 * shape[-1],)` to `tuple(shape[:-1]) + (shape[-1], 2)`.


What can I do to solve this question? Thank you for spending so much time helping me.

The issue will hopefully be resolved by #888 . If you already want to give it a try, you can already install the version on this branch
```console
pip install https://github.com/pykeen/pykeen/archive/fix-complex-initialization.zip
```
and re-try your code to see whether this resolves your issue.

Too large initialization (as mistakenly was done prior to the fix in #888) appears to be quite harmful. Here, I took your configuration, switched to use vanilla torch `uniform_` initializer and varied the symmetric range logarithmically from `[-2^0, 2^0]` to `[-2^(-9), 2^(-9)]`. The "correct" value from the `xavier_normal` adaption used by RotatE would be at `2^(-4)`. Too small values do not really seem to hurt.
![plot](https://user-images.githubusercontent.com/36984984/165354608-3bdf479a-db18-49ad-b5b0-162753e12bdc.svg)

Interestingly, I could also anecdotically observe a similar behaviour with ComplEx, where the default initialization uses a large value range, too.

<summary>
Code to reproduce
</summary>
<details>

#### training
```python
import pathlib
from pykeen.datasets import get_dataset
from pykeen.pipeline import pipeline
from pykeen.models import ERModel
import torch
import click


@click.command()
@click.option("-if", "--inverse-factor", type=int, default=16)
def main(inverse_factor: int):
    embedding_dim = 256
    bound = 1 / inverse_factor
    dataset = get_dataset(dataset="fb15k237")
    result = pipeline(
        dataset=dataset,
        model=ERModel,
        negative_sampler="basic",
        model_kwargs=dict(
            interaction="rotate",
            entity_representations_kwargs=dict(
                shape=embedding_dim,
                initializer="uniform_",
                initializer_kwargs=dict(
                    a=-bound,
                    b=bound,
                ),
                dtype=torch.cfloat,
            ),
            relation_representations_kwargs=dict(
                shape=embedding_dim,
                initializer="init_phases",
                constrainer="complex_normalize",
                dtype=torch.cfloat,
            ),
        ),
        training_kwargs=dict(
            num_epochs=40,
            batch_size=256,
            callbacks="evaluation",
            callback_kwargs=dict(
                evaluation_triples=dataset.validation.mapped_triples,
                additional_filter_triples=[dataset.training.mapped_triples],
                prefix="validation",
            ),
        ),
        result_tracker="wandb",
        result_tracker_kwargs=dict(
            # TODO: update to yours
        ),
        random_seed=1234,
        negative_sampler_kwargs=dict(num_negs_per_pos=30),
    )
    result.save_to_directory(pathlib.Path(__file__).parent.joinpath(f"results-{inverse_factor}"))


if __name__ == "__main__":
    main()
```

#### visualization
```python
import math
import pathlib
import tempfile
import pandas
import seaborn
import wandb

path = pathlib.Path(tempfile.gettempdir(), "pykeen-rotate.tsv")
if not path.is_file():
    api = wandb.Api()

    # TODO: Project is specified by <entity/project-name>
    runs = api.runs("")
    metrics = [
        f"validation.both.realistic.{key}"
        for key in (
            "inverse_harmonic_mean_rank",
            "hits_at_1",
            "hits_at_3",
            "hits_at_10",
            "adjusted_geometric_mean_rank_index",
        )
    ] + ["loss", "_step"]
    data = []
    for run in runs:
        # get the bound
        bound = run.config.get("model_kwargs.entity_representations_kwargs.initializer_kwargs.b", None)
        if not bound:
            print("warning")
            continue
        # get some metrics in long format
        for entry in run.scan_history(keys=metrics):
            step = entry.pop("_step")
            for key, value in entry.items():
                data.append((run.id, bound, step, key, value))
    df = pandas.DataFrame(data=data, columns=["id", "bound", "epoch", "key", "value"])
    df.to_csv(path, sep="\t", index=False)
else:
    df = pandas.read_csv(path, sep="\t")
df["log_bound"] = df["bound"].apply(math.log2)
df["key"] = df["key"].str.replace("validation.both.realistic.", "")
grid = seaborn.relplot(
    data=df,
    x="epoch",
    y="value",
    hue="log_bound",
    legend="full",
    col="key",
    col_wrap=3,
    height=3,
    kind="line",
    palette="husl",
    facet_kws=dict(sharey=False),
)
grid.savefig("./plot.pdf")
grid.savefig("./plot.svg")
```

</details>

@lovefxyo this PR is now merged to `master`. Until the next release, you still need to install from Github (cf. https://github.com/pykeen/pykeen#installation--) if you already want to use the fixed version.