Repository: pykeen/pykeen
Issue: Reproducing Kazemi et al (2018) results for SimplE
Root cause: Algorithmic Error

### Describe the bug

I have been trying to reproduce the results from [kazemi2018](https://arxiv.org/pdf/1802.04868.pdf), without luck.
I have used both homemade pipelines and the one provided in [Experiments](https://github.com/pykeen/pykeen/blob/493ef68942a8482a993a12ca749ca5f4e7bbf07e/src/pykeen/experiments/simple/kazemi2018_simple_wn18.json).

### How to reproduce

run `$ pykeen experiments reproduce simple kazemi2018 wn18`

**Got:**
2022-03-07 07:00:05 INFO                                                 difference   p  significant
hits_at_k.both.optimistic.1                   **0.023000** NaN        False
hits_at_k.both.optimistic.10                  **0.031800** NaN        False
hits_at_k.both.optimistic.3                   **0.026100** NaN        False
inverse_harmonic_mean_rank.both.optimistic    **0.025466** NaN        False


**Which, according to the paper, should have been:**
      "mean_reciprocal_rank": **0.942**,
      "hits_at_k": {
        "1": **0.939**,
        "3": **0.944**,
        "10": **0.947**
        }


### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.4.0-97-generic         |
| Time            | Mon Mar  7 08:19:55 2022 |
| Python          | 3.7.11                   |
| PyKEEN          | 1.7.0                    |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.10.2                   |
| CUDA Available? | true                     |
| CUDA Version    | 11.3                     |
| cuDNN Version   | 8200                     |

### Additional information

As always, thank you for your great work on PyKEEN!

Hi @Koenkalle , this is in-line with our results - we could not reproduce the results either, cf. https://ar5iv.labs.arxiv.org/html/2006.13365#Ax1.T10 and https://ar5iv.labs.arxiv.org/html/2006.13365#S6.SS5.



Thank you for the links, I had missed those tables.

 I have been comparing my results with one of the other SimplE implementations linked in the documentation. (https://github.com/baharefatemi/SimplE).

That implementation is able to reproduce the results, but I have been unable to find impactful differences compared to the PyKEEN implementation.


We also noticed that SimplE's result where sometimes unstable, likely due to the score clamping: when the score is outside of the clamping interval, it will not receive any gradients, and thus can only improve indirectly when the entities' embeddings are updated by a different triple.

@mberr something in the resent changes seems to have affected the results from this experiment.

The provided experiment now produced this odd output:
```
2022-03-17 12:35:16 INFO                                                 difference   p  significant
both.optimistic.hits_at_1                    -0.255800 NaN        False
both.optimistic.hits_at_10                   -0.009200 NaN        False
both.optimistic.hits_at_3                    -0.032100 NaN        False
both.optimistic.inverse_harmonic_mean_rank   -0.142809 NaN        False
```
But using my own pipeline, based on https://github.com/baharefatemi/SimplE:
```
pipeline_result = pipeline(
                dataset="wn18",
                model="simple",
                model_kwargs=dict(
                    embedding_dim=200,
                    entity_initializer=PretrainedInitializer(tensor=xavier_uniform_(zeros([40943,200])))
                ),
                evaluator=RankBasedEvaluator,
                evaluator_kwargs=dict(filtered=True),
                evaluation_fallback=True,
                result_tracker=PythonResultTracker(),
                loss=SoftplusLoss,
                training_kwargs=dict(batch_size=1415),
                optimizer=Adagrad,
                optimizer_kwargs=dict(lr=0.1),
                epochs=50,
                regularizer=LpRegularizer,
                regularizer_kwargs=dict(normalize=True,p=2.0, weight=0.03),
                negative_sampler='basic',
                negative_sampler_kwargs=dict(num_negs_per_pos=1, filtered=False, filterer='python-set'),
        )
```
I now get:
```
    model  total time  hits@1  hits@3  hits@10  MRR 
0  simple   60.781645  0.5846  0.9053    0.933  0.74  
```           

Which is a lot better than my earlier observed results.

The odd output seems okay. The second column is the *difference* between obtained and expected result, e.g., for MRR, you obtained 0.74, which is ~0.14 below the published result. After looking into you comment I re-read your initial text again, and noticed that there you got positive differences, i.e., values above the reported results.

Thank you for the clarification - I should have realized that last time I was looking at this...

Based on this, I do still notice a change in behavior based on different pykeen versions:

Using 1.7.0: the experiment can reproduce the published results, but my own pipeline's result is close to 0.
Using the latest commit: the experiment and my pipeline get matching results, but they are below the published results.

Does any change come to mind that could be the cause of this? Do not waste your time looking into it if not, I can do that myself :)
 

> Does any change come to mind that could be the cause of this? Do not waste your time looking into it if not, I can do that myself :)

Not really :confused: Most changes since 1.7 should be visible from https://github.com/pykeen/pykeen/milestone/6?closed=1 although I am not sure if that includes 100% of them. The [diff](https://github.com/pykeen/pykeen/compare/v1.7.0...master) is quite big :sweat_smile: the most relevant changes should be moving to data loaders, refactoring the evaluation code, and maybe something we changed when adding support for the `mode` parameter to enable inductive training.

#849 might be related