Repository: pykeen/pykeen
Issue: Filtered MRR/hits/ranks miss validation triplets
Root cause: Validation Gap

Recently, I wanted to reproduce results from your paper, when I noticed different values of MRR and hits@k for some models I've been playing for some time (with different implementations). I found a possible issue in the codebase. In your paper _Bringing Light Into the Dark_ you cite Bordes et al ([16] https://hal.archives-ouvertes.fr/docs/00/92/07/77/PDF/bordes13nips.pdf ), which says 

> To avoid such a misleading behavior, we propose to remove from the list of corrupted triplets all the triplets that appear either in the **training, validation or test** set (except the test triplet of interest).

When I examined your code (debugging: by loading already learned model) the size of _all_pos_triples_ was actually the size of train set plus test set size, i.e.
```
pykeen.evaluation.evaluator.py::evaluate
if filtering_necessary or positive_masks_required:
        all_pos_triples = torch.cat([model.triples_factory.mapped_triples, mapped_triples], dim=0)
```

In fact, the size of the _all_pos_triples_ should be (according to the definition above) the sum of train, test, and validation set sizes. Sure, I've run that evaluation with _filtered=True_ but got no clue how to filter validation triples out. 

Test both-avg MRR of the learned model is 0.664, however, after adding validation data into _all_pos_triples_ it turns to 0.802, which is the same value I get in my own implementation of MRR evaluation (different codebase; hits@ks change as well). Also, the latter score is the one obtained by evaluating the learned model's output in other codebases (RotatE, AnyBURL, GNTP).

**To Reproduce**
pykeen                        1.4.0
Learn a model using HPO on Kinships dataset with ComplEx as a model, e.g. 
```
{
  "type": "hpo",
  "metadata": {},
  "pipeline": {
    "stopper": "early",
    "stopper_kwargs": {
      "frequency": 50,
      "patience": 2,
      "relative_delta": 0.002
    },
    "dataset": "kinships",
    "dataset_kwargs": {
      "create_inverse_triples": false
    },
    "model": "complex",
    "model_kwargs": {
      "entity_initializer": "xavier_uniform",
      "relation_initializer": "xavier_uniform"
    },
    "model_kwargs_ranges": {
      "embedding_dim": {
        "type": "int",
        "low": 6,
        "high": 8,
        "scale": "power_two"
      }
    },
    "loss": "bceaftersigmoid",
    "regularizer": "NoRegularizer",
    "optimizer": "Adam",
    "training_loop": "lcwa",
    "training_kwargs": {
      "num_epochs": 1000
    },
    "training_kwargs_ranges": {
      "batch_size": {
        "type": "int",
        "low": 7,
        "high": 9,
        "scale": "power_two"
      },
      "label_smoothing": {
        "type": "float",
        "low": 0.001,
        "high": 1.0,
        "scale": "log"
      }
    },
    "evaluator": "rankbased",
    "evaluator_kwargs": {
      "filtered": true
    },
    "evaluation_kwargs": {}
  },
  "optuna": {
    "n_trials": 100,
    "timeout": 85200,
    "metric": "hits@10",
    "direction": "maximize",
    "study_name": "studyA",
    "storage": "sqlite:///optuna_results.db",
    "save_model_directory": "./"
  }
}
```
load the best model (you need to check the paths)
```
base = os.path.join(".","0000_kinships_complex","best_pipeline","replicates","replicate-00000", "trained_model.pkl")
model = torch.load(modelPath)
```
load dataset and evaluate
```
from pykeen.datasets import Kinships
dataset = Kinships(create_inverse_triples=False)
evaluator = RankBasedEvaluator(filtered=True)
print(evaluator.evaluate(model=model, mapped_triples=dataset.testing.mapped_triples))
```
which produces
```
RankBasedMetricResults(mean_rank={'tail': {'avg': 2.355679702048417, 'best': 2.355679702048417, 'worst': 2.355679702048417}, 'head': {'avg': 3.8221601489757915, 'best': 3.8221601489757915, 'worst': 3.8221601489757915}, 'both': {'avg': 3.0889199255121045, 'best': 3.0889199255121045, 'worst': 3.0889199255121045}}, mean_reciprocal_rank={'tail': {'avg': 0.7330930902072598, 'best': 0.7330930902072598, 'worst': 0.7330930902072598}, 'head': {'avg': 0.5953884055378353, 'best': 0.5953884055378353, 'worst': 0.5953884055378353}, 'both': {'avg': 0.6642407478725475, 'best': 0.6642407478725475, 'worst': 0.6642407478725475}}, hits_at_k={'tail': {'avg': {1: 0.5623836126629422, 3: 0.8994413407821229, 5: 0.9599627560521415, 10: 0.978584729981378}, 'best': {1: 0.5623836126629422, 3: 0.8994413407821229, 5: 0.9599627560521415, 10: 0.978584729981378}, 'worst': {1: 0.5623836126629422, 3: 0.8994413407821229, 5: 0.9599627560521415, 10: 0.978584729981378}}, 'head': {'avg': {1: 0.40782122905027934, 3: 0.7402234636871509, 5: 0.8575418994413407, 10: 0.9320297951582868}, 'best': {1: 0.40782122905027934, 3: 0.7402234636871509, 5: 0.8575418994413407, 10: 0.9320297951582868}, 'worst': {1: 0.40782122905027934, 3: 0.7402234636871509, 5: 0.8575418994413407, 10: 0.9320297951582868}}, 'both': {'avg': {1: 0.4851024208566108, 3: 0.8198324022346368, 5: 0.9087523277467412, 10: 0.9553072625698324}, 'best': {1: 0.4851024208566108, 3: 0.8198324022346368, 5: 0.9087523277467412, 10: 0.9553072625698324}, 'worst': {1: 0.4851024208566108, 3: 0.8198324022346368, 5: 0.9087523277467412, 10: 0.9553072625698324}}}, adjusted_mean_rank={'tail': 0.048502915568330654, 'head': 0.07895308601179159, 'both': 0.06372800079006112})
```
although the correct result should be more like this 
```
RankBasedMetricResults(mean_rank={'head': {'avg': 3.2672253258845436, 'best': 3.2672253258845436, 'worst': 3.2672253258845436}, 'both': {'avg': 2.5991620111731844, 'best': 2.5991620111731844, 'worst': 2.5991620111731844}, 'tail': {'avg': 1.9310986964618249, 'best': 1.9310986964618249, 'worst': 1.9310986964618249}}, mean_reciprocal_rank={'head': {'avg': 0.7330220846256758, 'best': 0.7330220846256758, 'worst': 0.7330220846256758}, 'both': {'avg': 0.8026087092232348, 'best': 0.8026087092232348, 'worst': 0.8026087092232348}, 'tail': {'avg': 0.8721953338207936, 'best': 0.8721953338207936, 'worst': 0.8721953338207936}}, hits_at_k={'head': {'avg': {1: 0.6163873370577281, 3: 0.8156424581005587, 5: 0.8798882681564246, 10: 0.9357541899441341}, 'best': {1: 0.6163873370577281, 3: 0.8156424581005587, 5: 0.8798882681564246, 10: 0.9357541899441341}, 'worst': {1: 0.6163873370577281, 3: 0.8156424581005587, 5: 0.8798882681564246, 10: 0.9357541899441341}}, 'both': {'avg': {1: 0.7067039106145251, 3: 0.8812849162011173, 5: 0.9236499068901304, 10: 0.957169459962756}, 'best': {1: 0.7067039106145251, 3: 0.8812849162011173, 5: 0.9236499068901304, 10: 0.957169459962756}, 'worst': {1: 0.7067039106145251, 3: 0.8812849162011173, 5: 0.9236499068901304, 10: 0.957169459962756}}, 'tail': {'avg': {1: 0.7970204841713222, 3: 0.946927374301676, 5: 0.9674115456238361, 10: 0.978584729981378}, 'best': {1: 0.7970204841713222, 3: 0.946927374301676, 5: 0.9674115456238361, 10: 0.978584729981378}, 'worst': {1: 0.7970204841713222, 3: 0.946927374301676, 5: 0.9674115456238361, 10: 0.978584729981378}}}, adjusted_mean_rank={'head': 0.06770638557189115, 'both': 0.05383340649487142, 'tail': 0.03996042741785169})
```

I haven't tried to examine the issue without the loading step, i.e. a _learn & evaluate_ scenario, however, it might behave the same way if _model.triples_factory.mapped_triples_ contains only train triples.

Hope this helps :)

best, Martin

Hi Martin (@martinsvat),

Thanks a lot for pointing this out! :)
We are currently working on a solution ([#398](https://github.com/pykeen/pykeen/pull/398)) that enables users to provide additional triples to the evaluator that should be filtered out during the filtered evaluation. In this context, one would provide the validation triples in addition.



Hi Mehdi (@mali-git), 

great work!

Was this (validation) filtering missing in the version used for _Bringing light into the dark..._ paper, i.e. version 0.1.2-dev (https://github.com/pykeen/benchmarking)? If this is the case, then it would be good to make a note somewhere (errata for the paper,...) that the values are not in fact filtered metrics. Once, I was struggling to reproduce the results of a paper claiming high MRRs; later, one of the authors recalled those were tail-MRR values only (instead of both-MRR).


best

> Hi Mehdi (@mali-git),
> 
> great work!
> 
> Was this (validation) filtering missing in the version used for _Bringing light into the dark..._ paper, i.e. version 0.1.2-dev (https://github.com/pykeen/benchmarking)? If this is the case, then it would be good to make a note somewhere (errata for the paper,...) that the values are not in fact filtered metrics. Once, I was struggling to reproduce the results of a paper claiming high MRRs; later, one of the authors recalled those were tail-MRR values only (instead of both-MRR).
> 
> best

Hi Martin,

Yes, this was also the case for the version used for the paper.
Therefore, we will re-evaluate the models and update the paper accordingly.

I think we've addressed the bug pointed out by the original issue with #398. Throughout the process, we made the interface a bit more general. This makes the HPO and pipeline work in the desired way with the same configuration that you posted.

I don't think it's how you're using PyKEEN, but in the case that you want to roll your own training pipeline, these generalizations can allow you to get a few different variations on the Bordes et al (2013) approach to inclusion of triples for the filtered setting. 

We updated the docs to better reflect the fixed behavior in HPO, normal pipeline usage, and HPO/pipeline usage with early stopping at https://pykeen.readthedocs.io/en/latest/tutorial/understanding_evaluation.html#filtering), but the actual.


@martinsvat @mali-git please let me know if we can close this one. I think if we want to continue discussing the benchmarking paper's results, we could open up a new issue.

@cthoyt Great work! Consider this issue closed :)

Regarding the benchmarking: 👍 for reevaluating thousands of GPU computational hours. That's the correct way of experiments reproducibility (which I've not seen in the community yet).