Repository: pykeen/pykeen
Issue: Why are evaluation results (Hits@K) different for the early stopper and the pipeline?
Root cause: Documentation Deficiency

Hello,

What is the difference between the hits@k from the `metrics` key and the `stopper` key in the results.json? I trained a graph with RotatE with early stopping. Here are excerpts of the results.json:

`"hits_at_k": {
      "both": {
        "optimistic": {
          "1": 0.1972839849351336,
          "3": 0.38009238699937736,
          "5": 0.44285079883793066,
          "10": 0.5284036914120543
        }`



`"stopper": {
    "best_epoch": 50,
    "best_metric": 0.1259676848898215,
    "frequency": 10,
    "larger_is_better": true,
    "metric": "hits_at_k",
    "patience": 2,
    "relative_delta": 0.01,
    "results": [
      0.11086157430791674,
      0.11909007393445901,
      0.12311223054262672,
      0.12464448067907156,
      0.1259676848898215,
      0.12600101924013501,
      0.12662363710531283
    ],
    "stopped": true
  }`

Interestingly, when I trained my model without early stopping for the exact same number of epochs provided by the previous stopper (50 epochs), the hits@10 is not the same as with early stopping.

`"hits_at_k": {
      "both": {
        "optimistic": {
          "1": 0.016774523064545473,
          "3": 0.04513866524741433,
          "5": 0.07038463320349886,
          "10": 0.12535354185946915
        }`

Thank you.

I'm also confused about it，The metric I use is hits@10, but the results of `stopper` and `metrics` in result.json are completely different. Which one should I use as the result of the evaluation?

Hi @chauvu and @Moon-xm ,

by default, the evaluation results of the `pipeline` contain the results for *test* triples:
https://github.com/pykeen/pykeen/blob/f0a167039711c1a4eab7ca89f6829200868a646e/src/pykeen/pipeline/api.py#L701

The early stopper, in contrast, evaluates on the *validation* triples. Hence, the two metrics are computed on two different sets of triples.

Hi @mberr,

That would explain why the hits@k is different between the stopper and the pipeline evaluation. But when I tried training the same dataset with and without early stopping, the hits@k are different:

With early stopping:
`"hits_at_k": { "both": { "optimistic": { "1": 0.1972839849351336, "3": 0.38009238699937736, "5": 0.44285079883793066, "10": 0.5284036914120543 }`

Without early stopping:
`"hits_at_k": { "both": { "optimistic": { "1": 0.016774523064545473, "3": 0.04513866524741433, "5": 0.07038463320349886, "10": 0.12535354185946915 }`

The early stopper stopped at 50 epochs, so I trained without early stopping for exactly 50 epochs. Is there a reason for this?
Thank you.

I'd suggest running a few times and looking at the distribution of hits@k values for a given k. There are a few things that introduce non-detereminism, including the training/testing/validation split (if you're using the pipeline to do that automatically), the random initialization, and negative sampling. Even if you set a random seed, you'll still see some different results between early stopping and no early stopping since more random numbers are generated in the mean time