Repository: pykeen/pykeen
Issue: Clarification on the negative sets constructed for evaluation
Root cause: Documentation Deficiency

I wanted to verify how the evaluation is being performed in pykeen.

Is the complete set of possible corrupted negative triples (both head and tail) used when assessing the rank of a given true triple in the test set? Or is some sampling being used? Is this affected by the type of training being performed (LCWA vs. sLCWA)? 

I wonder if the (already excellent) evaluation documentation could be updated to make this explicitly clear? It might help clear up some issues when users are comparing predictive results across frameworks...

Hi @sbonner0 ,

Good question and thanks for the suggestion! For now you'll get the quick answer. :)
By default PyKEEN always uses the global set of entities for both head and tails, i.e. _(h r)_ is scored with all possible tails and _(r t)_ is scored with all possible heads to assess the rank of a given true triple in the test set. The type of training is not related to the evaluation, so it does not affect the evaluation.

In general we don't use sampling, since it will lead to volatile metrics that are not deterministic, which makes them hard to interpret. However, the `evaluate()` method of the `Evaluator` class allows to pass the argument `restrict_entities_to`, which allows to specify a subset of entities that can be used to perform the ranking.

@sbonner0 

To extend on @lvermue 's answer (slightly repeating it :sweat_smile: )

During evaluation, we evaluate head and tail prediction as two separate ranking tasks, and the evaluation does not differ for different training assumptions. For simplicity, I restrict the following answer to tail prediction only, i.e.  `(h, r, ?)`, but head prediction is done analogously.

In general, *all* entities are ranked as possible tails. There are a few cases in which some entities are masked:

1. In the `filtered` evaluation, all entities `t'` for which a triple `(h, r, t')` exists in the training *or* evaluation triples are excluded. Thereby, the model is not "punished" when it predicts larger scores for other true entities. The setting was proposed in [Bordes et al., 2013](https://hal.archives-ouvertes.fr/file/index/docid/920777/filename/bordes13nips.pdf), and since then is the setting usually reported in knowledge graph link prediction literature. [The documentation](https://pykeen.readthedocs.io/en/latest/tutorial/understanding_evaluation.html#filtering) says that this is enabled by default, but it seems like this is not the case in the code. :sweat_smile:  I open an issue to fix it later on. The filtered setting can be enabled by passing `filtered=True` as part of the `evaluator_kwargs` (e.g. `evaluator_kwargs=dict(filtered=True)`) to the `pipeline`, or when explicitly using an `Evaluator` object by setting the corresponding.

2. You can manually restrict the set of entities against which is scored. This is done by passing a list of entities via `evaluation_entity_whitelist` to the pipeline (or as explained above to the evaluator instance). In that case, only those entities are scored against. An example with some additional explanation is given [here in the documentation](https://pykeen.readthedocs.io/en/latest/tutorial/understanding_evaluation.html#example).

When reducing the set of entities against which is scored, the task generally gets easier, and almost all ranking metrics improve automatically. A notable exception is the adjusted mean rank, which is normalized by the number of entities and thus remains (relatively) stable. If you are interested, you can read more [here](https://arxiv.org/abs/2002.06914), also showing the effect of reduced candidate sets on some rank-based metrics.

Remark: The ["sidedness"](https://pykeen.readthedocs.io/en/latest/tutorial/understanding_evaluation.html#ranking-sidedness) (head/tail/both) of the *aggregated* rank metrics does not affect the individual rankings: They are still done separately for head/tail, but the *aggregation* now considers either all head/tail ranks or both sets of ranks


> but it seems like this is not the case in the code. sweat_smile I open an issue to fix it later on.

sorry for the confusion. For the *rank-based* evaluator `filtered=True` is already the default, as also written in the documentation.

Hi @lvermue @mberr ,

Thank you so much to you both for your detailed replies. The evaluation process is now clear to me - it's great to see such thought and consideration being placed on this crucial aspect of the KGE field.

Also great to see that the filtered setting is enabled by default. 

I have reread the 'Understand the Evaluation' page and see that everything you have said here is indeed covered there. I guess the only suggestion might be to explicitly state that the training type does not have an impact upon the evaluation process. 

I'd like to comment on the eval procedure on large graphs - I'd say we need to think about sampling anyways, for instance

* `OGBWikiKG` consists of 2.5M entities (it's in the list of available datasets, but apparently nobody tried to spin up a model on it yet, otherwise we would have probably seen an issue 😅 ), so LCWA scoring would require an enormous `[bs, 2.5M]` tensor and in the sLCWA case with `score_hrt` it would generate eval batches of `[bs*2.5M, 3]` for _each_ eval triple ;
* `OpenBioLink` has 200K-400K entities
* `CKG` - 7.6M nodes

Let's admit it: current eval procedures of pykeen won't work on datasets larger than 100K entities, so I believe restricted negative sampling will be needed sooner than later. To be more precise: currently, supplying the `evaluation_entity_whitelist` only the amount of triples in the eval_factory changes, but ranking still happens against all entities
https://github.com/pykeen/pykeen/blob/0f1fa3ae7f5622445dc6896c63d919a23cdb0d2d/src/pykeen/models/base.py#L624-L625




> I'd like to comment on the eval procedure on large graphs - I'd say we need to think about sampling anyways, for instance
> 
>     * `OGBWikiKG` consists of 2.5M entities (it's in the list of available datasets, but apparently nobody tried to spin up a model on it yet, otherwise we would have probably seen an issue sweat_smile ), so LCWA scoring would require an enormous `[bs, 2.5M]` tensor and in the sLCWA case with `score_hrt` it would generate eval batches of `[bs*2.5M, 3]` for _each_ eval triple ;
> 
>     * `OpenBioLink` has 200K-400K entities
> 
>     * `CKG` - 7.6M nodes
> 
> 
> Let's admit it: current eval procedures of pykeen won't work on datasets larger than 100K entities, so I believe restricted negative sampling will be needed sooner than later. To be more precise: currently, supplying the `evaluation_entity_whitelist` only the amount of triples in the eval_factory changes, but ranking still happens against all entities
> 
> https://github.com/pykeen/pykeen/blob/0f1fa3ae7f5622445dc6896c63d919a23cdb0d2d/src/pykeen/models/base.py#L624-L625

The score tensor itself does not require significant amounts of memory. For your example, `[bs, 2.5M]` requires `bs * 2,500,000 * 4` bytes, i.e. 10 MiB * batch_size. With any reasonable batch size, this does not pose a serious OOM threat.

I agree that for *computing* these scores, depending on the employed model, we need more intermediate memory. afaik, this is partially solved by using slicing, where the score tensor is further divided into slices which are computed individually. @lvermue may be able to say more about slicing :slightly_smiling_face: 


Besides that, I do not see any harm in having an option to use sampled evaluation. One has only to be aware that most metrics are not directly comparable when the number of samples varies.