Repository: pykeen/pykeen
Issue: Split evaluation_entity_whitelist into head_entity and tail_entity
Root cause: Unimplemented Feature Gap

Hi!

I was just looking at your[ Drug Repurposing Tutorial](https://pykeen.readthedocs.io/en/latest/tutorial/understanding_evaluation.html#example) and wondered about the results I was getting. Then it occurred to me that even though it has the relation type fixed to "CtD" (Compound treats Disease) it still doesn't only score on compounds and diseases. It also allows to score triples like Compound0 - CtD - Gene0, which is 1) wasteful and 2) making it hard to interpret the performance metrics. Even when setting the evaluation_entity_whitelist to only compounds and diseases it would still allow triples like Compound0 - CtD - Compound1 to be scored, which is also nonsense here. What would actually be needed to have a fair setting is an evaluation_head_entity_whitelist and an evaluation_tail_entity_whitelist. In this case, one would be the compounds and the other the diseases, so that we only score triples that start in a compound and end in a disease via a CtD edge.

**Describe the solution you'd like**
an additional evaluation_head_entity_whitelist  and evaluation_tail_entity_whitelist parameter for the pipeline and the hpo_pipeline

**Describe alternatives you've considered**
Exporting the model for every evaluation, extracting the embeddings, filtering for triples that make sense and evaluate them outside of pykeen. Obviously this produces a huge overhead and doesn't inform the optimizer of the performance on the actual task. In the context of the HPO pipeline I haven't figured out yet how to get to the best performing model and save it.

Alternatively one could loop over all valid head entitites with the model.predict_tails() method and filter the results for the desired tail entity whitelist. This would also take a while and I am not sure if it is currently possible in the case of a HPO pipeline.


> Then it occurred to me that even though it has the relation type fixed to "CtD" (Compound treats Disease) it still doesn't only score on compounds and diseases. It also allows to score triples like Compound0 - CtD - Gene0, which is 1) wasteful and 2) making it hard to interpret the performance metrics.

The evaluation protocol is as follows: Assume we have evaluation triples `T = {(h_i, r_i, t_i)}_{i=1}^{n}`.Then the evaluation will consider all triples `(h_i, r_i, e)` and `(e, r_i, t_i)` for all `e in E`, i.e. all considered triples share at least two elements (head and relation, or relation and tail) with an existing, known-to-be-true triple, from the evaluation set. Thus, no triple `(Compound0, CtD, Gene0)` will be evaluated, as long as there is no true triple `(Compound0, CtD, *)` or `(*, CtD, Gene0)` in the set of true triples.

Hence, the evaluation in itself already considers a "soft typing", where either the head or the tail is guaranteed to be of the correct "type", since it comes from a true evaluation triple.

To better understand the use case:

You want to score all triples of the form `(Compound_i, CtD, Disease_j)`, where `Compound_i` comes from a provided list of compounds, and `Disease_j` from a provided list of diseases, correct?

If this is the use case, #107 will introduce a mechanism enabling to evaluate such queries by calling `scores = ERModel(h_indices=compound_indices, r_indices=CtD_index, r_indices=diseases_indices)`

Thanks for looking into my request!

> You want to score all triples of the form (Compound_i, CtD, Disease_j), where Compound_i comes from a provided list of compounds, and Disease_j from a provided list of diseases, correct?

Yes, thats entirely correct. I can further clarify that the set of compounds and the set of diseases is all compounds and all diseases, possibly provided by a pre-compiled list of node identifiers.

> Thus, no triple (Compound0, CtD, Gene0) will be evaluated, as long as there is no true triple (Compound0, CtD, *) or (*, CtD, Gene0) in the set of true triples.

Does that mean that if there is no true triple (CompoundX, CtD, *) in the set of true triples, meaning that there is no known Disease that is treated by Compound X, that in this case CompoundX is excluded from the scoring? And that the nodes that are eligible to be put in the place of the * must have an incident CtD edge? That means it includes all compounds and diseases that have an incident CtD edge, and only those?

> If this is the use case, #107 will introduce a mechanism enabling to evaluate such queries by calling scores = ERModel(h_indices=compound_indices, r_indices=CtD_index, r_indices=diseases_indices)

That is nice to hear. I assume that compound_indices, diseases_indices will be integer indices, will there be a way to map the node identifier that are used in the edgelist input to said integer indices? I guess that is what the TripleFactory does, right?

As always, thanks for helping me out! :)

> Does that mean that if there is no true triple (CompoundX, CtD, *) in the set of true triples, meaning that there is no known Disease that is treated by Compound X, that in this case CompoundX is excluded from the scoring? And that the nodes that are eligible to be put in the place of the * must have an incident CtD edge? That means it includes all compounds and diseases that have an incident CtD edge, and only those?

Partially :smile: 

If we do not have any triple true `(CompoundX, CtD, *)` in the evaluation set, then the right side /tail link prediction will not be taken into account. One reason for that is that we do not know any true triple, thus we cannot compute the rank of the true entity. However, it does not mean that the entity `CompoundX` will never take part in any evaluation, since for a triple `(*, CtD DiseaseY)`, `CompoundX` is included in the scoring.

If you are unsatisfied with not evaluating these triples, you might take a look at the task of [_triple classification_](https://paperswithcode.com/task/triple-classification) which is related to link prediction. Here, we (independently) predict a score for each evaluation _triple_ and consider the problem as a binary classification problem, i.e. a classification whether this triple holds. Notice however, that we need _true_ negative triples for evaluation, i.e. triples from which we _know_ that they are false. In the link prediction setting, and consequently in link prediction datasets, we do not have such triples, and generally follow the open world assumption, i.e. if a triple is not in the set of triples, we do not assume that it is false, but only unknown.

Currently, the evaluation in pykeen is primarily focussed on the task of link prediction, and thus the evaluation methods evaluate this task. The model however are generally capable of helping you: You can "manually" compute the scores for all evaluation triples, and then use e.g. the classification evaluation from sklearn. Please be aware that depending on the used loss function, the models' scores may not be calibrated: E.g. for margin based losses we do not enforce that scores of existing edges exceed a certain threshold, but only compare them against unknown triples _with shared head-relation / relation-tail part_. A loss suggestion would be e.g. `BCELossWithLogits`.

> That is nice to hear. I assume that compound_indices, diseases_indices will be integer indices, will there be a way to map the node identifier that are used in the edgelist input to said integer indices? I guess that is what the TripleFactory does, right?

You can directly provide a 1-D tensor (i.e. a list) of indices, e.g.,

```python
compound_indices = torch.as_tensor([1, 2, 3, 4])
r_indices = torch.as_tensor([1])
diseases_indices = torch.as_tensor([5, 6, 7])
scores = model(h_indices=compound_indices, r_indices=r_indices, r_indices=diseases_indices)
```

The result has then shape `(1, 4, 1, 3)` (batch size, number of different heads/relations/tails).

Thanks for the detailed reply! However, I am still confused. 

> Thus, no triple (Compound0, CtD, Gene0) will be evaluated, as long as there is no true triple (Compound0, CtD, *) or (*, CtD, Gene0) in the set of true triples. 

I tried that with model.predict_tails(Compound0, CtD) and it scored a lot of tail nodes that are not connected to a CtD edge and that can not be connected to a CtD edge (like genes). Is that different from what would happen in the evaluation process?

>  scores = model(h_indices=compound_indices, r_indices=r_indices, r_indices=diseases_indices)

I will try that and see if it does what i need.

BTW: I include inverted edges, so the right side and the left side predictions should be symmetrical, right?


> > scores = model(h_indices=compound_indices, r_indices=r_indices, r_indices=diseases_indices)
> 
> I will try that and see if it does what i need.

Just a word of caution: The code is not yet in `master` so the API might still change.

> > Thus, no triple (Compound0, CtD, Gene0) will be evaluated, as long as there is no true triple (Compound0, CtD, _) or (_, CtD, Gene0) in the set of true triples.
> 
> I tried that with model.predict_tails(Compound0, CtD) and it scored a lot of tail nodes that are not connected to a CtD edge and that can not be connected to a CtD edge (like genes). Is that different from what would happen in the evaluation process?

`predict_tails` is not aware of the restriction, and thus evaluates scores for all tails.

Notice that this is also done in the current implementation of the evaluation with entity restriction, as seen in

https://github.com/pykeen/pykeen/blob/9191dbbbdecfc2f550671e6dbc6b7ef87a9f9515/src/pykeen/evaluation/evaluator.py#L478-L484

Moreover, we summarize the procedure [here](https://pykeen.readthedocs.io/en/latest/tutorial/understanding_evaluation.html#entity-and-relation-restriction) (on the same page you can also find information about the general, non-restricted evaluation).

In any case, having a evaluation considering only triples between two sets of entities as you suggested seems like a valuable addition. From my point of view, I would postpone this feature until #107 is merged, since it makes it easier to implement.