Repository: pykeen/pykeen
Issue: Evaluate and Measure MRR, Hit@K, etc. against Specific Entities
Root cause: Documentation Deficiency

### What is your question

Hi,

Is it possible to set the pipeline to evaluate a model against a specific entity type(s) instead of all?
For example, in a biomedical knowledge graph, we want to limit the object/tail just to be the chemical compound entity type and not all the entities in the KG.

Thanks,

### Environment

| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 5.4.0-99-generic         |
| Time            | Tue Apr  5 23:58:26 2022 |
| Python          | 3.9.7                    |
| PyKEEN          | 1.6.1-dev                |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 1.10.1                   |
| CUDA Available? | true                     |
| CUDA Version    | 10.2                     |
| cuDNN Version   | 7605                     |

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Hi @ahmadpgh ,

there is a way to restrict evaluation to a certain subset of entities / relations - please take a look at 
* https://pykeen.readthedocs.io/en/stable/tutorial/understanding_evaluation.html#entity-and-relation-restriction 
* https://pykeen.readthedocs.io/en/stable/api/pykeen.evaluation.evaluate.html#pykeen.evaluation.evaluate

If I understand you correctly, you are asking for only restricting the *tail* prediction to a certain entity type, while looking at arbitrary head entities, or a different restriction thereupon. This is currently not supported out-of-the-box. A hacky solution could be to derive a subclass from `RankBasedEvaluator` and override the `process_scores` method.



I wanted to do something similar where after training the model I wanted to see how well it performed for both each given relationships as well as based on given node types, using the exact same logic as  https://github.com/pykeen/pykeen/discussions/670 I did something like this.

```python
import pandas

from pykeen.datasets import get_dataset
from pykeen.evaluation import RankBasedEvaluator
from pykeen.evaluation.evaluator import evaluate
from pykeen.pipeline import pipeline

dataset = get_dataset(dataset="nations")
result = pipeline(
    dataset=dataset,
    model="distmult",
)
entity_class_dict={'brazil': 'Americas',
                 'burma': 'Asia',
                 'china': 'Asia',
                 'cuba': 'Americas',
                 'egypt': 'MiddleEast',
                 'india':'Asia',
                 'indonesia': 'Asia',
                 'israel': 'MiddleEast',
                 'jordan': 'MiddleEast',
                 'netherlands': 'Europe',
                 'poland': 'Europe',
                 'uk': 'Europe',
                 'usa': 'Americas',
                 'ussr': 'Europe',}

evaluate_on_test = False
tf = dataset.testing if evaluate_on_test else dataset.validation
additional_filter_triples = [dataset.training.mapped_triples]
if evaluate_on_test:
    # bordes style filtering
    additional_filter_triples.append(dataset.validation.mapped_triples)
evaluator = RankBasedEvaluator()
data = []
keys = None
for entity_class in set(entity_class_dict.values()):
    logging.info(f'Evaluating for {entity_class}')
    allowed_entitys = [k for k, v in entity_class_dict.items() if v == entity_class]
    allowed_ids = [k for k, v in tf.entity_id_to_label.items() if v in allowed_entitys]
    mapped_triples = []
    for triple in tf.mapped_triples:
        if int(triple[0]) in allowed_ids or int(triple[2]) in allowed_ids:
            mapped_triples.append([int(triple[0]), int(triple[1]), int(triple[2])])
    mapped_triples = torch.LongTensor(mapped_triples)
    tf_restricted = tf.clone_and_exchange_triples(mapped_triples=mapped_triples)
    if tf_restricted.num_triples > 0:
        partial_result = evaluate(
            model=result.model,
            mapped_triples=tf_restricted.mapped_triples,
            additional_filter_triples=additional_filter_triples,
            evaluator=evaluator,mode=None
        ).to_flat_dict()
        if keys is None:
            keys = sorted(partial_result.keys())
        data.append([entity_class] + [partial_result[key] for key in keys])
df = pandas.DataFrame(data=data, columns=["entity_class"] + keys)
print(df)
```