Repository: pykeen/pykeen
Issue: Evaluating metrics on many subsets with multiple models
Root cause: Documentation Deficiency

For a project, I need to evaluate some metrics on many small subsets of the Wikidata5M test set. Specifically, I need evaluation metrics (such as Hits@K, arithmetic mean rank etc.) **for every predicate of the test set** to decide upon which model to use for prediction. To this end, I trained the ComplEx, DistMult, SimplE and TransE models with an embedding dimension of 32 and split the test set of Wikidata5M into predicate groups like this:

```python
def get_test_set_per_predicate(test_set_file):
    test_set = pd.read_csv(test_set_file, sep="\t", names=['S', 'P', 'O'], header=None)  # [subject, predicate, object]
    return test_set.groupby('P')
```

```
Result:
              S      P         O
0      Q7965079    P27       Q16
1      Q6719921    P31    Q11446
2      Q4925109   P175  Q5165801
3     Q11010724   P734    Q59853
4      Q1236794    P31  Q1134686
...         ...    ...       ...
5035   Q2199494  P1995    Q83042
5037   Q1453427  P1435   Q385405
5067  Q14709462   P149   Q176483
5069    Q776272  P2012   Q234138
5126  Q15507650  P2868   Q810198

[684 rows x 3 columns]
```

I'm loading the trained models like this:

```python
def get_trained_models():
    return {
        'ComplEx': {
            'model': torch.load('embeddings/ComplEx/trained_model.pkl'),
            'factory': TriplesFactory.from_path_binary('embeddings/ComplEx/training_triples')
        },
        'DistMult': {
            ...
        },
        'SimplE': {
            ...
        },
        'TransE': {
            ...
        }
    }
```

The test set has 211 unique predicates, so the resulting dataframe consists of 211 groups. For evaluation, I iterate over all testing subsets and all trained models and use [`RankBasedEvaluator.evaluate()`](https://pykeen.readthedocs.io/en/stable/api/pykeen.evaluation.Evaluator.html#pykeen.evaluation.Evaluator.evaluate) for computing the metrics per predicate per model. This is the simplified code:

```python
test_splits = get_test_set_per_predicate('knowledge_graph/wikidata5m_transductive_test.txt')
trained_models = get_trained_models()  # Dict of loaded model instances and triple factories

# Cycle through all predicate groups
for predicate, triples in test_splits:
    evaluator = RankBasedEvaluator()

    # Converting subset dataframe into triples factory
    test_factory = TriplesFactory.from_labeled_triples(triples=triples.values)

    # Cycle through all trained models
    metrics = pd.DataFrame()   # Dataframe for accumulating results
    for model_name, result in trained_models.items():
        model = result['model']
        training_factory = result['factory']   # This is the loaded factory of the training data

        model_metrics = evaluator.evaluate(
            model=model,
            mapped_triples=test_factory.mapped_triples,
            additional_filter_triples=[
                dataset.training.mapped_triples,     # dataset is the Wikidata5M instance of the library
                dataset.validation.mapped_triples
            ]
        ).to_dict()
        flattened_metrics = pd.concat([
            # Aggregating optimistic, realistic and pessimistic metrics into one dataframe
            # and adding new column for metric type
        ])

        flattened_metrics['P'] = predicate
        flattened_metrics['model'] = model_name
        metrics = pd.concat([metrics, flattened_metrics])
```

One evaluation run takes about 1:30 mins, which would sum up to about 21 hours for all 211 predicates and 4 models. This is very inefficient, and I'm asking whether there is a better way of evaluating so many subsets or whether it is possible to get the individual metrics from the evaluator. I tried to compute the metrics myself using [`predict_triples()`](https://pykeen.readthedocs.io/en/stable/api/pykeen.predict.predict_triples.html), but haven't managed to do it so far, due to the elaborate prediction post-processing, and I don't want nonsense results in the end.

Is there a more efficient way to compute this or am I missing something? Also, please let me know if there are errors in the code.

## Environment
| Key             | Value                    |
|-----------------|--------------------------|
| OS              | posix                    |
| Platform        | Linux                    |
| Release         | 6.5.0-14-generic         |
| Time            | Sat Jan 13 18:43:33 2024 |
| Python          | 3.10.13                  |
| PyKEEN          | 1.10.1                   |
| PyKEEN Hash     | UNHASHED                 |
| PyKEEN Branch   |                          |
| PyTorch         | 2.1.2                    |
| CUDA Available? | true                     |
| CUDA Version    | 12.1                     |
| cuDNN Version   | 8902                     |

I didn't see the discussions forum at first, so closing this and moving over to discussions.

For reference, this is the discussion:  #1360