Repository: pykeen/pykeen
Issue: Large difference in predictive performance between PyKEEN and DGL-KE
Root cause: Interface Contract Mismatch

Hi,

I have been doing a few experiments comparing the predictive performance of PyKEEN with other KGE frameworks - namely DGL-KE (although others will be added soon if I get time). 

I have thus far found a really rather large variation between the performance of a PyKEEN model versus DGL-KE. For example, I have trained a version of ComplEX using both frameworks on a pre-split version of Hetionet using as similar parameters as I was able to.

The PyKEEN model was specified as follows:

``` python

from pykeen.pipeline import pipeline

pipeline_result = pipeline(
    training='het_train.tsv',
    testing='het_test.tsv',
    validation='het_val.tsv',
    model='ComplEx',
    model_kwargs=dict(
        embedding_dim=400
    ),
    training_kwargs=dict(
        num_epochs=500,
        batch_size=2000
    ),
    optimizer_kwargs=dict(
        lr=0.02
    ),
    negative_sampler_kwargs=dict(
        num_negs_per_pos=200
    ),
    evaluation_kwargs=dict(
        batch_size=8,
    ),
    evaluator_kwargs=dict(filtered=True),
    use_testing_data=True,
)

```

Whereas, DGL-KE was run as follows (data files are identical just re-exported as CSV for DGL-KE):

`DGLBACKEND=pytorch dglke_train --model_name ComplEx --dataset hetionet --data_path 'Data/' --delimiter ',' --data_files het_train.csv het_val.csv het_test.csv --format raw_udd_hrt --max_step 450039 --test --valid --batch_size 2000 --neg_sample_size 200 --hidden_dim 400  --lr 0.02 --neg_sample_size_eval 200 --log_interval 100000  --batch_size_eval 8 --save_path complex/ --gpu 0 --no_eval_filter
`

A quick explanation for some of the parameters - DGL-KE uses max step rather than number of epochs so I set it as follows: (1800157  triples in training graph / 2000 BS) * 500 = 450039. I left the regularization parameters as default for both frameworks. Finally it seems possible I made a mistake by setting `filtered=True`, maybe it should have been set to false to compare with the `--no_eval_filter` I set on DGL-KE (if so I can rerun)? 

The results from this experiments are presented below for ComplEx:

| Metric      | DGL-KE      | PyKEEN        |
| :---        |    :----:   |          ---: |
| MRR         | 0.434       | 0.011         |
| MR          | 10.5        | 4398.3        |
| HITS@1      | 0.297       | 0.001         |
| HITS@3      | 0.491       | 0.006         |
| HITS@10     | 0.712       | 0.0238        |


Just for complete transparency, these numbers were taken from each frameworks respective evaluation protocols. For PyKEEN, I took the 'both' and 'avg' numbers.

Do you have any thoughts on the large disparity here? Would you expect your numbers to be comparable to other frameworks? Have I made some obvious mistake in my comparison setup? 

I will add more models as well as results from LibKGE over the next few days. 

-------

Forgot to add the code I used to create the dataset -

``` python
import numpy as np
import pandas as pd

np.random.seed(42)
df = pd.read_csv('Hetionet/edges.sif', sep='\t')
train, val, test = np.split(df.sample(frac=1), [int(.8*len(df)), int(.9*len(df))])
train.to_csv('het_train.tsv', index=False, sep='\t')
val.to_csv('het_val.tsv', index=False, sep='\t')
test.to_csv('het_test.tsv', index=False, sep='\t')
```

Hi @sbonner0 . 

I am not familiar with DGL-KE, however, https://github.com/awslabs/dgl-ke/issues/132 reads like they use a sampled evaluation instead of 1-n ranking. Please keep in mind that metrics using absolute ranks are not (directly) comparable  when the number of choices against which they are ranked is different (cf. e.g., https://arxiv.org/abs/2002.06914 ).

As for the filtered setting: `filtered=True` corresponds to the filtered evaluation setting, where other true heads (tails) ranked higher do not decrease the score. Thus, with filtered evaluation, you usually obtain better (absolute) ranks than in the unfiltered setting.

Hi @mberr, thanks so much for your reply - I'll dive into the paper you linked over the next few days. If you have any more reading suggestions covering the differences between the various evaluation strategies, I would love to hear them. Many thanks!

Hi @sbonner0, I am wondering what resolution you found to the differences in model performance? I'm seeing a similarly large difference between dgl-ke and pykeen when evaluating Hits@5 using my own evaluation method.