Repository: pykeen/pykeen
Issue: Unable to reproduce RGCN results on benchmark FB15k237
Root cause: Configuration Error

Hi, thanks for the continued good work on such a great library!

I recently faced some confusion relating to the RGCN implementation on PyKEEN v1.5.0.

## Issue

> I tried to reproduce the Link Prediction training on FB15k237. But the resultant evaluation scores were orders of a magnitude off from the reported benchmark results, and I am not sure if the error lies in my choice of configuration, or some place else.

I checked the [PyKEEN benchmarking repo](https://github.com/pykeen/benchmarking) to see if there was a RGCN-specific branch documenting PyKEEN benchmark runs on the same, but was unable to find it.

I was wondering if you had tried the benchmarking for RGCN, and/or if you have any insight into the following.

## What I tried:

Used default hyperparameters as mentioned in the original RGCN paper ([Schlichtkrull et al 2017](https://arxiv.org/abs/1703.06103v4)), and also double checked with the default settings provided within PyKEEN
 (Reference PR #548, and issue #544)
My pipeline object:

```python
dataset = FB15k237()
pipeline_result = pipeline(
    dataset=dataset,
    model='RGCN',
    loss='BCEWithLogitsLoss',
    model_kwargs= dict(
        embedding_dim=500,
        interaction='DistMult',
        num_layers= 2, 
        decomposition=None,
        activation=None
     ),
     training_kwargs=dict(
         num_epochs=50,
         sampler='schlichtkrull',
         # batch_size=1000,
     ),
     optimizer_kwargs=dict(
         lr=0.01,
     ),
     evaluator_kwargs=dict(
         filtered=True,
     ),
)
```
* Ran training and evaluation routines on both CPU and GPU

## Results

When evaluating using a `RankBasedEvaluator`, tried filtered setting with both (1) training triples (2) training + validation triples as filtered instances. In both cases the MRR, and hits@k were much lower than the results reported in ([Schlichtkrull et al 2017](https://arxiv.org/abs/1703.06103v4))
In the setting with training and validation triples provided as additional filtering instances, the following were the results.
 _Note: presenting only averaged 'both' results below, and not individual head/tail metrics:_
  
```python
from pykeen.evaluation import RankBasedEvaluator

evaluator = RankBasedEvaluator()
results_all_rels = evaluator.evaluate(
    pipeline_result.model,
    dataset.testing.mapped_triples,
    additional_filter_triples=[
        dataset.training.mapped_triples,
        dataset.validation.mapped_triples,
    ],
 )
```

  * MRR 
  `results_all_rels.inverse_harmonic_mean_rank`

    > {'both': {'realistic': 0.0018078132498358154,
      'optimistic': 0.0068752076631400675}}

  * hits@k 
  `results_all_rels.hits_at_k`

    > {'both': {  'realistic': {1: 0.0,
                         3: 4.892846658185733e-05,
                         5: 0.00022017809961835796,
                         10: 0.0003914277326548586},
                        'optimistic': {1: 9.785693316371466e-05,
                         3: 0.00014678539974557198,
                         5: 0.0003180350327820726,
                         10: 0.0016146393972012918}}
                         }

     * for reference, original scores reported on FB15k237 in [Schlichtkrull et al 2017](https://arxiv.org/abs/1703.06103v4) (Table 5)
     > MRR (filtered): 0.248
     > hits@1    | 0.153
         hits@3   | 0.258
         hits@10  | 0.414

Hi @pmitra01 ,

RGCN has not been included in the benchmarking study and has not yet been verified thoroughly.

While I am relatively confident that the message passing implementation should be correct, the configuration and also the training approach might differ from the official code, cf. https://github.com/tkipf/relational-gcn

We collected the experimental configuration for FB15k and WN18RR at https://github.com/pykeen/pykeen/tree/master/src/pykeen/experiments/rgcn, but I think we did not yet verify that these indeed reproduce the numbers reported in the paper. In particular, I just now noticed that, e.g., gradient clipping from the original code ( https://github.com/MichSchli/RelationPrediction/blob/c77b094fe5c17685ed138dae9ae49b304e0d8d89/settings/gcn_basis.exp#L25 ) is not contained there.


From where did you take the configuration information for FB15k-237? It would also be great, if you could contribute the experimental setting as a experiment JSON as those linked above.


Hi @mberr, thanks for the clarification and pointers. I'll try out pytorch geometric RGCN implementation for now. On comparison, if I stumble upon any fundamental differences with PyKEEN, I'll bring it forth in a fresh issue/contrib.

As for the experiment configuration information for FB15k-237 - I mostly pieced it together from the paper [Schlichtkrull et al 2017](https://arxiv.org/abs/1703.06103v4), and author's code, so it is not indisputable. One parameter that I can already see that I got wrong was the block dimension for the block decomposition step. The authors report a 5x5  block diagonal values (so 25), while I had left mine unchanged at PyKEEN default of 100. 

Closing this issue for now, as my original question stands answered. Once I have verified the experiment config with a few experiments, I'll contribute the experiment JSON back to https://github.com/pykeen/pykeen/tree/master/src/pykeen/experiments/rgcn.

@pmitra01 thanks!