Repository: pykeen/pykeen
Issue: Unable to get satisfactory results by hpo or other published configuration files
Root cause: Configuration Error

### What is your question

I really appreciate the Pykeen team for develop such a good platform, but I have been troubled by a problem for quite a few days:

Since there are so many hyper-parameters to be chosen from and they have a big impact on the results, when conducting a trail on a new model or a new dataset without a published configuration file like those in [https://github.com/pykeen/benchmarking](url), I often cannot get a satisfactory link prediction result.

For example, I use TransE for codex-s with num_epochs=400,  embedding_dim=512, and other parameters by default. I only get a hits@1=0.0626, hits@10=0.4557 (while the published results in the original paper are 0.219 and 0.634 respectively).
My code is as follows:
```python
result = pipeline(
        dataset='codexsmall',
        training_kwargs=dict(
            num_epochs=400,
        ),
        model='transe',
        model_kwargs=dict(
            embedding_dim=512,
        ),
        stopper='early',
        stopper_kwargs=dict(frequency=5, patience=5),
    )
    pprint.pprint(result.configuration)

    print(result.get_metric('hits@1'))
    print(result.get_metric('hits@10'))
    print(result.get_metric('mrr'))
```
And the output is like this:
![image](https://user-images.githubusercontent.com/44865171/148018890-2b750724-b83f-4a7a-99e0-dca3eed09dad.png)

I have tried a few hpo searches by the hpo_pipeline, but don't get a much better result (the hits@1 metric never exceeds 0.1).

As the authors of codex have provided their pretrained libkge models and config files in [https://github.com/tsafavi/codex](url), I load the pretrained embeddings to test on pykeen, and I get similar results as those published, which excludes the possibility of the difference between the evaluation metrics of libkge and pykeen.

The next attempt I made is to follow the published config file [https://github.com/tsafavi/codex/blob/master/models/link-prediction/codex-s/transe/config.yaml#L50](url) to reproduce it on pykeen. As the implementation of libkge and pykeen is not exactly the same, I have tried my best to keep the configurations consistent. My code is like this:
```python
result = pipeline(
        dataset='codexsmall', 
        training_kwargs=dict(
            num_epochs=400,
            batch_size=128,
        ),
        model='transe',
        model_kwargs=dict(
            scoring_fct_norm=2,
            embedding_dim=512,
        ),
        loss='CrossEntropyLoss',
        training_loop='lcwa',
        negative_sampler_kwargs=dict(num_negs_per_pos=2),
        stopper='early',
        stopper_kwargs=dict(frequency=5, patience=5),
        lr_scheduler='ReduceLROnPlateau',
        lr_scheduler_kwargs=dict(factor=0.95, mode='max', patience=6, threshold=0.0001, ),
        optimizer='Adagrad',
        optimizer_kwargs=dict(lr=0.04121772717931592),
    )

    pprint.pprint(result.configuration)

    print(result.get_metric('hits@1'))
    print(result.get_metric('hits@10'))
    print(result.get_metric('mrr'))
```
I also did a slight change to make the pytorch lr_scheduler _'ReduceLROnPlateau'_ work in pykeen, and in _transe.py_, I changed the ___init___ function as follows to make the initializer and regularizer kwargs exactly the same as those published.
```python
super().__init__(
    entity_representations=EmbeddingSpecification(
                embedding_dim=embedding_dim,
                initializer=torch.nn.init.xavier_normal_,
                initializer_kwargs=dict(gain=1.0, ),
                constrainer=torch.nn.functional.normalize,
                constrainer_kwargs=dict(p=-1.0, ),
                regularizer_kwargs=dict(weight=1.3175179879955912e-07),
            ),
    relation_representations=EmbeddingSpecification(
                embedding_dim=embedding_dim,
                initializer=torch.nn.init.xavier_normal_,
                initializer_kwargs=dict(gain=1.0, ),
                constrainer=torch.nn.functional.normalize,
                constrainer_kwargs=dict(p=-1.0, ),
                regularizer_kwargs=dict(weight=3.7237496170376805e-18),
            ),
            **kwargs,
 )
 self.scoring_fct_norm = scoring_fct_norm
```
The only difference left I think is the _num_negs_per_pos_, which can be set differently for head and tail in libkge. I have tried different nums for this, but larger num often comes with a _Loss is non-finite_ problem.

However, such a reproduction experiment returns an even worse result, where the hits@1 is 0.0 like this:

![image](https://user-images.githubusercontent.com/44865171/148021852-c4ffb478-5ff4-4994-9b52-4c6ac5c1220c.png)

So until now, I have tried the default configurations、the hpo searches and the official published libkge config file, but I never get a satisfactory result of TransE on codex-s in pykeen, which is indeed achieved by the pretrained model [https://github.com/tsafavi/codex](url).

**I wonder if I have overlooked some differences between libkge and pykeen in the above reproduction attempt that leads to such a big difference. Or do you have some suggestions on the hyper-parameter selection?**
I want to get a satisfactory result on a new model or a new dataset without a published pykeen config file, but I usually come with such a problem.
Looking forward to your reply. Thank you very much!




### Environment

| Key             | Value                       |
|-----------------|-----------------------------|
| OS              | posix                       |
| Platform        | Linux                       |
| Release         | 3.10.0-1160.11.1.el7.x86_64 |
| Time            | Tue Jan  4 14:16:17 2022    |
| Python          | 3.7.4                       |
| PyKEEN          | 1.6.1-dev                   |
| PyKEEN Hash     | UNHASHED                    |
| PyKEEN Branch   |                             |
| PyTorch         | 1.10.0+cu102                |
| CUDA Available? | true                        |
| CUDA Version    | 10.2                        |
| cuDNN Version   | 7605                        |

### Issue Template Checks

- [X] This is not a bug report (use a different issue template if it is)
- [X] This is not a feature request (use a different issue template if it is)
- [X] I've read the text explaining why including environment information is important and understand if I omit this information that my issue will be dismissed

Hi @xbc0112 ,

for
```
result = pipeline(
   ...,
   training_loop='lcwa',
   negative_sampler_kwargs=dict(num_negs_per_pos=2),
)
```
`training_loop` should be `slcwa` (=Stochastic LCWA, with negative sampling).

#704 adds the necessary CE support to SLCWA (not yet in `master`).

I am not an expert in reading `libkge` configurations, so I may miss some points. Some things I noticed which may not be part of the current configuration you run are:

* [ ] https://github.com/tsafavi/codex/blob/3132e426c2a6b643b70bad679905a3a6270be440/models/link-prediction/codex-s/transe/config.yaml#L49 - there seems to be a dropout on entity (and also relation) embeddings. I am not sure what a negative dropout values means though, and it is not mentioned in https://github.com/uma-pi1/kge/blob/master/kge/config-default.yaml
* [ ] https://github.com/tsafavi/codex/blob/3132e426c2a6b643b70bad679905a3a6270be440/models/link-prediction/codex-s/transe/config.yaml#L11 - there is an additional "import" of `"reciprocal_relations_model"` - this *may* be a hint that inverse relations are used. To enable this in `pykeen`, pass `dataset_kwargs=dict(create_inverse_triples=True)`.
* [ ] https://github.com/tsafavi/codex/blob/3132e426c2a6b643b70bad679905a3a6270be440/models/link-prediction/codex-s/transe/config.yaml#L37-L42 - as you already mentioned `ReduceLROnPlateau` is currently not available as a LR scheduler. Since you implemented it on your own, you may want to verify that you reduce on the right plateau; since the mode is set to `max` in the configuration, I *assume* that the quantity which is monitored is some validation or train *metric*, not the *loss*. https://github.com/uma-pi1/kge/blob/4ec7d3d570897386e3e331622697d8056ddfbd60/kge/config-default.yaml#L442-L457 looks like it could be `mean_reciprocal_rank_filtered_with_test`, which, I assume, refers to filtered MRR on validation.

Hi @mberr ,
Thank you very much for your patient analysis, I have checked the things you mentioned and conducted a few trails:

1. As the default training type is _KvsAll_ in libkge(https://github.com/uma-pi1/kge/blob/master/kge/config-default.yaml#L164), which is said to be lcwa in https://github.com/pykeen/pykeen/blob/master/src/pykeen/training/lcwa.py, I don't quite understand why here the training_loop should be slcwa. Still I have tried the newly update of slcwa for CE in issue # 704, but it brings worse results.
2. As the libkge developer said in the issue https://github.com/uma-pi1/kge/issues/132, negative dropout with train.auto_correct=True mean that the dropout is effectively 0, so it is omitted.
3. I have tried to use the inverse relations as you said, it helps to get better results, which is still far below the reported results yet.
4. Thank you for your last reminder, I indeed wrongly used the _loss_ in the implemention of _ReduceLROnPlateau_ yesterday. I then follow the libkge implemention in https://github.com/uma-pi1/kge/blob/master/kge/util/optimizer.py, add _metric='mrr'_ in the stopper_kwargs, and do the lr update according to mrr after every validation. It helps to improve the results for a little.

However, after I have checked these things and tried several combination strategies, I still don't get a satisfactory result. The highest result I can achieve now is hits@1=0.1364, hits@10=0.3392, MRR=0.204, higher than before but still lower than expected and the published. 

 Maybe to fully replicate the configurations of libkge in pykeen, I need to have a better understanding of the two platforms. But it comes to my original question here: as my ultimate purpose is to get a satisfactory result on a new model or a new dataset, I have no idea how to search from the numerous optional configurations, especially I often come to a result much lower than expected.  

Can you give me some suggestions for the hyper-parameter selection? I wonder how you did the hpo searches in the benchmarking paper, by conducting hundreds of trials for each model on each dataset? And for the whole pipeline, which components (loss、regularizer、optimizer、training_loop, etc.) do you think are more important? For example, should I first specify some key components and then do the hpo_pipeline to get better results? 
I'm quite confused about this problem, not solely on this model and this dataset.

Looking forward to your reply again. Thank you very much!