Repository: pykeen/pykeen
Issue: Details on how to reproduce experiments 
Root cause: Documentation Deficiency

Can you perhaps help me realize how I can use your framework properly? I tried following the documentation on how to train and evaluate embedding models in pykeen but after the OpenBioLink Dataset caused CUDA oom issues (partly because for most models slicing is not implemented or can I adapt any parameters to fit it?) (sidenote: how are the OBLF1 and F2 supposed to me used?) and I did not get the Neural Network implementations I was particularly interested in using to run I now tested some data set model combination that was actually used in your experiments and still the results seem far off anything you report (I am aware I did not run the same hyper parameter search -because different to your way where 24 hours get you through a good of parameters, for a training time of 107170 sec that was not feasible for a simple try out for me) the results are around hits@3 : 0.0005 and hits@10: 0.0007 for the attatched configuration. I have the 1.0.1-dev version installed and use Geforce GTX1080 2560 Cude-Kerne 11GB GDDR5 AERO GPUs. 

```python
from pykeen.pipeline import pipeline

pipeline_result = pipeline(
    dataset='WN18RR',
    model='RotatE',
    model_kwargs=dict(
        embedding_dim=500,
        #automatic_memory_optimization=True,
        ),
    training_kwargs=dict(num_epochs=1000),
    optimizer='Adam',
    optimizer_kwargs=dict(
        lr=0.00008,
        weight_decay=0.91,
    ),
    training_loop='sLCWA',
    loss='MarginRankingLoss',
    evaluator='RankBasedEvaluator',
    evaluator_kwargs=dict(filtered=True,),
    negative_sampler="basic",
    negative_sampler_kwargs=dict(num_negs_per_pos=1024)
)
print(pipeline_result)
```


Hi @LFGMUW,

> I tried following the documentation on how to train and evaluate embedding models in pykeen but after the OpenBioLink Dataset caused CUDA oom issues (partly because for most models slicing is not implemented or can I adapt any parameters to fit it?

Please consider that OpenBioLink is a comparably large dataset. To avoid CUDA out of memory exceptions, you can reduce the embedding dimension of your model and make use of the [automatic memory optimization](https://pykeen.readthedocs.io/en/latest/tutorial/performance.html#automated-memory-optimization), i.e. `pipeline(model_kwargs=dict(automatic_memory_optimization=True, ...), ...)`. Please note that if your model uses batch normalization, the `automatic_memory_optimization` cannot be used since it relies on [sub-batching](https://pykeen.readthedocs.io/en/latest/tutorial/performance.html#sub-batching) whereas batch normalization requires the full batch when applied.

> sidenote: how are the OBLF1 and F2 supposed to me used?

Openbiolinkf1 and Openbiolinkf2 are subsets of OpenBioLink that we created, which we will upload soon.

> from pykeen.pipeline import pipeline
> pipeline_result = pipeline(
> dataset='WN18RR',
> model='RotatE',
> model_kwargs=dict(
> embedding_dim=500,
> #automatic_memory_optimization=True,
> )

We performed a reproducibility study and integrated the corresponding configurations. The configuration for RotatE on WN18RR can be found at https://github.com/pykeen/pykeen/blob/master/src/pykeen/experiments/rotate/sun2019_rotate_wn18rr.json
The best RotatE-WN18RR configuration that we found is available at https://github.com/pykeen/benchmarking/blob/master/ablation/results/rotate/wn18rr/random/adam/2020-04-25-19-04_217bcf38-2101-461b-9593-b133b4201e6a/0000_wn18rr_rotate/best_pipeline/pipeline_config.json

Please let us know if you need further help :)

Hi @mali-git,
Thank you for replying to my questions.

> Please consider that OpenBioLink is a comparably large dataset. To avoid CUDA out of memory exceptions, you can reduce the embedding dimension of your model and make use of the automatic memory optimization

Yes,  but I thought since it is built-in there would perhaps be a way of training, I did use using the memory optimization parameter as well and ran it with dimension 50, making it much smaller does not seem promising for the learned embeddings.

> Please note that if your model uses batch normalization, the automatic_memory_optimization cannot be used since it relies on subbatching whereas batch normalization requires the full batch when applied.

Yes, thank  you.  I was hoping there was a way around the issue of memory constrains seeing that you published results to the models.
I did see the experiment configs, they where the basis of my attempt of recreating. I was surprised how many orders my metric results differed. But thank you.