Repository: pykeen/pykeen
Issue: Replicating experiment results 
Root cause: Documentation Deficiency

I would like to recreate the results listed under experiments, for example, through this link, [this ](https://github.com/pykeen/pykeen/blob/master/src/pykeen/experiments/convkb/nguyen2018_transe_fb15k237.json) is a TransE model would like to run on Fb15k-237. 
 
In the experiments folder, some models have the results there as well and some don't. How can I get the expected results for the configuration of TransE that I linked to above? (I got MRR .16 here which is lower than all results reported by other papers) 

Also, In the benchmarking paper "Bringing Light Into the Dark:A Large-scale Evaluation of Knowledge GraphEmbedding Models under a Unified Framework", there are some good results with the same model but no mention of the hyperparameters, are these provided somewhere?

Thanks in advance! 

Hi @PhaelIshall,

> In the experiments folder, some models have the results there as well and some don't. How can I get the expected results for the configuration of TransE that I linked to above? (I got MRR 1.6 here which is lower than all results reported by other papers)
> 

The results provided in the configuration files are the reported results from the original papers and not necessarily those which we obtained  (due to different reasons described in our paper ["Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge GraphEmbedding Models under a Unified Framework"](https://arxiv.org/abs/2006.13365)).
The configurations and results of our reproducibility and benchmarking studies reported in "Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge GraphEmbedding Models under a Unified Framework" are available at https://github.com/pykeen/benchmarking.

Please note that the configuration you linked above corresponds to the pre-training of the embeddings for ConvKB on FB15K-237 (and therefore no results are reported).

> Also, In the benchmarking paper "Bringing Light Into the Dark:A Large-scale Evaluation of Knowledge GraphEmbedding Models under a Unified Framework", there are some good results with the same model but no mention of the hyperparameters, are these provided somewhere?

The results of the reproducibility study can be found at https://github.com/pykeen/benchmarking/tree/master/reproducibility/results, and the results of the benchmarking study can be found at https://github.com/pykeen/benchmarking/tree/master/ablation/results .


@mali-git Thanks for answering. I have another question. In some config files there are some parameters like `entity_initializer` and `relation_initializer`. When I specify these in the pipeline under `model_kwargs`, such as giving `uniform` as a value, I get an error saying these are not recognizable. 

I also want to specify some details in the regularization such as entity embedding weight and relation embedding weight but I don't see this in the documentation or the source code, there is only one weight parameter. Any chance these can be changed? 

> @mali-git Thanks for answering. I have another question. In some config files there are some parameters like `entity_initializer` and `relation_initializer`. When I specify these in the pipeline under `model_kwargs`, such as giving `uniform` as a value, I get an error saying these are not recognizable.
> 
> I also want to specify some details in the regularization such as entity embedding weight and relation embedding weight but I don't see this in the documentation or the source code, there is only one weight parameter. Any chance these can be changed?

This is going to require a bit of a different kind of answer. Could you open a new issue so we can discuss there?