Repository: pykeen/pykeen
Issue: Unable to reproduce past landmark experiments using curated settings
Root cause: Configuration Error

**Describe the bug**
I tried to run several experiments with 
`pykeen experiments reproduce [model] [paper] [dataset]`
but quite a few of them produce weak results.

**To Reproduce**
For example, I tried to reproduce the `conve` model on the dataset fb15k237 with

`pykeen experiments reproduce conve dettmers2018 fb15k237`

**Expected behavior**
Here is my result (compared with the output from benchmark, and the paper):

|                                | mr (arithmetic_mean_rank) | mrr (inverse_harmonic_mean_rank) | Hits@1 | Hits@3 | Hits@10 |
|--------------------------------|---------------------------|----------------------------------|--------|--------|---------|
| Paper                          | 244                       | 0.325                            | 0.237  | 0.356  | 0.501   |
| pykeen/benchmark (replicate 0) | 244.7                     | 0.2966                           | 0.211  | 0.322  | 0.473   |
| user (me)                      | 6042                      | 0.011                            | 0.006  | 0.010  | 0.013   |


**Environment:**

| Key             | Value                      |
|-----------------|----------------------------|
| OS              | posix                      |
| Platform        | Linux                      |
| Release         | 3.10.0-1160.6.1.el7.x86_64 |
| Time            | Tue Nov 23 01:58:37 2021   |
| Python          | 3.7.11                     |
| PyKEEN          | 1.6.0                      |
| PyKEEN Hash     | UNHASHED                   |
| PyKEEN Branch   |                            |
| PyTorch         | 1.10.0+cu102               |
| CUDA Available? | true                       |
| CUDA Version    | 10.2                       |
| cuDNN Version   | 7605                       |


**Additional information**

I can also provide the results of other models. The results are sometimes better and sometimes worse, but I was not able to consistently reproduce the results in `pykeen/benchmark`. The good results I get are usually trained with `TransE` and `DistMult` with `MarginalRankingLoss`. The results from `LCWA` training loop is usually very weak.


Hi @Otamio ,

thanks for reporting the issue. I can confirm the issue for the current `master` branch (d473afe02914ae1d9e7293666a90cf602068858a) for `ConvE` on `fb15k237`.

I attach the `results.json`
 
<summary>Results</summary>
<details>

```json
{
  "losses": [
    0.00015769455933050934,
    1.5736911077561565e-05,
    1.3907649390836619e-05,
    1.2191183205795448e-05,
    1.1065662245915643e-05,
    1.0350252234919768e-05,
    9.732301984879646e-06,
    9.195092730793668e-06,
    8.724149137046549e-06,
    8.26930187102579e-06,
    7.86839677427621e-06,
    7.50775141416707e-06,
    7.11335127365703e-06,
    6.786873433701368e-06,
    6.557376685748488e-06,
    6.318650367310028e-06,
    6.1131843155509485e-06,
    5.926822472755629e-06,
    5.7935911986267785e-06,
    5.661806653513696e-06,
    5.536608879050252e-06,
    5.463718181003371e-06,
    5.308103962395996e-06,
    5.251838882472225e-06,
    5.157114114169516e-06,
    5.090611158716304e-06,
    5.040481336727853e-06,
    4.990157012788019e-06,
    4.9046080360058085e-06,
    4.847697807595979e-06,
    4.780115641886031e-06,
    4.784062781497894e-06,
    4.733890356303265e-06,
    4.681439771321108e-06,
    4.6467699006104446e-06,
    4.604807832438029e-06,
    4.567288476457432e-06,
    4.548195241241208e-06,
    4.53442562116458e-06,
    4.508071291626887e-06,
    4.472153170306626e-06,
    4.422793613197086e-06,
    4.385940052469937e-06,
    4.381353419912639e-06,
    4.3441945782044656e-06,
    4.347157503843425e-06,
    4.3324519967851525e-06,
    4.31644563838983e-06,
    4.2838222538679476e-06,
    4.254327251823406e-06,
    4.231440595203848e-06,
    4.2338237703780635e-06,
    4.225909090144762e-06,
    4.215364240160739e-06,
    4.138986756081537e-06,
    4.158177616062537e-06,
    4.14863149590013e-06,
    4.1375978013452176e-06,
    4.13363028386079e-06,
    4.102168712413924e-06,
    4.129914797821523e-06,
    4.087851412295377e-06,
    4.066219213225696e-06,
    4.065773352552286e-06,
    4.048778646047637e-06,
    4.032534755918289e-06,
    4.015679871611997e-06,
    4.014595574324017e-06,
    3.988723377680153e-06,
    3.976302050114897e-06,
    3.9708643784618775e-06,
    3.978105458686789e-06,
    3.970733640358988e-06,
    3.955002348148917e-06,
    3.955370332953428e-06,
    3.946489715040492e-06,
    3.939194473603768e-06,
    3.896746722075912e-06,
    3.909200907305605e-06,
    3.90598471365526e-06,
    3.916573081048982e-06,
    3.883121258401148e-06,
    3.875783574939597e-06,
    3.881082571523304e-06,
    3.861766753803233e-06,
    3.86156054160858e-06,
    3.8500722217310645e-06,
    3.864694419611763e-06,
    3.826007791194036e-06,
    3.846178759009242e-06,
    3.826529286751706e-06,
    3.797544732439981e-06,
    3.8055256607366605e-06,
    3.834631874499808e-06,
    3.817483556916817e-06,
    3.823126304329019e-06,
    3.787107822438425e-06,
    3.8059737946698676e-06,
    3.7688332922498445e-06,
    3.809523244862229e-06,
    3.7532249722734534e-06,
    3.791483365169483e-06,
    3.7566282895022776e-06,
    3.746313356259704e-06,
    3.745503137070567e-06,
    3.7714530408192606e-06,
    3.757957445956883e-06,
    3.7418270226388575e-06,
    3.733886630188502e-06,
    3.7600068999046046e-06,
    3.7248318285071497e-06,
    3.744841053460081e-06,
    3.7272677243712966e-06,
    3.7133821870990475e-06,
    3.709894508250358e-06,
    3.7184998369683788e-06,
    3.705753782057203e-06,
    3.7162072342428815e-06,
    3.684495230915373e-06,
    3.682931895454414e-06,
    3.671680607434859e-06,
    3.678396185510875e-06,
    3.693672755767302e-06,
    3.671880915609072e-06,
    3.6745464457098373e-06,
    3.655941061922949e-06,
    3.6383179365455843e-06,
    3.68340630622548e-06,
    3.651082587390731e-06,
    3.6498242406845327e-06,
    3.6383246264466347e-06,
    3.66534108917412e-06,
    3.654862888359785e-06,
    3.6383980495105506e-06,
    3.627596559927891e-06,
    3.6401135304466755e-06,
    3.6142704475491297e-06,
    3.6396124498974437e-06,
    3.626956304044426e-06,
    3.612422797904096e-06,
    3.653099418932685e-06,
    3.623208524961527e-06,
    3.607465556533202e-06,
    3.5984725118078016e-06,
    3.591972050369833e-06,
    3.586343446864333e-06,
    3.6150762378480075e-06,
    3.5951990499757876e-06,
    3.5847536119447072e-06,
    3.6153429051043596e-06,
    3.582213248788243e-06,
    3.622390023111824e-06,
    3.606049500970397e-06,
    3.596930479894218e-06,
    3.597846167683229e-06,
    3.570134226863313e-06,
    3.572301024140817e-06,
    3.5557868435771904e-06,
    3.5711715023595743e-06,
    3.571362871287664e-06,
    3.5565895333223583e-06,
    3.571222400669753e-06,
    3.5765564404903717e-06,
    3.582447530064068e-06,
    3.5890913367475646e-06,
    3.5394294983022387e-06,
    3.571189018242385e-06,
    3.5577164952559713e-06,
    3.55265640754443e-06,
    3.5492492478221605e-06,
    3.563303882609295e-06,
    3.543979418647079e-06,
    3.528817025838697e-06,
    3.536515680480003e-06,
    3.5278391078543958e-06,
    3.528376386741159e-06,
    3.5596392941024916e-06,
    3.518827405949999e-06,
    3.5481969414356875e-06,
    3.5330885086054633e-06,
    3.53443476836528e-06,
    3.5331671063879464e-06,
    3.5192509874330584e-06,
    3.4998663553858334e-06,
    3.512490468340755e-06,
    3.5187522678309455e-06,
    3.4907506343471615e-06,
    3.502968539185632e-06,
    3.546369636415919e-06,
    3.497089966928538e-06,
    3.5164230252204487e-06,
    3.5233218731046355e-06,
    3.494591662772456e-06,
    3.49420268433984e-06,
    3.5385981314975048e-06,
    3.511645050760643e-06,
    3.4872497985116056e-06,
    3.489717709580841e-06,
    3.4929282383579885e-06,
    3.4896487867624992e-06,
    3.478387634988582e-06,
    3.4732277565770843e-06,
    3.4777481332938575e-06,
    3.4617533749007574e-06,
    3.4718717116120806e-06,
    3.4921911618666517e-06,
    3.46824905222732e-06,
    3.514212138329723e-06,
    3.4995198984993624e-06,
    3.4648836217544402e-06,
    3.469376416822671e-06,
    3.4732764630930713e-06,
    3.503317560002734e-06,
    3.4703042624297066e-06,
    3.4620479148653433e-06,
    3.463569261514664e-06,
    3.4537149128328714e-06,
    3.4804705549679923e-06,
    3.4962120885128673e-06,
    3.4641602114586086e-06,
    3.4352676642846234e-06,
    3.436537341514933e-06,
    3.4515033088892783e-06,
    3.460370265856283e-06,
    3.4441337927911427e-06,
    3.4286963880431025e-06,
    3.461904163847219e-06,
    3.456113376515262e-06,
    3.456542648536102e-06,
    3.4419897471656136e-06,
    3.431834606082651e-06,
    3.4702691503650527e-06,
    3.424804360399159e-06,
    3.446870673929682e-06,
    3.468925951884358e-06,
    3.4385708060391894e-06,
    3.455616605671526e-06,
    3.414907017709598e-06,
    3.4614142536128897e-06,
    3.4154417046680045e-06,
    3.432040002260252e-06,
    3.429939723690811e-06,
    3.442964681058967e-06,
    3.4772528854097507e-06,
    3.443466169130653e-06,
    3.4329680877922443e-06,
    3.442592340663799e-06,
    3.4401888659717656e-06,
    3.430096023911836e-06,
    3.4134452599423332e-06,
    3.4295017432106575e-06,
    3.417361568614026e-06,
    3.434253076667739e-06,
    3.4237542152815794e-06,
    3.400815365068165e-06,
    3.4056917929476598e-06,
    3.411138716876932e-06,
    3.4427064395626686e-06,
    3.429023042565818e-06,
    3.392033549379919e-06,
    3.4226737439606086e-06,
    3.449882266226755e-06,
    3.4067522873079456e-06,
    3.3917091558679952e-06,
    3.3905519589180293e-06,
    3.3937762172744226e-06,
    3.4304136252532387e-06,
    3.3822594732890166e-06,
    3.398250550758881e-06,
    3.382242463892844e-06,
    3.3916817816748247e-06,
    3.39692831732528e-06,
    3.398493810360988e-06,
    3.4115731478676186e-06,
    3.4302825721759337e-06,
    3.426648314149951e-06,
    3.373745942114649e-06,
    3.3880229867529875e-06,
    3.380442546676252e-06,
    3.4256297739930044e-06,
    3.372138612893793e-06,
    3.3741658922537793e-06,
    3.381103946286741e-06,
    3.4058770222077837e-06,
    3.3834793406207404e-06,
    3.3634142631211545e-06,
    3.3981552608877852e-06,
    3.3711813576200315e-06,
    3.3969280548465996e-06,
    3.3875680470392534e-06,
    3.383838673933533e-06,
    3.3748861864416084e-06,
    3.38131714177528e-06,
    3.373378318558573e-06,
    3.3688170391086094e-06,
    3.3805763441139768e-06,
    3.380021975920865e-06,
    3.3728705030008434e-06,
    3.392650838573321e-06,
    3.3818053466759044e-06,
    3.3992843975058682e-06,
    3.3642091659988183e-06,
    3.3682563873703295e-06,
    3.3916686974042743e-06,
    3.404568551017441e-06,
    3.3556061754797643e-06,
    3.384136000060699e-06,
    3.3495714416292736e-06,
    3.356297890600068e-06,
    3.354803722354266e-06,
    3.38758175775892e-06,
    3.3975728871916703e-06,
    3.3750562133254495e-06,
    3.356196345764815e-06,
    3.349554825562264e-06,
    3.3536328603753666e-06,
    3.3883429301819294e-06,
    3.3535997964221304e-06,
    3.3480967006943574e-06,
    3.371363132077442e-06,
    3.3516378991403086e-06,
    3.345840392354146e-06,
    3.3656409387849764e-06,
    3.3473215624612905e-06,
    3.346497364824234e-06,
    3.371986128140631e-06,
    3.3500570322422467e-06,
    3.349233976873519e-06,
    3.3273428293649612e-06,
    3.355216200600308e-06,
    3.3593787238007574e-06,
    3.338956453647835e-06,
    3.346588045180854e-06,
    3.331383330310123e-06,
    3.3793809504887964e-06,
    3.330418360107459e-06,
    3.334636796710731e-06,
    3.324115348936951e-06,
    3.3172674979382687e-06,
    3.3552009348402845e-06,
    3.3461585301241987e-06,
    3.341527289023607e-06,
    3.3643025207215433e-06,
    3.3401513464248087e-06,
    3.34573106201239e-06,
    3.3237679821243893e-06,
    3.3605708321650085e-06,
    3.3527242048762617e-06,
    3.318258482500113e-06,
    3.3295086469164887e-06,
    3.344945741161972e-06,
    3.317496594968522e-06,
    3.3456067857457428e-06,
    3.340616502586479e-06,
    3.316618065800154e-06,
    3.3470742342452165e-06,
    3.334089687511333e-06,
    3.333365774034151e-06,
    3.328287375226613e-06,
    3.3528776711052584e-06,
    3.319162168208449e-06,
    3.330957860341569e-06,
    3.3223318282774204e-06,
    3.336245830293148e-06,
    3.310215111886484e-06,
    3.334631404037645e-06,
    3.3260401225952868e-06,
    3.3093800038856323e-06,
    3.3276256017298266e-06,
    3.3116795138834127e-06,
    3.3120644221466323e-06,
    3.3326025249187118e-06,
    3.328052056090645e-06,
    3.320702723047565e-06,
    3.313115212961765e-06,
    3.3280593496926593e-06,
    3.304568924448145e-06,
    3.3299520805370385e-06,
    3.3422370072649227e-06,
    3.311499553056466e-06,
    3.3465902494184787e-06,
    3.306566928686032e-06,
    3.301923896459853e-06,
    3.308124796424446e-06,
    3.3250145167067055e-06,
    3.304637905206225e-06,
    3.297280876871536e-06,
    3.3106589470021995e-06,
    3.3093374510364743e-06,
    3.3010643419724694e-06,
    3.3317810770860775e-06,
    3.2873757659796217e-06,
    3.300300181375511e-06,
    3.296760864998909e-06,
    3.2947704393954397e-06,
    3.3377774872994437e-06,
    3.3332181484418538e-06,
    3.2816330548259625e-06,
    3.322282638022948e-06,
    3.306707573512015e-06,
    3.335682285650958e-06,
    3.2967664083542e-06,
    3.2988449482449425e-06,
    3.3222095476264556e-06,
    3.2923295912391035e-06,
    3.2883029028942217e-06,
    3.3055015355014356e-06,
    3.2912493188186437e-06,
    3.292417951867479e-06,
    3.3239386928137986e-06,
    3.2928290953608265e-06,
    3.2983435618594417e-06,
    3.3017528394930147e-06,
    3.2969516267263193e-06,
    3.295863063673719e-06,
    3.307219132793435e-06,
    3.2944057944863094e-06,
    3.283924430512238e-06,
    3.3033858117143256e-06,
    3.305728259141137e-06,
    3.3139548948966496e-06,
    3.3111460524087986e-06,
    3.2817994309230087e-06,
    3.289832460071078e-06,
    3.3050967966823064e-06,
    3.3076748805526454e-06,
    3.282849347586923e-06,
    3.3139853231750823e-06,
    3.3015145753462583e-06,
    3.2798111778652947e-06,
    3.2851403091401705e-06,
    3.2771430533086e-06,
    3.291740784871045e-06,
    3.2989620644820593e-06,
    3.276619824808358e-06,
    3.299215707351886e-06,
    3.2816199252535362e-06,
    3.2807738853073095e-06,
    3.2685483786974114e-06,
    3.2759458734673954e-06,
    3.2825869385124735e-06,
    3.284551817940956e-06,
    3.2723856105144225e-06,
    3.2758418399454e-06,
    3.274042810490032e-06,
    3.2617607333084682e-06,
    3.3124397917401953e-06,
    3.2840938272528188e-06,
    3.272767276874302e-06,
    3.2883796156909256e-06,
    3.2880009828068714e-06,
    3.285334953024471e-06,
    3.2754913972715675e-06,
    3.2703100117458954e-06,
    3.2674710192070124e-06,
    3.280327009716337e-06,
    3.304291352077571e-06,
    3.2983541694978255e-06,
    3.259423520873106e-06,
    3.259138540576471e-06,
    3.3009692150325837e-06,
    3.2787831688385772e-06,
    3.2911263699164024e-06,
    3.269974975209573e-06,
    3.2588399666062176e-06,
    3.2679306532061147e-06,
    3.2752718807131304e-06,
    3.2624725204650676e-06,
    3.2597378721451654e-06,
    3.2617721419928974e-06,
    3.255555583911262e-06,
    3.269454135070889e-06,
    3.259800675127264e-06,
    3.2569472161245446e-06,
    3.2841733472103973e-06,
    3.2846011948469344e-06,
    3.2658506335189037e-06,
    3.294443284218946e-06,
    3.258777602644015e-06,
    3.269612023968428e-06,
    3.2545966728831017e-06,
    3.2592941665189314e-06,
    3.2537729004615064e-06,
    3.276349519014216e-06,
    3.266250115570575e-06,
    3.260626717697797e-06,
    3.254398610748674e-06,
    3.2509423222113865e-06,
    3.287349518694917e-06,
    3.2413301635796874e-06,
    3.2690694686769127e-06,
    3.296751916809068e-06,
    3.2479241523930235e-06,
    3.2553701656664886e-06,
    3.2523207575799296e-06,
    3.2498945322080652e-06,
    3.291933089389807e-06,
    3.2721728874875075e-06,
    3.2572426800140838e-06,
    3.252045605068356e-06,
    3.249896140521873e-06,
    3.2498313465903805e-06,
    3.240401294889086e-06,
    3.2595857239479994e-06,
    3.245945734897517e-06,
    3.2725386816643987e-06,
    3.250050129763943e-06,
    3.2563941933777033e-06,
    3.2611225134818447e-06,
    3.2614942976166404e-06,
    3.2593552701946133e-06,
    3.2438707540232455e-06,
    3.2396691559133617e-06,
    3.248184978040533e-06,
    3.241614096683604e-06,
    3.2460636535416796e-06,
    3.2480779417624396e-06,
    3.246061172243226e-06,
    3.2384360250478704e-06,
    3.2403966653484583e-06,
    3.2666121986877897e-06,
    3.2437828020838963e-06,
    3.235695660525606e-06,
    3.2389675113218113e-06,
    3.2522858572480773e-06,
    3.250655800095543e-06,
    3.2684961996857037e-06,
    3.2320749080970623e-06,
    3.2459293698378967e-06,
    3.247693178154137e-06,
    3.243291151324276e-06,
    3.2248444605918303e-06,
    3.245070894235102e-06,
    3.241909468608391e-06,
    3.2360174881626125e-06,
    3.236266294619726e-06,
    3.2348351445885396e-06,
    3.2457070327030083e-06,
    3.2547936779089493e-06,
    3.2402165272025817e-06,
    3.2316747943467013e-06,
    3.2437128752359967e-06,
    3.2358549979166005e-06,
    3.228476105496733e-06,
    3.2460990399451614e-06,
    3.231227052187228e-06,
    3.223612870962262e-06,
    3.23838536355179e-06,
    3.2540489636768847e-06,
    3.233545487286687e-06,
    3.2330231189388e-06,
    3.2662640512441912e-06,
    3.2221193099171394e-06,
    3.242381083495445e-06,
    3.241917227478169e-06,
    3.2387043820836356e-06,
    3.229888763613277e-06,
    3.2554901114237746e-06,
    3.2231944325058994e-06,
    3.214504270095247e-06,
    3.2359243235911088e-06,
    3.220142419853121e-06,
    3.2423120236049035e-06,
    3.2660861336679464e-06,
    3.241016237911947e-06,
    3.24738213333573e-06,
    3.235088593224143e-06,
    3.2543480124419056e-06,
    3.2269254562136594e-06,
    3.2327638397767872e-06,
    3.214655234610781e-06,
    3.2340996684394358e-06,
    3.2331190638400244e-06,
    3.252757265068617e-06,
    3.246150095742548e-06,
    3.2201658844749615e-06,
    3.22583946360505e-06,
    3.2234635168992508e-06,
    3.240732946033724e-06,
    3.2227569905931114e-06,
    3.2369569261891534e-06,
    3.2246573702606515e-06,
    3.2368206731175563e-06,
    3.226557507475663e-06,
    3.244386093969799e-06,
    3.2178381418815132e-06,
    3.213405358772084e-06,
    3.2444436518501584e-06,
    3.2241804914122794e-06,
    3.223961054180732e-06,
    3.2104456186521547e-06,
    3.218103582098644e-06,
    3.249458331333362e-06,
    3.2185041257307538e-06,
    3.2205953464525942e-06,
    3.2153641858310653e-06,
    3.2178643086727558e-06,
    3.2284405913536268e-06,
    3.2217871841721623e-06,
    3.2302184432513855e-06,
    3.2156903002309862e-06,
    3.2424968369821413e-06,
    3.234800942255547e-06,
    3.2133614911963827e-06,
    3.2173210977678268e-06,
    3.2407953774626685e-06,
    3.222535426117685e-06,
    3.2081393781177356e-06,
    3.2379588166429334e-06,
    3.2204253613709137e-06,
    3.2093790688115092e-06,
    3.215998930828824e-06,
    3.2040368756253777e-06,
    3.237043451799913e-06,
    3.211108123783958e-06,
    3.201021567991198e-06,
    3.2426570673839335e-06,
    3.213948237592876e-06,
    3.214350108978249e-06,
    3.21612321448376e-06,
    3.201277334410759e-06,
    3.202782000248329e-06,
    3.2324448766571073e-06,
    3.2170384101738536e-06,
    3.194949025933939e-06,
    3.206513906176096e-06,
    3.227271630498085e-06,
    3.2164150505290856e-06,
    3.2273256246956857e-06,
    3.207632768795362e-06,
    3.212212749365197e-06,
    3.2209416860985885e-06,
    3.243665456032113e-06,
    3.224958469081377e-06,
    3.2101255535108767e-06,
    3.209390965706283e-06,
    3.2302025576528204e-06,
    3.2061062254570367e-06,
    3.225007811962341e-06,
    3.218657861049005e-06,
    3.226221585705689e-06,
    3.2328800631975346e-06,
    3.238086852187563e-06,
    3.2159791504355064e-06,
    3.2343089014720516e-06,
    3.2303957159077926e-06,
    3.202742589755041e-06,
    3.2161941377785004e-06,
    3.191190626185936e-06,
    3.231221081672189e-06,
    3.205510898535283e-06,
    3.187076372231436e-06,
    3.2019949205963267e-06,
    3.2037618757402962e-06,
    3.217051098976526e-06,
    3.2098337921261536e-06,
    3.2187973418306496e-06,
    3.202636466125039e-06,
    3.2192666117137345e-06,
    3.222712876870737e-06,
    3.223445176833385e-06,
    3.2122649931216455e-06,
    3.212951521468806e-06,
    3.213339356660949e-06,
    3.195025048703358e-06,
    3.2102098272489916e-06,
    3.1877613669253907e-06,
    3.1941290565367724e-06,
    3.2229824708615846e-06,
    3.195571894063075e-06,
    3.19312291937238e-06,
    3.2391939998972728e-06,
    3.187412157706925e-06,
    3.204777369394698e-06,
    3.1856922591574035e-06,
    3.203923211663402e-06,
    3.2134134088959825e-06,
    3.189280757233617e-06,
    3.201498846196021e-06,
    3.1889069953705754e-06,
    3.2113792384012913e-06,
    3.1918289702524683e-06,
    3.1913755739133724e-06,
    3.2012806933601477e-06,
    3.187497122055611e-06,
    3.2115282413261576e-06,
    3.1804322303529437e-06,
    3.234545894249854e-06,
    3.1935942017864394e-06,
    3.212482233698285e-06,
    3.2034979254299238e-06,
    3.215007161358513e-06,
    3.187040447649446e-06,
    3.2205864770063575e-06,
    3.194007753084211e-06,
    3.190276031863663e-06,
    3.1856071834011e-06,
    3.193697426874101e-06,
    3.2085457800795524e-06,
    3.2028288549286216e-06,
    3.2247117807299956e-06,
    3.1966563118968193e-06,
    3.1956320051804895e-06,
    3.222916258767506e-06,
    3.1991094080860745e-06,
    3.1943475146256294e-06,
    3.1926632206285365e-06,
    3.187278396627358e-06,
    3.197235314394541e-06,
    3.186869860227802e-06,
    3.1928029378354224e-06,
    3.1880839689717174e-06,
    3.2022193143974935e-06,
    3.1900623938555126e-06,
    3.1987706177157723e-06,
    3.1982906495905516e-06,
    3.1993488145011604e-06,
    3.191472137486566e-06,
    3.1885364118327046e-06,
    3.1901520839870084e-06,
    3.211762044890785e-06,
    3.181363289282307e-06,
    3.1992242339536735e-06,
    3.1790422515609154e-06,
    3.2153346843940156e-06,
    3.179156467700262e-06,
    3.210698793532217e-06,
    3.2078507474231632e-06,
    3.179462211032621e-06,
    3.2134877928262955e-06,
    3.178463555871607e-06,
    3.1737872014909416e-06,
    3.1999620928259045e-06,
    3.2060848624144935e-06,
    3.1811809449545185e-06,
    3.182321986244522e-06,
    3.177029314813714e-06,
    3.179729118602787e-06,
    3.1963996808530436e-06,
    3.1853488614113394e-06,
    3.186553192727214e-06,
    3.17744760861516e-06,
    3.20192863034193e-06,
    3.182510921703606e-06,
    3.1822882925368824e-06,
    3.196430651976272e-06,
    3.2155312766478357e-06,
    3.2185283937317833e-06,
    3.1935982628176892e-06,
    3.1854276833700583e-06,
    3.1816842639608184e-06,
    3.171448729546773e-06,
    3.1792728995485244e-06,
    3.198488625593086e-06,
    3.179781381607672e-06,
    3.193379690015928e-06,
    3.190269776510649e-06,
    3.169728300693104e-06,
    3.182949535632307e-06,
    3.2042864529920958e-06,
    3.1809959258717672e-06,
    3.1770253006397696e-06,
    3.17825594534609e-06,
    3.1707926535871985e-06,
    3.1695904584589216e-06,
    3.173283162320916e-06,
    3.1665214929887087e-06,
    3.1961814306084155e-06,
    3.182078247570217e-06,
    3.1821828334640124e-06,
    3.1715684027150896e-06,
    3.1695568590491783e-06,
    3.1922770744380913e-06,
    3.179432398898586e-06,
    3.1699382252112195e-06,
    3.171293096021595e-06,
    3.164512838726991e-06,
    3.174202614054197e-06,
    3.1878514849943495e-06,
    3.166891728693721e-06,
    3.181360165980445e-06,
    3.17691847570562e-06,
    3.1656622694188903e-06,
    3.1795815567830875e-06,
    3.184360630931348e-06,
    3.1706189784441356e-06,
    3.1708225922942863e-06,
    3.173460749951739e-06,
    3.173925282580723e-06,
    3.1890256870630574e-06,
    3.160144096138029e-06,
    3.201080921614585e-06,
    3.1655883805039245e-06,
    3.1911792803019616e-06,
    3.1781696124141244e-06,
    3.1757332265897755e-06,
    3.192413395170859e-06,
    3.171679249794758e-06,
    3.183479543859637e-06,
    3.1984449771384756e-06,
    3.1706123546488267e-06,
    3.1736984926408514e-06,
    3.158370988104945e-06,
    3.1597077786061343e-06,
    3.176073562279002e-06,
    3.169837533917748e-06,
    3.159563405805182e-06,
    3.1684857434405027e-06,
    3.1704370525268057e-06,
    3.163812088118382e-06,
    3.1994699620219434e-06,
    3.1565006807169762e-06,
    3.1764835506717296e-06,
    3.1816730047920223e-06,
    3.190656782270164e-06,
    3.193005828605417e-06,
    3.181003091539729e-06,
    3.1829414487613933e-06,
    3.164633272694622e-06,
    3.164863768250168e-06,
    3.173771518487032e-06,
    3.1604630505084193e-06,
    3.1599341680229833e-06,
    3.182102831517806e-06,
    3.1541741846056143e-06,
    3.159349697608288e-06,
    3.1623806410002155e-06,
    3.193147137016389e-06,
    3.1710065525186e-06,
    3.1602519525161758e-06,
    3.1733933450378813e-06,
    3.1736316290163603e-06,
    3.2046453688665728e-06,
    3.1683831035830833e-06,
    3.175147862775425e-06,
    3.1710687858247485e-06,
    3.1921242157986315e-06,
    3.158880187216961e-06,
    3.1709268546588047e-06,
    3.1622743371348533e-06,
    3.1806017180861333e-06,
    3.1692029061604664e-06,
    3.1586174942766798e-06,
    3.1674972994778513e-06,
    3.178527473318737e-06,
    3.1653190985710633e-06,
    3.1644385422895713e-06,
    3.1839638514379366e-06,
    3.1529229417390663e-06,
    3.1726208103988905e-06,
    3.157519840323608e-06,
    3.1733891387684285e-06,
    3.1584236146914093e-06,
    3.1521479454389156e-06,
    3.1729611729192714e-06,
    3.1814956162561387e-06,
    3.1631667193446612e-06,
    3.169842995807434e-06,
    3.1837220448011445e-06,
    3.157725559252771e-06,
    3.175690929608723e-06,
    3.1664403532460318e-06,
    3.170931159892441e-06,
    3.1955293184657648e-06,
    3.162423295146701e-06,
    3.1606860854477394e-06,
    3.161396605318386e-06,
    3.151615514241727e-06,
    3.185299229609399e-06,
    3.178737395212066e-06,
    3.172249948633619e-06,
    3.150906172025424e-06,
    3.156841297938891e-06,
    3.166684891410953e-06,
    3.15844590826948e-06,
    3.1555740137364278e-06,
    3.1698648060358765e-06,
    3.173433598573803e-06,
    3.151467172763134e-06,
    3.1531158019349155e-06,
    3.1571007907779925e-06,
    3.169925971345441e-06,
    3.158653615231608e-06,
    3.173323115270142e-06,
    3.1494939346524825e-06,
    3.1765789301744344e-06,
    3.1613321043908715e-06,
    3.1500628614686112e-06,
    3.1521657311886943e-06,
    3.1515100184218414e-06,
    3.1729343299051776e-06,
    3.145365326030621e-06,
    3.148708870253456e-06,
    3.1577673376926087e-06,
    3.161508519228067e-06,
    3.1708978659301097e-06,
    3.149409082877986e-06,
    3.148542768689666e-06,
    3.156237135790343e-06,
    3.156832923119144e-06,
    3.1499320599819814e-06,
    3.175898598073978e-06,
    3.1427561318261388e-06,
    3.1852126909719196e-06,
    3.1700198014462194e-06,
    3.143077547468832e-06,
    3.145600759685065e-06,
    3.154220993303531e-06,
    3.1931212084002445e-06,
    3.1797102870212897e-06,
    3.17519200527324e-06,
    3.14731582668259e-06,
    3.170011443152908e-06,
    3.1741602054710985e-06,
    3.1449108294197844e-06,
    3.1737117217607472e-06,
    3.168057978630571e-06,
    3.1687428988583525e-06,
    3.150764678323947e-06,
    3.1502830777757658e-06,
    3.1634124783270867e-06,
    3.170332360280538e-06,
    3.1573235612943457e-06,
    3.1429595512476374e-06,
    3.142778231947701e-06,
    3.144750709259038e-06,
    3.1481435279239516e-06,
    3.1472229628922105e-06,
    3.1566434875895674e-06,
    3.155534241411419e-06,
    3.173614415275675e-06,
    3.1543521076258615e-06,
    3.140002342199897e-06,
    3.168480885057352e-06,
    3.1745510951373614e-06,
    3.1487883272161513e-06,
    3.143455109051015e-06,
    3.142757264956321e-06,
    3.186722288103384e-06,
    3.150821825765422e-06,
    3.152751613155402e-06,
    3.160636469588948e-06,
    3.1495041441009874e-06,
    3.1420938454265687e-06,
    3.1401587682162597e-06,
    3.1470046881495726e-06,
    3.1598303685930847e-06,
    3.1428114899407316e-06,
    3.175140737548623e-06,
    3.1570543334126495e-06,
    3.1400817373342815e-06,
    3.1473487308154745e-06,
    3.1426075000655687e-06,
    3.135056206593593e-06,
    3.16946907567947e-06,
    3.1457639809828075e-06,
    3.1426898831794734e-06,
    3.1360273042152184e-06,
    3.1717837277806516e-06,
    3.1437479730406228e-06,
    3.14724991906378e-06,
    3.1359454855276897e-06,
    3.1646900229183616e-06,
    3.1547201171228114e-06,
    3.1222772757416875e-06,
    3.1480510327702962e-06,
    3.1474150916474715e-06,
    3.1523531753800194e-06,
    3.18255175035948e-06,
    3.1431635555105206e-06,
    3.140159935177028e-06,
    3.1355127563709924e-06,
    3.155876574660614e-06,
    3.1552534325815078e-06,
    3.1450882253439565e-06,
    3.1679282844151134e-06,
    3.1338755529933613e-06,
    3.155448775825669e-06,
    3.148468226105573e-06,
    3.135874789714472e-06,
    3.1491562923833685e-06,
    3.1485250406851965e-06,
    3.1392411431401626e-06,
    3.162187020005052e-06,
    3.1467301930012706e-06,
    3.170007007068789e-06,
    3.1464196251163463e-06,
    3.1579343077691863e-06,
    3.1441069254676864e-06,
    3.1539421657015817e-06,
    3.1548510168931245e-06,
    3.163261626191313e-06,
    3.132476156077441e-06,
    3.160240753036976e-06,
    3.1417804248844653e-06,
    3.1407822942919536e-06,
    3.1530944068116445e-06,
    3.136238143617355e-06,
    3.171821733721359e-06,
    3.160625620664607e-06
  ],
  "metrics": {
    "adjusted_arithmetic_mean_rank": {
      "both": {
        "realistic": 0.8842599636137988
      },
      "head": {
        "realistic": 0.8869639287888795
      },
      "tail": {
        "realistic": 0.8816300237482667
      }
    },
    "adjusted_arithmetic_mean_rank_index": {
      "both": {
        "realistic": 0.11575625858756722
      },
      "head": {
        "realistic": 0.11305213742522224
      },
      "tail": {
        "realistic": 0.11838633993598013
      }
    },
    "arithmetic_mean_rank": {
      "both": {
        "optimistic": 6309.785668852138,
        "pessimistic": 6309.787063313436,
        "realistic": 6309.786366082787
      },
      "head": {
        "optimistic": 6241.243859477444,
        "pessimistic": 6241.244838046776,
        "realistic": 6241.244348762109
      },
      "tail": {
        "optimistic": 6378.327478226833,
        "pessimistic": 6378.329288580096,
        "realistic": 6378.328383403464
      }
    },
    "geometric_mean_rank": {
      "both": {
        "optimistic": 3361.049537465182,
        "pessimistic": 3361.0503085256337,
        "realistic": 3361.049923016706
      },
      "head": {
        "optimistic": 3865.320752624691,
        "pessimistic": 3865.3213440989953,
        "realistic": 3865.3210483770304
      },
      "tail": {
        "optimistic": 2922.565736782403,
        "pessimistic": 2922.5666305044256,
        "realistic": 2922.566183668969
      }
    },
    "harmonic_mean_rank": {
      "both": {
        "optimistic": 93.04044741384659,
        "pessimistic": 93.04044785316799,
        "realistic": 93.04044763353873
      },
      "head": {
        "optimistic": 381.6118900909571,
        "pessimistic": 381.6118946725886,
        "realistic": 381.61189238206657
      },
      "tail": {
        "optimistic": 52.97854968409628,
        "pessimistic": 52.97854988067762,
        "realistic": 52.978549782401686
      }
    },
    "hits_at_k": {
      "both": {
        "optimistic": {
          "1": 0.005871415989822879,
          "3": 0.009785693316371465,
          "5": 0.012794794011155691,
          "10": 0.01917995890008807
        },
        "pessimistic": {
          "1": 0.005871415989822879,
          "3": 0.009785693316371465,
          "5": 0.012794794011155691,
          "10": 0.01917995890008807
        },
        "realistic": {
          "1": 0.005871415989822879,
          "3": 0.009785693316371465,
          "5": 0.012794794011155691,
          "10": 0.01917995890008807
        }
      },
      "head": {
        "optimistic": {
          "1": 0.0004892846658185732,
          "3": 0.0014678539974557197,
          "5": 0.0023485663959291517,
          "10": 0.004501418925530874
        },
        "pessimistic": {
          "1": 0.0004892846658185732,
          "3": 0.0014678539974557197,
          "5": 0.0023485663959291517,
          "10": 0.004501418925530874
        },
        "realistic": {
          "1": 0.0004892846658185732,
          "3": 0.0014678539974557197,
          "5": 0.0023485663959291517,
          "10": 0.004501418925530874
        }
      },
      "tail": {
        "optimistic": {
          "1": 0.011253547313827184,
          "3": 0.01810353263528721,
          "5": 0.023241021626382228,
          "10": 0.03385849887464527
        },
        "pessimistic": {
          "1": 0.011253547313827184,
          "3": 0.01810353263528721,
          "5": 0.023241021626382228,
          "10": 0.03385849887464527
        },
        "realistic": {
          "1": 0.011253547313827184,
          "3": 0.01810353263528721,
          "5": 0.023241021626382228,
          "10": 0.03385849887464527
        }
      }
    },
    "inverse_arithmetic_mean_rank": {
      "both": {
        "optimistic": 0.0001584839885982875,
        "pessimistic": 0.00015848395357336728,
        "realistic": 0.00015848397108582545
      },
      "head": {
        "optimistic": 0.00016022447167826675,
        "pessimistic": 0.00016022444655655493,
        "realistic": 0.00016022445911740988
      },
      "tail": {
        "optimistic": 0.0001567809121456396,
        "pessimistic": 0.00015678086764670844,
        "realistic": 0.00015678088989617089
      }
    },
    "inverse_geometric_mean_rank": {
      "both": {
        "optimistic": 0.0002975261116663501,
        "pessimistic": 0.0002975260434107166,
        "realistic": 0.00029752607753664404
      },
      "head": {
        "optimistic": 0.0002587107420052952,
        "pessimistic": 0.00025871070241718794,
        "realistic": 0.0002587107222102236
      },
      "tail": {
        "optimistic": 0.0003421651008271073,
        "pessimistic": 0.0003421649961928852,
        "realistic": 0.00034216504850699634
      }
    },
    "inverse_harmonic_mean_rank": {
      "both": {
        "optimistic": 0.010748013662831728,
        "pessimistic": 0.010748013612081409,
        "realistic": 0.010748013637452935
      },
      "head": {
        "optimistic": 0.002620463423615156,
        "pessimistic": 0.002620463392153878,
        "realistic": 0.0026204634078825
      },
      "tail": {
        "optimistic": 0.0188755639020483,
        "pessimistic": 0.01887556383200894,
        "realistic": 0.018875563867023368
      }
    },
    "inverse_median_rank": {
      "both": {
        "optimistic": 0.00016342539630658605,
        "pessimistic": 0.00016342539630658605,
        "realistic": 0.00016342539630658605
      },
      "head": {
        "optimistic": 0.00016605778811026238,
        "pessimistic": 0.00016605778811026238,
        "realistic": 0.00016605778811026238
      },
      "tail": {
        "optimistic": 0.00016025641025641026,
        "pessimistic": 0.00016025641025641026,
        "realistic": 0.00016025641025641026
      }
    },
    "median_rank": {
      "both": {
        "optimistic": 6119.0,
        "pessimistic": 6119.0,
        "realistic": 6119.0
      },
      "head": {
        "optimistic": 6022.0,
        "pessimistic": 6022.0,
        "realistic": 6022.0
      },
      "tail": {
        "optimistic": 6240.0,
        "pessimistic": 6240.0,
        "realistic": 6240.0
      }
    },
    "rank_mad": {
      "both": {
        "optimistic": 3831.5,
        "pessimistic": 3832.0,
        "realistic": 3831.75
      },
      "head": {
        "optimistic": 3484.5,
        "pessimistic": 3484.5,
        "realistic": 3484.5
      },
      "tail": {
        "optimistic": 4226.0,
        "pessimistic": 4226.0,
        "realistic": 4226.0
      }
    },
    "rank_std": {
      "both": {
        "optimistic": 4385.869459221195,
        "pessimistic": 4385.8699430354545,
        "realistic": 4385.869701088643
      },
      "head": {
        "optimistic": 4133.875014784683,
        "pessimistic": 4133.875462541162,
        "realistic": 4133.875238633367
      },
      "tail": {
        "optimistic": 4623.135649051696,
        "pessimistic": 4623.136154317189,
        "realistic": 4623.13590163559
      }
    },
    "rank_var": {
      "both": {
        "optimistic": 19235850.913329214,
        "pessimistic": 19235855.15722182,
        "realistic": 19235853.034927383
      },
      "head": {
        "optimistic": 17088922.63786106,
        "pessimistic": 17088926.339799903,
        "realistic": 17088924.48858608
      },
      "tail": {
        "optimistic": 21373383.229532648,
        "pessimistic": 21373387.901354726,
        "realistic": 21373385.564991917
      }
    }
  },
  "times": {
    "evaluation": 3.3644840717315674,
    "training": 18908.355619192123
  }
}
```
</details>


Could you provide an overview of the configurations you tested, and where you could reproduce the results (from https://github.com/pykeen/benchmarking )? This may help us to track down the root cause.

Hi, @mberr. Sure, I would like to provide as much information as I can.

I am still inspecting some of the results I have got (and also the parameters). Here are some of the results of the `reproduce` command. I can also upload the detailed config (same as what pykeen provides) and results later if that helps.
![Screenshot from 2021-11-24 02-31-59](https://user-images.githubusercontent.com/32283936/143222287-0820ce8a-e921-415f-bb34-b4c099434a52.png)

I have just skimmed some of the replicates on `pykeen/benchmark`. Overall, similar performance for `DistMult` and `SimplE`. Slightly worse result on `TransE`, bad result on `RotatE`, `ConvE` and `TuckER`. 

I mainly trained the model on fb15k237. I remembered I was able to tune the parameters for `DistMult` and `TransE` with `MarginalRankingLoss` and `sLCWA`, but not `RotatE`, `ConvE`, and `TuckER`. If I train `DistMult` with `LCWA`, the results become very bad.

Meantime, would you like to share the version of pykeen when you compiled the benchmarks? Thanks, I would like to reproduce some of the results and see how `hpo` works on some of the datasets I have.



 
> Meantime, would you like to share the version of pykeen when you compiled the benchmarks? Thanks, I would like to reproduce some of the results and see how `hpo` works on some of the datasets I have.

Hi @Otamio,
For the reproduction experiments, we mainly used version 1.0.
Some experiments might have run based on the version 1.1 or 1.2 due to bug fixes/optimizations.
We will run the reproduction experiments based on 1.0-1.X in the next few days in order to trace back the root of the bug and add to each experiment the used pykeen version.



@mali-git Thanks. I can confirm version 1.5.0 produces similar results on `conve fb15k237`. I will continue to work on other versions to identify the issue.

Hi, authors. I am still running experiments to verify the root cause, but this commit may worth checking: 
https://github.com/pykeen/pykeen/commit/a007a273ff146107fd1b0c099d561c7b7279965a

@Otamio, we could set up a joint call in order to discuss this issue. 

Hi @Otamio, we [fixed a bug](https://github.com/pykeen/pykeen/pull/699) which is closely related to the issue reported here.


No response, so marking as stale