Repository: pykeen/pykeen
Issue: Clarify formula of Adjusted Mean Rank
Root cause: Documentation Deficiency

Thanks for making such a great library!
I'm currently evaluating some models and I noticed that PyKEEN implements the AMR metric proposed in Berrendorf et al. 2020. In the first version of the paper, the metric would take values in the interval (0, 2), with 0 being the best performance, 1 random performance, and 2 the worst. From looking at the code, this is the version implemented in PyKEEN:

https://github.com/pykeen/pykeen/blob/d79fe39f83bc2831137f22be6421b37568694cf4/src/pykeen/evaluation/rank_based_evaluator.py#L86-L89

The paper, also linked in PyKEEN's documentation, was later updated on arXiv on 4 Nov. 2020, with a modified version of the metric that instead has a range of (-1, 1) where 1 is the best performance and -1 is the worst. This was the version I was familiar with so I was quite confused when seeing that during training AMR was going down to zero, which is expected with the version above, but not with the new version.
I think it would be useful to add the explicit formulas used during evaluation to the documentation.




Hi @dfdazac, luckily we have Berrendorf ;) we'll try and clear that up in the docs ASAP 

I made a short update here that's reflected in these docs: https://pykeen.readthedocs.io/en/latest/reference/evaluation/rank_based.html

We're also going to add the AMRI (AMR that's reindexed to [-1,1]) in #378 so you can decide from either option

Hi @dfdazac ,

thanks for opening the issue. Indeed there are two versions of the paper, where we addressed reviewers concerns about larger values corresponding to better performance being more intuitive. We still tried to maintain "backwards compatability".

#### Adjusted Mean Rank (AMR)
This is  `AMR = MR / E[MR]`. Since MR is 1-based, we end up with a value range of `(0, 2)`, where a value close to 1 indicates performance similar to the expected performance for a random scoring model. This is what is currently implemented in pykeen as `adjusted_mean_rank` (although we found a mistake in this implementation, which will be fixed in #378 ).

#### Adjusted Mean Rank Index (AMRI)
This is given by `AMRI = 1 - (MR - 1) / (E[MR- 1])`, and in analogy to other performance indices, e.g., [Adjusted Rand Index (ARI)](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.adjusted_rand_score.html), has larger scores for better models. It also switches to 0-based ranks, which allows to has precise extreme values of -1 and 1. The value range is `[-1, 1]`, where larger values correspond to better predictions. The optimal model which always predicts the "correct" entity at first place, obtains exactly the score of `1`. `0` corresponds to performance similar to a random scoring model. We will additionally add this metric in #378 .

P.S.: I just noticed that the Arxiv version has a typo:
![Screenshot from 2021-04-15 17-59-51](https://user-images.githubusercontent.com/36984984/114900708-6cfd3480-9e14-11eb-9d08-009e5db1c344.png)
The highlighted `AMR` should be `AMRI`. I will fix this soon.


P.P.S.: A more precise value range of `AMR` is `[eps, 2 - eps]` with a candidate-set-size dependent `eps`. While in the unfiltered setting the `eps` is rather easy to calculate, it becomes more difficult for the filtered setting. This is somewhat annoying since without knowing this constant, it is hard to judge whether a small value (e.g. `0.01`) is already a perfect fit, or not.

Thanks @mberr a lot for the detailed explanation! It's much more clear now.

@mberr in the screenshot above that has the AMR/AMRI typo, is there also a typo in Equation 8? Should the final expression on the right-hand side should have a "1 - ..." in front?

@ejin700 Yes, you are right 👍 I have scheduled an Arxiv replacement that fixes this typo. Thanks for reporting the issue.