metadata

title: mrr
tags:
  - evaluate
  - metric
description: >-
  This is the mean reciprocal rank (mrr) metric for retrieval systems. It is the
  average of the precision scores computer after each relevant document is got.
  You can refer to
  [here](https://amenra.github.io/ranx/metrics/#mean-reciprocal-rank)
sdk: gradio
sdk_version: 3.19.1
app_file: app.py
pinned: false

Metric Card for mrr

Metric Description

This is the mean reciprocal rank (mrr) metric for retrieval systems. It is the average of the precision scores computer after each relevant document is got. You can refer to here

How to Use

>>> my_new_module = evaluate.load("mrr")
>>> references= [json.dumps({"q_1":{"d_1":1, "d_2":2} }), 
             json.dumps({"q_2":{"d_2":1, "d_3":2, "d_5":3}})] 
>>> predictions = [json.dumps({"q_1": { "d_1": 0.8, "d_2": 0.9}}),
         json.dumps({"q_2": {"d_2": 0.9, "d_1": 0.8, "d_5": 0.7, "d_3": 0.3}})]
>>> results = my_new_module.compute(references=references, predictions=predictions)
>>> print(results)
{'mrr': 1.0}

Inputs

predictions: a list of dictionaries where each dictionary consists of document relevancy scores produced by the model for a given query. One dictionary per query. The dictionaries should be converted to string.
references: a lift of list of dictionaries where each dictionary consists of the relevant order for the documents for a given query in a sorted relevancy order. The dictionaries should be converted to string.
k: an optional paramater whose default is None to calculate mrr@k

Output Values

mrr (float): mean reciprocal rank. Minimum possible value is 0. Maximum possible value is 1.0

Limitations and Bias

Note any known limitations or biases that the metric has, with links and references if possible.

Citation

@inproceedings{ranx,
  author       = {Elias Bassani},
  title        = {ranx: {A} Blazing-Fast Python Library for Ranking Evaluation and Comparison},
  booktitle    = {{ECIR} {(2)}},
  series       = {Lecture Notes in Computer Science},
  volume       = {13186},
  pages        = {259--264},
  publisher    = {Springer},
  year         = {2022},
  doi          = {10.1007/978-3-030-99739-7\_30}
}