zhiminy/EvalEng huggingface dataset:
Repository: AmenRa/ranx
Issue: bug: using built-in `type` function instead of type argument
Root cause: Validation Gap

It’s all in the title :)
https://github.com/AmenRa/ranx/blob/f41e64d92e62a6ecb484f1271b481ec581aa05f9/ranx/run.py#L140

Thank you. It should be fixed now (0.1.8). Closing.
Repository: AmenRa/ranx
Issue: set Report’s rounding_digits in compare
Root cause: Unimplemented Feature Gap

Hi,

`compare` does not have a `rounding_digits` argument and thus always uses the default from `Report` (which is 4). Why is that?

Also, would you like to add an option in Report to print results as percentages rather than ratios ?

Hi Paul,

I have chosen the number of digits based on the max number of digits I found in IR / SecSys papers for reporting performance scores. However, I was planning on adding an argument as you suggested, but it seems I forgot about it. My bad.

Regarding using percentages instead of ratios, do you mean 0.432 -> 43.2?

Best,

Elias


Thanks for your (very) quick answer!
Yes, I mean 0.432 -> 43.2 (optionally).

In the meantime, mind that you can change the Report parameter before printing:
`report.rounding_digits = 3`

Added in `0.1.11`. Please, take a look at the [changelog](https://github.com/AmenRa/ranx/blob/master/changelog.md) I added as some changes could break your code.

Closing.

How thoughtful of you :smiley:
Repository: AmenRa/ranx
Issue: [Feature Request] Benchmarking
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
To claim "blazing-fast", it would be nice to have benchmarks against existing implementations.

**Describe the solution you'd like**
The implementation is benchmarked against some/all of the sources below:
1. naive [numpy implementation](https://gist.github.com/bwhite/3726239/7e4dd612254e45fa275f9ddd9eb4af384a02794c)
2. [scikit-learn](https://github.com/scikit-learn/scikit-learn/) metrics module
3. [AstraZeneca/rexmex](https://github.com/AstraZeneca/rexmex)
4. [microsoft/rankerEval](https://github.com/microsoft/rankerEval)
5. [pytrec_eval](https://github.com/cvangysel/pytrec_eval)
6. ?


Hi, thank you for your interest in `ranx`!

I compared `ranx` against `pytrec_eval` for my ECIR paper, you can re-run the benchmark using this [notebook](https://github.com/AmenRa/ranx/blob/master/dev/efficiency_test.ipynb).

Here are the results reported in the ECIR paper that will be out in April:

![Screenshot 2022-02-16 at 15 26 36](https://user-images.githubusercontent.com/11797097/154285566-e458a316-1846-497b-bc45-8ffce55d0b39.png)

Adding a comprehensive comparison with other libraries is on my to-do list, but I have not had the time to do it yet.

Please, let me know what you think about the benchmark results.

Elias
Repository: AmenRa/ranx
Issue: [Feature Request] Reciprocal Rank Fusion
GitHub labels: enhancement

Hi,

Thank you for this nice library.

How would you go about implementing [Reciprocal Rank Fusion](https://plg.uwaterloo.ca/~gvcormac/cormack) in ranx?

Thanks,

Maxime.

Hi Maxime and thanks for your question,

I plan to add many fusion methods to `ranx` this year, but I am currently busy with other projects.

Right now, your best bet is to implement `RRF` with Python dictionaries and pass the resulting fused run to `ranx `.

Alternatively, you can use an available Python implementation of `RRF` and convert the output to a data structure that can be loaded into `Ranx`.

Best,

Elias
Repository: AmenRa/ranx
Issue: [Feature Request] from ranx import Report
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
Hi! I’d like to be able to import Report so that I can load a previously saved report (output of `compare`) and tweak the runs.

**Describe the solution you'd like**
[`from .report import Report`](https://github.com/AmenRa/ranx/blob/master/ranx/__init__.py)

**Describe alternatives you've considered**
Re-run `compare` with different runs :sweat_smile: 



mmh actually it’s not straightforward to load the saved report into a new one. it would need something like a `from_dict` method that reverses `to_dict` (https://github.com/AmenRa/ranx/blob/master/ranx/report.py#L240) 

Hi Paul,

Please, give me a more detailed description of the use case.
What do you mean by "tweak the runs"? Do you want to switch the compared runs without re-computing the metric scores, or do you need to expand the compared set of runs for printing tables?

As you pointed out, loading a `report` is not straightforward, and other stuff needs to be implemented first to make it doable.

I will add some new functionalities for saving/loading `Runs` with pre-computed metric scores (basically, saving/loading the data produced by the `evaluate` function, which `compare` also uses).
This and the addition of a mechanism to avoid re-computing metric scores during comparisons should make them much smoother.

Could this work in your use case?

I meant "comparing only models A and B instead of A, B and C".
Saving metric scores along with runs sounds great! But the statistical tests also take some time, don’t they? 
My idea was basically to separate the computation from the visualization/printing/formatting of the results.

I timed `compare` with five `runs` and `evaluate` with the same runs (`metrics=["map@100", "mrr@100", "ndcg@10"]`).
I got 40+ secs for `compare` and less than 1 sec in total for `evaluate`.
The `Fisher's Randomization Test` is very slow, not sure I can make it much faster.

However, I added the `Two-sided Paired Student's t-Test` few days ago on my local branch.
`Compare` with it takes 2.5 sec on my machine.

I'll commit it in the afternoon so you can use that for faster comparisons.

The new version is out. Hope it helps.

Thanks! So, should I close the issue?

Well, I don't know honestly :D

Do you think the speed of the comparison with the `Two-sided Paired Student's t-Test` "solves" your problem?
Do you still need a `report` import function?

Well, the issue was not really about speed (I’m patient enough even to run Fisher's Randomization Test) 
It was more generally to separate the computation from the visualization/printing/formatting of the results.

But anyway, consider it closed :) (and thanks again for your quick feedback)

You are always welcome! :)
Repository: AmenRa/ranx
Issue: [BUG] win_tie_loss in Report.to_dict
Root cause: Algorithmic Error

**Describe the bug**

When converting a Report to dict, you only keep one `metric` while iterating over `metrics` (overwriting the previous `metric` in each loop)
https://github.com/AmenRa/ranx/blob/master/ranx/report.py#L315

**How to fix**
replace the line above with:
`d[m1]["win_tie_loss"][m2][metric] = self.win_tie_loss[(m1, m2)][metric]`
and init `d[m1]["win_tie_loss"][m2] = {}` at the same place as
https://github.com/AmenRa/ranx/blob/master/ranx/report.py#L309 (just above)

Well spotted, thanks!

Fixed in `v0.1.14`.
Repository: AmenRa/ranx
Issue: [BUG] from_ir_datasets seems to be missing from Qrels
Root cause: Unimplemented Feature Gap

**Describe the bug**
Qrels.from_ir_datasets("msmarco-document/dev") does not seem to work, it seems to have been removed by a commit 10 days ago. (https://github.com/AmenRa/ranx/commit/e0ca82c07392d90998b298515f48d0059c48c93d#)



Hi and thank you very much for pointing it out.
I must have used an old version of the code when I did the refactoring from the commit you mentioned, gotta check everything... :S

I restored "from_ir_datasets" in `v0.2.4`.

Thanks again!

Thanks a lot for this library, just came back after a while and this feature is pretty great, thanks for fixing it!

Super glad you like it! :D
Please, give it a star if you haven't yet and spread the word!
Repository: AmenRa/ranx
Issue: Why in Recall@k you divide on len(relevant), but not min(len(relevant), k)
Root cause: Documentation Deficiency

The question about Recall@k arose when I looked at the best scores R@1 of Stanford Online Products dataset in paperswithcode https://paperswithcode.com/sota/metric-learning-on-stanford-online-products-1. This benchmark use R@1 metric to measure best models and approach in retrieval task in SOP dataset. Sop dataset has 4.3 images per class (query), so maximum score R@1 with ranx formula would be `1 / 4.3`.

In benchmark SOP and many others benchmark they use `divide coefficient = min(len(relevant), k)`.

What do you think about overwrite this coefficient? Why papers with code write R@1 but it's actually not R@1 it's HitRate@1?

Hi, that's an interesting question!

`ranx` is built to reproduce [`trec_eval`](https://github.com/usnistgov/trec_eval)'s scores, as it is the standard evaluation library used in Information Retrieval research.
`ranx`'s `recall@k` works as [`trec_eval`](https://github.com/usnistgov/trec_eval)'s `recall@k`.

I was not aware of what you brought to my attention. However, it seems to be known.
From [`PyTerrier`](https://github.com/terrier-org/pyterrier)'s documentation [section about Recall@k](https://pyterrier.readthedocs.io/en/latest/experiments.html#pyterrier.measures.R):
```
Recall@k (R@k). The fraction of relevant documents for a query that have been retrieved by rank k.

NOTE: Some tasks define Recall@k as whether any relevant documents are found in the top k results.
This software follows the TREC convention and refers to that measure as Success@k.
```

[`trec_eval`](https://github.com/usnistgov/trec_eval)'s `Success` measure is called `Hit Rate` in `ranx`.

Let me know if this answers your question, and consider giving `ranx` a star if you like it!

Best,

Elias

Yes you answered my question, thank you very much.
Repository: AmenRa/ranx
Issue: Spearman, Kendall correlation functions 
Root cause: Unimplemented Feature Gap

Hi! This lib is extremely good tool to have in arsenal, but I think it could be nice to have Spearman and Kendall correlation functions included to this package to evaluate ranking. Maybe not the most popular metrics, but sometimes they could come in handy.
Best regards,
Ivan Savchuk

Hi and thanks for the kind words! :D

Kendall, Spearman, Pearson, and AP correlation metrics are already in my local branch and are coming soon!

If you haven't, please consider giving `ranx` a star. 

Best,

Elias

Any update on this?

Hi, sorry for the delay.
I will work on this as soon as I can.

Oh, don't worry. No hurry. I was just curious.

Is this already available?
Repository: AmenRa/ranx
Issue: Problems with MAP
Root cause: Documentation Deficiency

I understood that, when evaluating MAP@k, relevance judgment scores equal to 0 are ignored.
In my case, I get a bit of a weird behaviour.

I'm working on a balanced dataset with binary relevancy and define qrels by including both 1s and 0s documents.
While ndcg@10 gives me results at about 0.7, MAP@10 is extremely low at about 0.10.

Can this be because, besides the very first documents, the model perform poorly or am I doing something wrong when evaluating?

```
qrels = Qrels.from_df(
    df=test_loaded_pdf,
    q_id_col="user_id",
    doc_id_col="run_session_id",
    score_col="target_binary",
)

run = Run.from_df(
    df=test_loaded_pdf,
    q_id_col="user_id",
    doc_id_col="run_session_id",
    score_col="predictions",
)

evaluate(qrels, run, ["map@10", "mrr", "ndcg@10"])
```

predictions in test_loaded_pdf is not a list of binary relevancy but it's a float relevancy score

Hi Stefano,

Almost all the metrics ignore qrels with zero scores, including NDCG.
So the difference you get is not because of that.

However, I think your results are entirely possible if you have many relevance judgments for each query.
Note that the [Average Precision](https://amenra.github.io/ranx/metrics/#mean-average-precision) denominator is equal to the number of relevant documents, regardless of the cut-off.
Conversely, [DCG](https://en.wikipedia.org/wiki/Discounted_cumulative_gain) and, therefore, NDCG depends on the cut-off and not on the number of relevant documents.
_(Please, cross check this for correctness.)_

Here is a toy example:
```python
from ranx import Qrels, Run, evaluate

# 100 relevant docs
qrels = Qrels({
    "q1": {f"d{i}": 1 for i in range(100)}
})

# Only one relevant doc is returned
run = Run({
    "q1": {**{"d1":1000}, **{f"dd{i}": i for i in range(99)}}
})

>>>
{
    "map": 0.01,
    "map@10": 0.01,
    "ndcg@10": 0.22009176629808017,
    "ndcg@100": 0.047758523260819974,
}
```

From my experience, MAP is commonly used with larger cut-offs than 10 (usually 100) or with no cut-off at all.

Best,

Elias

Closing for inactivity.
Feel free to re-open if needed.
Repository: AmenRa/ranx
Issue: [Feature Request] infer run/qrels format from file extension in `from_file`
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I think it’s quite frustrating to have to specify the format of qrels/run (TREC or JSON). I often get exceptions if I forget to specify the 'trec' format because JSON is default.

**Describe the solution you'd like**
You could infer the format from the file extension: if `.json` then JSON else TREC. I can open a PR if you’re interested :)


Hi Paul,

I agree with you, and I thought about it in the past.

However, I plan to have a custom JSON-based file for `Run`s (_in addition to those already supported_) and using the sole file extension would not be enough/safe to infer the input file type.
So, do not bother opening a PR, as your code could be changed soon after.

I will add the requested feature in the upcoming weeks for sure!

Best,

Elias

Added in `0.2.9`.
Works with both `save` and `from_file`.
`kind` now can be used to override the default behaviour.
Repository: AmenRa/ranx
Issue: Qrels and Run query ids do not match
Root cause: Validation Gap

**Describe the bug**
I am evaluating my test set against my algorithm recommendations, but it gives the error
Qrels and Run query ids do not match 

**To Reproduce**
Steps to reproduce the behavior:
metrics = ["recall@10", "mrr@10","ndcg@10"]
person_date_indexs = df_recs_train_top10['person_date_index'].unique()
run = tranform_recs_to_ranx(df_recs_train_top10, person_date_indexs, "person_date_index", "gym_id", "rank")
qrels = tranform_test_to_ranx(df_test_checkins_new_col_renamed, df_recs_train_top10)
evaluate(qrels, run, metrics)

**Expected behavior**
print the recall, mrr and ndcg

**Screenshots**
![image](https://user-images.githubusercontent.com/8708125/190869532-357985ad-f272-436f-bce3-ef9e60ced10d.png)

**Desktop (please complete the following information):**
Ubuntu 22.04

**Additional context**
It worked fine with another test set. The only difference is that in this test set I removed items that the user already interacted in train.


Does your run contains all the elements on the qrel (and vice-versa)? I had this problem many times working with a dataset that had many queries that did not have relevance labels. It can also happen in the opposite sense (although rarer), if you have relevance data for a query that is not evaluated by your system it will also crash (reading your additional context it could be this case if you have the same qrel file for train and test)

Hi @EmotionFever and thank @cadurosar,

As @cadurosar told you, this is not a bug, it is a sanity check to let you know your `qrels` and `run` query ids to not match, thus you have results for queries without relevance judgements or queries without results, or both.

Maybe, you should just remove the queries you removed from `qrels` from the `run` too.

Please let me know if this solve your issue and, if you like it, consider giving `ranx` a star.

Thank you,

Elias

Closing for inactivity. Feel free to re-open.
Repository: AmenRa/ranx
Issue: [Feature Request] relevance_level parameter
Root cause: Unimplemented Feature Gap

I was wondering if similar to trec_eval that we can specify the relevance_level, with -l parameter, this feature exists in ranx. If not that would be a useful feature for evaluation

Dear Behrooz,

Thanks again for your interest in `ranx`! :)

I added the requested feature in `v0.3.3`.
I added two different ways to set relevance levels.

__Qrels-wise__
Qrels now provides the `set_relevance_level` function: `qrels.set_relevance_level(rel_lvl)`.
It changes the document relevance judgments stored in qrels.

__Metric-wise__
The relevance level can now be appended to metric names: `map@100-l2`, `ndcg-l3`.
This way, the original relevance judgments stored in qrels are preserved.

I checked some metric scores against `trec_eval` after setting the relevance level: they seem to work as intended.
I noticed that setting the relevance level for `ndcg` does not work on `trec_eval`.
I will open an issue on its repo to check whether this works as intended.

Let me know if you have further feature requests.

Best,

Elias
Repository: AmenRa/ranx
Issue: [BUG] `MRR@1` is not equal `Recall@1`
Root cause: Documentation Deficiency

**Describe the bug**
`MRR@1` should be equal to `Recall@1`. However, these metrics diverge for the case below.

**To Reproduce**
```python
%%capture
!pip install ranx

from ranx import Qrels, Run, evaluate
import pickle

# download files from https://drive.google.com/drive/folders/1ZLyB6mKKiQsypw36nhdZ4dGqmFw27K-3?usp=sharing
with open("qrels.pkl", "rb") as f:
    qrels = pickle.load(f)
with open("run.pkl", "rb") as f:
    run = pickle.load(f)

evaluate(
    Qrels(qrels),
    Run(run),
    ['mrr@1', 'mrr@5', 'mrr@10', 'recall@1', 'recall@5', 'recall@10'])


# {'mrr@1': 0.8133879123525163,
#  'mrr@5': 0.820242395055783,
#  'mrr@10': 0.8206332007078454,
#  'recall@1': 0.04814167526511499,
#  'recall@5': 0.05089848464127321,
#  'recall@10': 0.05171913427724859}
```
or use [Google Colab](https://colab.research.google.com/drive/1KyR4EV0zBSIvQT_NplNw8lDH5dxx67mh?usp=sharing).

**Expected behavior**
`mrr@1`=`recall@1`

Am I missing something?


MRR@k is only equal to Recall@k if you have exactly one positive per query. Unfortunately, there are many meanings for Recall, but at least for trec eval (which ranx follows) it always uses positives_found/total_positives, even if k is smaller than the amount of positives you have. 

For the other definitions, one I have seen called as R_cap (using positives_found/min(k,total_positives)) or Success ( the function "found any positives @ k").

Recall@1 is equal to MRR@1 only if you have exactly 1 relevant document per query, otherwise they are not as Recall takes into consideration the total number of relevant documents (retrieved + non-retrieved).

You can find the definition of both metrics in ranx's documentation ([Recall](https://amenra.github.io/ranx/metrics/#recall) - [MRR](https://amenra.github.io/ranx/metrics/#mean-reciprocal-rank)) and Wikipedia ([Recall](https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)#Recall) - [MRR](https://en.wikipedia.org/wiki/Mean_reciprocal_rank)).

I see,
Thank you for your answers.
Repository: AmenRa/ranx
Issue: [Feature Request] optimize norm and method in optimize_fusion
GitHub labels: enhancement

**Is your feature request related to a problem? Please describe.**
Instead of manually trying every possible fusion techniques, it’d be nice to be able to grid-search all of them, as `optimize_fusion` is already doing for fusion hyperparameters (e.g. weights in `wsum`).

**Describe the solution you'd like**
Allow to pass `List[str]` instead of `str` for `norm` and `method` parameters.
Then do something like:
```py
for norm in norms:
    for method in methods:
        optimize_fusion_current_behavior(*args, norm=norm, method=method, **kwargs)
```

The trick is to report the results in a readable manner.

I am aware of that, but I think the users should be aware of the normalization strategies and fusion methods they are using.
Therefore, I do not plan to add this feature any time soon.
Repository: AmenRa/ranx
Issue: [BUG] Could not find a version that satisfies the requirement ranx
Root cause: Documentation Deficiency

**Describe the bug**
Could not find a version that satisfies the requirement ranx (`pip3 install ranx`)

```
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.6 LTS
Release:	18.04
Codename:	bionic

pip 21.3.1 from /usr/local/lib/python3.6/dist-packages/pip (python 3.6)
```

Error message:
```
ERROR: Could not find a version that satisfies the requirement ranx (from versions: none)
ERROR: No matching distribution found for ranx
```



This is not a bug, `ranx` requires Python >= 3.7.

> This is not a bug, `ranx` requires Python >= 3.7.

Sorry, I could not find this requirement in the documentation.

No problem.
Python 3.7 is from mid-2018, that's why I did not write it down.
You can find all the requirements and dependencies in [setup.py](https://github.com/AmenRa/ranx/blob/master/setup.py).

> No problem. Python 3.7 is from mid-2018, that's why I did not write it down. You can find all the requirements and dependencies in [setup.py](https://github.com/AmenRa/ranx/blob/master/setup.py).

I understand, but in addition to the ranx, a project is often composed of a set of packages (some requiring a specific Python version). Then, I'd like to suggest completing the documentation with the requirements.