zhiminy/EvalEng huggingface dataset:
Repository: AmenRa/ranx
Issue: ValueError: max() arg is an empty sequence
Root cause: Validation Gap

Hello,
I'd like to determine what query is causing the following error and how to get around it: 
```python
Traceback (most recent call last):
  File "main.py", line 43, in perform_tasks
    eval(params)
  File "main.py", line 25, in eval
    eval_helper.perform_eval()
  File "/home/celso/projects/XMTC-Baselines/source/helper/EvalHelper.py", line 62, in perform_eval
    qrels = Qrels(filtered_relevance_map)
  File "/home/celso/projects/venvs/XMTC-Baselines/lib/python3.8/site-packages/ranx/data_structures/qrels.py", line 62, in __init__
    max_len = max(len(y) for x in doc_ids for y in x)
ValueError: max() arg is an empty sequence
```

My evaluation code is shown in the code snippet below.
```python
ranking = self._retrieve(...)
filtered_relevance_map= {key: value for key, value in self.relevance_map.items() if key in ranking.keys()}
qrels = Qrels(filtered_relevance_map)
run = Run(ranking, name=cls)
result = evaluate(qrels, run, self.metrics, threads=12)
```

I suspect your `filtered_relevance_map` is not valid.
Can you please post a print of it?

> I suspect your `filtered_relevance_map` is not valid. Can you please post a print of it?
`filtered_relevance_map`:
![image](https://user-images.githubusercontent.com/11181748/212191503-35972428-772f-441f-a075-7d941b7f3eba.png)

`ranking`:
![image](https://user-images.githubusercontent.com/11181748/212191653-a78c1af8-1a49-4639-9a40-73b9c99e9c19.png)



What the code does is simply this:
```python
doc_ids = [list(doc.keys()) for doc in qrels.values()]
max_len = max(len(y) for x in doc_ids for y in x)
```
The code finds the max string length of the doc ids in the dictionary to reduce memory consumption.

You should be able to find the issue by checking what happens with your `filtered_relevance_map` in place of `qrels`.

Please, let me what you find out.

PS: `doc` variable name in the list comprehension is probably a wrong name. It should be `q` or `query`. 

> What the code does is simply this:
> 
> ```python
> doc_ids = [list(doc.keys()) for doc in qrels.values()]
> max_len = max(len(y) for x in doc_ids for y in x)
> ```
> 
> The code finds the max string length of the doc ids in the dictionary to reduce memory consumption.
> 
> You should be able to find the issue by checking what happens with your `filtered_relevance_map` in place of `qrels`.
> 
> Please, let me what you find out.

I can't see any problem since I just replaced `doc` with `label` and `query`  with `text` and concatenated it with their ids.

I can't help you without reproducible code.
The best I can do is guessing.
I never encountered such a problem.

The problem should lay in this line:
```python
doc_ids = [list(doc.keys()) for doc in qrels.values()]
```

`doc.keys()` is probably empty. That should be why the next line throws the exception:

```python
ValueError: max() arg is an empty sequence
```

Have you checked if those two lines of code (isolated from `ranx`) raise the exception?



Closing for inactivity.
Repository: AmenRa/ranx
Issue: question: why student rather than fisher stat test?

Hi,

Just a quick question: I wondered what motivated the choice for changing the default value of the stat test to student instead of fisher https://github.com/AmenRa/ranx/commit/0dc8d9cf6b76ece0c8bccad267e2ce4f226127c3 (I almost published them as is before figuring it out :sweat_smile:).

I thought that one of your documentation pointed me to [this paper](https://dl.acm.org/doi/abs/10.1145/1321440.1321528) of Smucker et al. that suggests using Fisher (and especially not Student), but maybe I don’t recall correctly.

Btw, the docstring still shows 'fisher' as default https://amenra.github.io/ranx/compare/

Hey, I got a pull request for fixing the docs today :D

Student's and Fisher's tests very often agree (always?), but Student's is immensely faster to compute.
That's why I changed the default stat test.

The paper you mentioned says the opposite. From the introduction:

```
Student’s t, bootstrap, and randomization tests largely agree with each other.
Researchers using any of these three tests are likely to draw the same
conclusions regarding statistical significance of their results.
```

There's also a nice table showing that:
![Screenshot 2023-02-03 at 12 13 06](https://user-images.githubusercontent.com/11797097/216589561-58e1387f-2474-4664-9457-27b500cac368.png)


This is the ‘practical’ conclusion with their experiments on TREC data. They have further discussion in §5.2 and in the end recommend to use Fisher’s randomization.

Thank you for your quick answer (as always :))

Sorry, I completely forgot about that section.

From my experiments on very different datasets from those used in the paper, Student's and Fisher's tests always agreed (comparing mean values).

I see the point about tiny test sets.
However, I question the validity of tiny test sets.
Are they representative of the general user behavior?
What's the confidence that a model outperforming another on a small set of queries will work better on different ones?
Is a small set of queries representative of the actual population of queries in the real world?

Obviously, we could be interested in testing a specific niche of queries depending on the use case.

What do you think about it?

It makes sense but I am not so familiar with statistical significance testing.
But I think that small test sets will not get low p-values anyway. 
The opposite problem, namely that huge datasets will get low p-values very easily, is discussed in §4.4 of https://www.morganclaypool.com/doi/abs/10.2200/S00994ED1V01Y202002HLT045
Repository: AmenRa/ranx
Issue: [Feature Request] Add interpolated recall-precision plot function
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
First of all: This is a really nice library! It helps a lot!
My request is regarding a recall-precision graphic. When I read TREC-related papers, they very often used the interpolated precision-recall-plot to visualize the performance of IR-Systems which are being compared to each other. They also use the graphs to understand which IR-System yields a higher recall value, shows a higher precision value, and generally has a better performance.

**Describe the solution you'd like**
Since I love using this library, it would be great, if there were a function for generating such a precision-recall-plot or just an example/notebook for generating this plot using the ranx library in the documentation.

**Describe alternatives you've considered**
I've already considered creating these plots myself with the functions that the library offers. However, it seems quite complex as the interpolation greatly complicates this work for me.

**Additional context**
Here is an example interpolated recall-precision plot:
![image](https://user-images.githubusercontent.com/64859371/218447746-665b15fb-5962-41bd-aa82-2a328369911b.png)

More information can be found on:
https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html
and 
https://trec.nist.gov/pubs/trec15/appendices/CE.MEASURES06.pdf (p.4)



Hi, thanks for the very detailed suggestion/request!
I think this would be a nice addition to the library!
I will take a look to the linked documents in the coming days.

I found a really nice lecture on youtube, which explains the recall-precision plot really nice. It can be found here:

https://youtu.be/x0SkRpIVMUs?t=1066

I also found out, that the official trec_eval program (https://github.com/usnistgov/trec_eval) returns the values for creating this plot. My workaround is actually, that i create runs using ranx, export the runs to a .trec-file and use the official trec_eval program

Hi, sorry for the delay.
I finally have a working implementation for computing the graph!
I still have to figure out a couple of usability things.
You can expect to be able to compute the Precision-Recall graph with `ranx` soon.

Hi, I added the Interpolated Precision-Recall Curve in `v0.3.8`.
I tested my implementation against [trec_eval](https://github.com/usnistgov/trec_eval) for correctness.
You can find a usage example [here](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/7_plot.ipynb).

Please, let me know if you have any comment / suggestion.
Repository: AmenRa/ranx
Issue: [BUG] Missing results causes AssertionError
Root cause: Validation Gap

Missing results causes AssertionError: Qrels and Run query ids do not match
 You should make `check_keys` optional in evaluate because sometimes queries do not return any results for lexical-based systems.

https://github.com/AmenRa/ranx/blob/da0aa5246513acb87ec7ac7f516194fea464a2c1/ranx/meta/evaluate.py#L128

Hi, I've added the `make_comparable` function to `Run` in v0.3.7.

```python
run.make_comparable(qrels)
```

The `run` will be modified based on the `qrels` passed as an argument.
Query results not appearing in `qrels` will be removed from `run` to avoid unpredictable behavior or other bugs.
For convenience, I also added the boolean parameter  `make_comparable`  to evaluate and compare (it defaults to `False`).

Please, consider giving `ranx` a star if you like it.

Best,

Elias
Repository: AmenRa/ranx
Issue: [Feature Request] Expose DCG as metric
Root cause: Unimplemented Feature Gap


In industry DCG (in both formulations) is a standard and widely used metric. I see it is already implemented as part of NDCG. Is it possible to expose it to the users as a real metric?




Hi, `dcg` is now available among the other metrics.
Please, update your `ranx` installation to `v0.3.13` to use it.

If you use `ranx` to evaluate results for your scientific publication, please remember to cite the related papers. Also, `GitHub` stars are much appreciated.

Thank you,

Elias

Thank you for the quick response

You're welcome! ☺️
Repository: AmenRa/ranx
Issue: [BUG] dcg and dcg_burges do not work in the compare function
Root cause: Validation Gap

**Describe the bug**

when adding the newly available `dcg` or `dcg_burges` metric in the compare function I get this error:

```
report = compare(
    qrels=qrels,
    runs=runs,
    metrics=["recall@10","ndcg","rbp.90","rbp.50","dcg_burges"],
    max_p=0.05,   # P-value threshold
    stat_test='student'
)
```

```
Traceback (most recent call last):
  File "/local/home/mkp/data/gap2kic/eval/./run.py", line 32, in <module>
    print(report)
  File "/home/mkp/.asdf/installs/python/3.10.9/lib/python3.10/site-packages/ranx/data_structures/report.py", line 338, in __str__
    return self.to_table()
  File "/home/mkp/.asdf/installs/python/3.10.9/lib/python3.10/site-packages/ranx/data_structures/report.py", line 143, in to_table
    label = self.get_metric_label(x)
  File "/home/mkp/.asdf/installs/python/3.10.9/lib/python3.10/site-packages/ranx/data_structures/report.py", line 122, in get_metric_label
    return f"{metric_labels[m]}"
KeyError: 'dcg_burges'
```

however in the same file when I do 

```
res = evaluate(qrels, run, ["recall@10","ndcg","rbp.90","rbp.50","dcg","dcg_burges"])
```

everything works as intended.



I think metric labels are missing here: https://github.com/AmenRa/ranx/blob/cf7b00c829a223c988c080ab39562637527346ca/ranx/data_structures/report.py#L24C38-L24C38

Yep, I forget about them. Thanks for pointing it out!
Fixed in `v0.3.14`
Repository: AmenRa/ranx
Issue: [Feature Request] Use black to indent the code
GitHub labels: enhancement

**Is your feature request related to a problem? Please describe.**

`black` https://pypi.org/project/black/  is the Uncompromising Code Formatter. You can install it, run `black` and it will format the code properly. How about using it for the project in order to make sure style is consistent? 

Hi Diego, the code was already `black` formatted.

The inconsistency was because I initially forced 80 chars line length in my IDE.
Then, I reverted it to 88 as suggested in `black` documentation but forgot to run `black` project-wise.

Now code formatting should be consistent.
Repository: AmenRa/ranx
Issue: [BUG] RBP with multiple relevance levels
Root cause: Algorithmic Error

**Describe the bug**

Given the formula of RBP of

```
    \operatorname{RBP} = (1 - p) \cdot \sum_{i=1}^{d}{r_i \cdot p^{i - 1}}
```

where r_i is the reward/utility, RBP should support multiple relevance levels similar to DCG such that if max relevance level is 2 max rbp value should be 2


Hi, thanks for pointing it out.

I suspect I assumed it was a binary relevance metric because the example in the original paper does not account for graded relevance.

I will fix it in the coming days.

Looking at other implementations of RBP I also saw an option to normalize back to [0,1] by dividing by the max relevance level in the qrls. 

So if max relevance level RBP in the limit goes to 2. So we divide by 2 to make it go back to [0,1]. Not sure if this is useful to implement or not. 


Fixed in `v0.3.15`.

`RBP` now uses graded relevance scores from `qrels` instead considering them binary.
Do you think I should leave an option for the old behaviour?
Repository: AmenRa/ranx
Issue: [Feature Request] Support gzipped files? 
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
trec files can be several megabytes, for example `run_*.trec` used for the examples are all more than 20Mb, but once compressed they become less than 10. That would make downloads faster and also loading the file in memory. 

**Describe the solution you'd like**
support `*.trec.gz`.  

**Describe alternatives you've considered**
It would cool to evaluate also alternative formats to store the trec file, like [parquet] (https://arrow.apache.org/docs/python/parquet.html), this library focus on computing metrics fast, but if you spend ages to load/parse the trec file it is not very useful - parquet is much faster to load in memory and it is supports compression natively. 

**Additional context**
💨 

Both the suggestions sound good to me.

You can already save imported runs as highly compressed as `lz4` files with `run.save("save/path/run.lz4")`. Under the hood they are `JSON` files.
I did several tests a few months ago and they should be smaller and faster to extract than gzipped TREC files.

I'll let you know when the request features are available. 🍻

Added support for gzipped TREC files in `v0.3.15`.

Added support for Parquet files in `v0.3.16`.
Repository: AmenRa/ranx
Issue: [Feature Request] memory issue / make Run more efficient
Root cause: Resource Mishandling

Hi Elias,

**Is your feature request related to a problem? Please describe.**
I've noticed that `Run` (and I guess also `Qrels`) consume a lot of memory (RAM) compared to standard python `dict`, e.g. a few GB instead of a few 100s of MB. This gets problematic for somewhat large datasets (e.g. 1M queries)

**Describe the solution you'd like**
I guess it's related to Numba representation? I've no clue on how to make it more efficient, sorry :)

**Reproduce**
Just open your system monitor and see how the memory grows.

```py
In [1]: import ranx
# this weighs only a few 100s of MB
In [2]: run_d = {str(i): {str(j): 0.0 for j in range(100)} for i in range(100000)}
# this grows to a few GB
In [3]: run_r = ranx.Run(run_d)
```


Best,

Paul

Dear Paul, 

The issue is probably due to `numba` (I cannot do anything about that) and a [forced conversion](https://github.com/AmenRa/ranx/blob/7a55106c238d7e147ceb93d0201285b946ef366e/ranx/data_structures/common.py#L53) of the strings used as ids to `numba.types.unicode_type` that I introduced to avoid errors when I implemented the fusion algorithms.

I have tested a snippet similar to yours (I do not save the Python dict in memory) with and without the conversion.
Memory usage went down from 2.42 GB to 1.59 GB (including `ranx` import).
The Python dict alone is around 1.15 GB.

I'll try to remove the forced conversion without breaking the fusion algorithms and get back to you.

Thanks for pointing it out!

Fixed in `v0.3.15`.
Repository: AmenRa/ranx
Issue: question: relevance scores in Qrels
Root cause: Documentation Deficiency

Why is the relevance score mandatory in [Qrels](https://github.com/AmenRa/ranx/blob/master/ranx/qrels.py)?

I don’t see where you use it to compute the metrics https://github.com/AmenRa/ranx/blob/master/ranx/metrics.py

Is there any way to make it optional? If not, would a filler value like `0` for all documents be appropriate?

Hi, if you only have a list of relevant document ids but no relevance judgment, you should use `1` as relevance judgment score.

`0` relevance judgment scores are removed before computing metric scores to prevent miscomputations.

Among the implemented metrics only NDCG uses the relevance judgment scores. You can see it [here](https://github.com/AmenRa/ranx/blob/ab1467230e7bf658e8c335a291d9892dd052b7ca/ranx/metrics.py#L214).

By definition, all the other metrics consider relevance judgment scores to be binary even if they are not.

Got it, thanks !
Repository: AmenRa/ranx
Issue: [BUG] Misleading exception message on dataframe types 
Root cause: Validation Gap

**Describe the bug**
I'm using the library for the first time with a Pandas dataframe and ran into an exception that was misleading.

**To Reproduce**
Steps to reproduce the behavior:
1. Create a dataframe where the `id` column is of type `int64`  e.g. `df['id'] = df.index + 1`
2. Create the qrel like this: 
```
qrels = Qrels.from_df(
    df=df,
    q_id_col="id",
    doc_id_col="best_document",
    score_col="score",
)
```
3. Observe this error:
```
[/usr/local/lib/python3.10/dist-packages/ranx/data_structures/qrels.py](https://localhost:8080/#) in from_df(df, q_id_col, doc_id_col, score_col)
    293         """
    294         assert (
--> 295             df[q_id_col].dtype == "O"
    296         ), "DataFrame scores column dtype must be `object` (string)"
    297         assert (

AssertionError: DataFrame scores column dtype must be `object` (string)
```

**Expected behavior**
The assertion message should point to the correct column, in this case, it is the ID column that is of the wrong type. From inspecting the code, the assertion message is wrong when the document ID column is of the wrong type as well.



Hi, sorry for that!
I probably copy-pasted or duplicated lines there.
I will fix it in the next release.

Fixed in `v0.3.17`.

Please, give `ranx` a star if you haven't yet.
Repository: AmenRa/ranx
Issue: [BUG] Issues when storing/loading Qrels from a dataframe and a parquet file.
Root cause: Validation Gap

**Describe the bug**
Bug when reconstructing Qrels from a pandas dataframe. This bug affects also when reading a Qrel from a parquet file as the pandas to Qrels is used internally.

Pandas version: 1.5.2 
Ranx: last pip version

**To Reproduce**
Code: 
```python
from ranx import Qrels
qrels = Qrels({'1':{'1':1}, '2':{'2': 1}})
df = qrels.to_dataframe()
Qrels.from_df(df)
```
Output:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "...\lib\site-packages\ranx\data_structures\qrels.py", line 300, in from_df
    assert df[score_col].dtype == int, "DataFrame scores column dtype must be `int`"
AssertionError: DataFrame scores column dtype must be `int`
```

**About the dataframe dtypes**
It is using int64 instead of int
```python
>>> df.dtypes
q_id      object
doc_id    object
score      int64
dtype: object

>>> import numpy as np
>>> df.dtypes['score'] == np.int64
True
```



Hi, thanks for reporting this issue.
I will fix it in the next release.

Hi, I cannot reproduce.

`int64` is standard Python `int`.

With `ranx`:
```python
from ranx import Qrels

qrels = Qrels({"1": {"1": 1}, "2": {"2": 1}})
df = qrels.to_dataframe()

df.dtypes["score"] == int
```
Output:
```bash
True
True
```

With `Pandas`, to be sure there is no `dtype` conversion:
```python
from pandas import DataFrame

df = DataFrame.from_dict({"q_id": ["1", "2"], "doc_id": ["1", "2"], "score": [1, 1]})

df.dtypes["score"] == int
```
Output:
```bash
True
True
```

Hi,

I looked more into the problem, and the issue is most likely related to the operating system. 

I run the following code on different machines to which I have access.
```
import pandas as pd

df = pd.DataFrame(data=[[1,2],[2,3]], columns=['a', 'b'])

df.dtypes['a'] == int
```
and the output changes with the OS.
For Linux & MacOS: true
For Windows: false

All the installations are based on a conda environment.
That might be why you cannot reproduce the bug.

The numpy behaviour of dtype is OS dependent: https://numpy.org/doc/stable/user/basics.creation.html


Sorry to hear that.
Yes, I work only with MacOS and Linux machines.
Have you tried using [WSL](https://learn.microsoft.com/en-us/windows/wsl/)?

WSL output is consistent with Linux. So, it is only a windows issue. 
Anyway, I think that testing for np.int64 instead of should work and it is platform independent.

Thank you for answering. I really like your library.



I changed the data type check as you suggested.
Please update to `v0.3.18`.

Thank you.
Repository: AmenRa/ranx
Issue: [Feature Request] Run.from_df and Run.from_parquet does not allow specifying run name
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I'm using Pandas dataframes, and comparing different embedding models. I'd like to be able to name my runs so when I compare them, the report shows something other than `run_1`, `run_2`.

**Describe the solution you'd like**
Allow `name` as a named parameter in `Run.from_df`

**Describe alternatives you've considered**
You can just set the name afterward, e.g.

```python
my_run.name = "My Run"
```

**Additional context**
I guess this is just for consistency, since `Run.from_file` allows you to pass a name.

Added in `v0.3.17`.

Please, give `ranx` a star if you haven't yet.