zhiminy/EvalEng huggingface dataset:
Repository: AmenRa/ranx
Issue: Question on rank aggregation usage

Thanks for your amazing work. I am very interested in this framework and try to use it to solve my rank aggregation problems. However, I am a little confused about the usage.

For example, I have scores for several items under different ranking rules, as follows:
```
item | rank1 | rank2 | rank3
item1 | 0.4 | 0.8 | 0.2
item2 | 0.8 | 0.7 | 0.7
item3 | 0.7 | 0.7 | 1.0
item4 | 0.5 | 0.6 | 0.7
```
Now I want a comprehensive rank that satisfies all the subrank (e.g., rank1, rank2, rank3) constraints as much as possible.
```
item | rank
item1 | 0.5
item2 | 0.9
item3 | 0.7
item4 | 0.4
```
I'm not sure if that can be addressed with ranx. If so, could you show me an example?
Thanks a lot.

Hi, you can find an example on [Colab](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/5_fusion.ipynb) and other information [here](https://amenra.github.io/ranx/fusion/).

As for your example, you could to something like this:

```python
from ranx import Run, fuse

run_1 = Run({"q1": {"item1": 0.4, "item2": 0.8, "item3": 0.7, "item4": 0.5}})
run_2 = Run({"q1": {"item1": 0.8, "item2": 0.7, "item3": 0.7, "item4": 0.6}})
run_3 = Run({"q1": {"item1": 0.2, "item2": 0.7, "item3": 1.0, "item4": 0.7}})

combined_run = fuse(runs=[run_1, run_2, run_3], method="max")

print(combined_run.to_dict()["q1"])
```

Output:

```python
{
    "item2": 1.0,
    "item3": 1.0,
    "item1": 1.0,
    "item4": 0.625,
}
```

EDIT: copy-pasted wrong output dict.

Thank you for your reply! it is very helpful and I will try it soon.
By the way, any suggestions for speeding up when the number of items and the subranks (e.g., tens of thousands of items and dozens of subranks) are both large?

Neither the implemented methods nor the implementation itself are meant to be used in the context you describe.
Usually, we aggregate two or three rankings for 1k documents each (the top 1k in each ranking).
Give `ranx` a try for your use case. Maybe use it to find the method that works best with a sample of your data and then build your custom implementation that can run on the GPU.
I do not think you can do much better than my implementation performance-wise, at least not in Python.

Thanks a lot. I got it.
Repository: AmenRa/ranx
Issue: Getting "Segmentation fault (core dumped)" error
Root cause: Environment Incompatibility

Hello,
Thank you for your amazing work. I am trying to use supervised fusion methods like this:
```python
best_params = optimize_fusion(
qrels=qrels,
runs=[run_4, run_5],
norm='min-max',  # Default value
method='posfuse',  
metric="mrr@10",  
)

combined_run = fuse(
runs=[run_4, run_5],
norm='min-max',  # Default value
method='posfuse',  # Alias for Weighted Sum
params=best_params,
)
``` 
I tried changing the norm and metric to every possible value but I still get "Segmentation fault (core dumped)" error.
I could not find any hints in the documentation about. sorry if I am missing something but can you help me with using these fusion methods?
thanks,

Hi and thanks for the kind words!

Can you tell me the OS you are using and the version of `numba` installed?
The issue is probably related to `numba`,  which is a library `ranx` heavily relies on.


At first I was using an Ubuntu remote server (Ubuntu 20.04.6 LTS). I was getting that error for most of the approaches. I changed to macOS and now they are working fine! Thanks!
Repository: AmenRa/ranx
Issue: [Feature Request] stddev statistic
Root cause: Unimplemented Feature Gap

It would be a good to additionally calculate the sample variance within the average of the metrics.

Like `ndcg@50: 11, stddev: 2.8`


Hi! Yes, I should add it.

In the mean time, you can compute it as follows:
```python
from ranx import Qrels, Run, evaluate
import numpy as np

qrels_dict = { "q_1": { "d_12": 5, "d_25": 3 },
               "q_2": { "d_11": 6, "d_22": 1 } }

run_dict = { "q_1": { "d_12": 0.9, "d_23": 0.8, "d_25": 0.7,
                      "d_36": 0.6, "d_32": 0.5, "d_35": 0.4  },
             "q_2": { "d_12": 0.9, "d_11": 0.8, "d_25": 0.7,
                      "d_36": 0.6, "d_22": 0.5, "d_35": 0.4  } }

qrels = Qrels(qrels_dict)
run = Run(run_dict)

evaluate(qrels, run, ["map@5", "mrr"])

print(np.std(list(run.scores["map@5"].values())))
print(np.std(list(run.scores["mrr"].values())))
```

Added support for standard deviation in `v0.3.19`.

Example:
```python
from ranx import Qrels, Run, evaluate

qrels_dict = { "q_1": { "d_12": 5, "d_25": 3 },
               "q_2": { "d_11": 6, "d_22": 1 } }

run_dict = { "q_1": { "d_12": 0.9, "d_23": 0.8, "d_25": 0.7,
                      "d_36": 0.6, "d_32": 0.5, "d_35": 0.4  },
             "q_2": { "d_12": 0.9, "d_11": 0.8, "d_25": 0.7,
                      "d_36": 0.6, "d_22": 0.5, "d_35": 0.4  } }

qrels = Qrels(qrels_dict)
run = Run(run_dict)

evaluate(qrels, run, ["map@5", "mrr"], return_std=True)
```

Output:
```python
{'map@5': {'mean': 0.6416666666666666, 'std': 0.19166666666666662},
 'mrr': {'mean': 0.75, 'std': 0.25}}
```

Metrics standard deviations can later be accessed as follows:
```python
run.std_scores
```

Output:
```python
{'map@5': 0.19166666666666662, 'mrr': 0.25}
```

Please, consider giving `ranx` a star if you haven't yet. :)

@AmenRa thank you!
Repository: AmenRa/ranx
Issue: Couldn't find any documentation about Qrel and run score range
Root cause: Documentation Deficiency

Hi,

In the current api documentation I see some examples of how to structure `qrel` and `run` dicts, but there is no detail about what should be the range of `qrel` relevance score, I assume its 0 for non-relevant cases and 1 or more for graded relevance, but is there a upper limit to integer value? 

Similarly, for `run` score which is a float type, I assume the range is 0.0 to 1.0? 

Thank you.




Hi and thanks for your question.

`qrels` scores can be any integer, positive or negative.
All the implemented metrics but [`bpref`](https://amenra.github.io/ranx/metrics/#bpref) only considers relevant documents, i.e. discard all `qrels` < 1. You can change this threshold by adding `-ln` (`n` is a parameter) to a metric name when using `evaluate` and `compare` methods (e.g., `recall-l2` means that all `qrels` < 2 are considered as non-relevant when computing the `recall` score).
All the implemented metrics but `DCG` and `nDCG` consider relevance to be binary, i.e., graded scores are not taken into account.

`run` scores can be any float, positive or negative. They are used only for sorting as they are model-dependent.

Let me know if you have further questions.
Please consider giving `ranx` a star if you like it.

Thank you very much.
Repository: AmenRa/ranx
Issue: question: Comparing models with multiple runs
Root cause: Unimplemented Feature Gap

First of all, great work on this code. I have been looking for a definitive package to evaluate ranking models and I believe this is that package.

My question is perhaps a bit out of the domain, but it could help others in the future. How would you deal with comparing 2 models where each has multiple runs (e.g., runs with different random initialization and/or batch shuffling, for confidence intervals). I was thinking that perhaps the significance testing could be performed between the mean (across runs) metric_scores vectors.

Thanks in advance,

Tiago

Hi and thanks for the kind words! :)

That's a tricky question.
I actually don't have a _scientific_ answer, but your idea sounds reasonable to me.

Right now `ranx` does not support what you say out of the box but it can probably be achieved using the "low level" function you find in the source code.

An alternative could be to consider every variation of your model separately, run the comparison and then do something like "3 out of 4 model variations significantly improved over the baseline(s) so we can say that, generally, the model is better".

It would be safer to say that only if _all_ your model variations improved over the baseline(s) to be honest...

You should talk with someone more experienced than me to find a scientific answer to your problem.

If you do, please, keep me posted!

Elias

Thank you for the quick response!

Surprisingly haven't been able to find any discussions on this subject specifically, but have no doubt that they must exist :).
Your alternative is definitely the safest approach, but testing all combinations for a large number of models and runs can become pretty computationally expensive. 

If I come across the _scientific_ answer to this I will definitely update you.

Best regards,

Tiago

Closing. Feel free to re-open if needed.
Repository: AmenRa/ranx
Issue: How do we compare different runs with multiple folds per run?

How do we compare different runs with multiple folds per run?
For instance, assume we have 10-folds for `run_1`, ... `run_5`?

```python
from ranx import compare

# Compare different runs and perform Two-sided Paired Student's t-Test
report = compare(
    qrels=qrels,
    runs=[run_1, run_2, run_3, run_4, run_5],
    metrics=["map@100", "mrr@100", "ndcg@10"],
    max_p=0.01  # P-value threshold
)
```

Also, could you provide a simple explanation of how to interpret the report?
```md
#    Model    MAP@100    MRR@100    NDCG@10
---  -------  --------   --------   ---------
a    model_1  0.320ᵇ     0.320ᵇ     0.368ᵇᶜ
b    model_2  0.233      0.234      0.239
c    model_3  0.308ᵇ     0.309ᵇ     0.330ᵇ
d    model_4  0.366ᵃᵇᶜ   0.367ᵃᵇᶜ   0.408ᵃᵇᶜ
e    model_5  0.405ᵃᵇᶜᵈ  0.406ᵃᵇᶜᵈ  0.451ᵃᵇᶜᵈ
```
Repository: AmenRa/ranx
Issue: [Question] About the correction among multiple hypotheses
Root cause: Documentation Deficiency

I came across a point in the [Fuhr](https://www.sigir.org/wp-content/uploads/2018/01/p032.pdf) article where it talks about the need for additional correction for multiple hypotheses test (e.g., Bonferroni correction). Looking at the `paired_student_t_test` function code, it seems like the correction procedure is not taken into account.

Am I on the right track with this? If the current version does indeed consider this, could you guide me on how to verify it?

Hi and sorry for the delay.

You can apply Bonferroni correction by changing the `max_p` parameter of the `compare` function (Maximum p-value to consider an increment as statistically significant. Defaults to 0.01.).

Otherwise, you can use the [Tukey's HSD Test](https://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm), as suggested by Fuhr in the same article, by passing `Tukey` as `stat_test` parameter of `compare`.

Closing.
Repository: AmenRa/ranx
Issue: [Question] How to compute precision for a retriever operating at passage-level
Root cause: Documentation Deficiency

If the retriever is operating at passage / chunk-level and hence the retrieved results can have duplicate IDs as shown below:
Top-10 retrieval list for each of the queries:
```
q_1: ['d_1', 'd_1', 'd_1', 'd_1', 'd_1', 'd_1', 'd_5', 'd_5', 'd_5', 'd_5']
q_2: ['d_4', 'd_4', 'd_4', 'd_4', 'd_2', 'd_2', 'd_6', 'd_6', 'd_6', 'd_6']
```

Encoding them into a dictionary results in:
```
run_dict = { "q_1": { "d_1": 0.9, "d_5": 0.8 }, 
             "q_2": { "d_4": 0.9, "d_2": 0.8, "d_6": 0.7 } }
```

where qrels could be:
```
qrels_dict = { "q_1": { "d_1": 5, "d_5": 3 },
               "q_2": { "d_4": 6, "d_6": 1 } }
```

Considering above scenario, the `precision@10` would yield very low score, even though it is not the case. How can we fix this issue?

Hi and sorry for the delay.

I suggest you use ids for passages OR perform a pooling operation on the scores for the passages for a given document, e.g. taking the maximum scores.

Hope it helps.
Repository: AmenRa/ranx
Issue: JIT compilation on serverless (i.e. Modal Labs)
Root cause: Documentation Deficiency

I'm noticing that it takes a really long time to start up anything with ranx in a serverless setting, e.g. Modal Labs. I tried adding a step during the build process where I call the functions that need to be cached, but it doesn't seem to help. This is true even if I explicitly set the NUMBA_CACHE_DIR to a directory that will persist from build -> deployment. Do you have advice on how to avoid the JIT slow start in this setting?

Hi and sorry for the delay.
Unfortunately, I cannot help with this.
If you find a solution, please post it here for others.
Thanks.

@andersonbcdefg @AmenRa please check https://github.com/AmenRa/ranx/issues/75
Repository: AmenRa/ranx
Issue: [BUG] Run with one empty results error
Root cause: Validation Gap

**Describe the bug**
A Run with only one empty results lead to ValueError
May related to the issue here: https://github.com/AmenRa/ranx/issues/51  

**To Reproduce**
run = Run({"q_1": {}})   
qrel = Qrels({"q_1": { 'reference_document': 1.0}})
Repository: AmenRa/ranx
Issue: Fisher's Randomization Test yields nondeterministic results
Root cause: Algorithmic Error

**Describe the bug**
When running Fisher's Randomization Test multiple times using the same data and same parameters, the significance results vary between runs. This happens even when the random seed is fixed and also when just a single thread is used.

**To Reproduce**
Run `ranx.compare(qrels, runs, metrics=["precision@1", "recall@20"], stat_test="fisher", max_p=0.05, n_permutations=1000, make_comparable=True, threads=1, random_seed=0)`.
`qrels` has a few thousand entries, each element in `runs` about a thousand entries.

When running `compare()` multiple times, the significance assessments are slightly different between runs. On my data, this can be observed after three to five `compare()` runs.

**Expected behavior**
I expect multiple `compare()` runs on the same data and with the same parameters to always show the exact same significance results.
Repository: AmenRa/ranx
Issue: [Feature Request] Support for non-integer relevance labels
Root cause: Unimplemented Feature Gap

I would like to be able to use real-valued labels, and compute DCG and NDCG metrics that respect the differences in labels that at less than 1. So for example:
- A document with relevance label = 1 would have Burges gain of `2**1 - 1 = 1.0`
- But a document with relevance label = 1.9 would have Burges gain of `2**1.9 - 1 = 2.73`

However, it seems like Ranx's `Qrels` object will cast my relevance labels to integers:
```python
from ranx import Qrels

qrels_dict = {"q_1": {"d_11": 1.9}}
qrels = Qrels(qrels_dict)
print(qrels)

>>> {q_1: {d_11: 1}}
```

I believe this happens in the Qrels constructor [here](https://github.com/AmenRa/ranx/blob/f60b1ac230ab97bd3c6ad2c210f79882293b306d/ranx/data_structures/qrels.py#L82). A consequence of this is that I can't properly use real-valued labels. The relevance values will be truncated, and a lot of information will be lost.

Questions for the maintainers:

1. Am I missing something? Is there a way to achieve what I want with Ranx?
2. If not, what is required to make Ranx support this? Is this a big lift or a small one?

+1 for this.  Continuous labels are produced from clickstream data, and may be used in some of these metrics.
Repository: AmenRa/ranx
Issue: Integer relevance score
Root cause: Architectural Constraint

Hi,

Thank you for this nice library.

Is there a fundamental reason why relevance score need to be integers in Qrels?

Thanks.

Hi and thank you! :)

In Information Retrieval, relevance score judgments can be binary or graded.
To the best of my knowledge, graded relevance score judgments are usually expressed as integers.

Moreover, if I recall correctly, the choice of requiring integers was taken for similarity with [`trec_eval`](https://github.com/usnistgov/trec_eval), the standard evaluation library in Information Retrieval.

Do you need`ranx` to support floating point relevance score judgments?

Best,

Elias

Thank make sense! No no, it was just something I was wondering about :)

Following up on this, I'd now need `ranx` to support floating point relevance score judgments?
Is that supported right now? Or is there a turnaround you'd recommend to use floating relevance score?

Best,

Stefano

Hi Stefano,

I apologize if my question sounds stupid: are your relevance judgments truly continuous, or are they discrete but defined with floats?

In the latter case, you should always be able to convert them to integers by simply re-scaling them.

If that's not the case, you could, for example, round your floats at the 4th decimal position, multiply them by 10^4, and then cast them to integers.

If this still does not solve your problem, could you please provide more information about your use case so I can better help you find a solution?

I do not plan to change the implementation of Qrels at the moment.

Best,

Elias
Repository: Cloud-CV/EvalAI
Issue: Create end point to get all the challenge host of a particular challenge host team

Create end point to get all the challenge host of a particular challenge host team

Details about the endpoint:

Url: /api/hosts/challenge_host_team/<challenge_host_team_id>/challenge_host
Header: Authorization: Token
Method: GET
Response:
status: HTTP_200_OK

Support for pagination should be there.

Fixed via #110
Repository: Cloud-CV/EvalAI
Issue: Replace %s with string format {} across all str functions of models
GitHub labels: easy_to_fix

backend

This is a pretty simple task. Some part of the code base contains `%s` function(used for string formatting). It involves replacing all the occurrences of such `%s` with string format function `{}` everywhere in models of each app.

For example one such case can be found in `challenges/models.py`, __LeaderBoard__ model.
The function right now looks like 
```
def __unicode__(self):
        return "%s" % (self.id)
```

After replacing `%s` with string format function, the above function will looks something like

```
def __unicode__(self):
    return '{}'.format(self.id)
```

Note: Only replace in models of each app, not across the whole project as of now.

I would be happy to help! Is anyone working on this?

No, I don't think so. Please go ahead. :) 

Okay, on it. Why are we doing it? Is the latter method more efficient than the other?

For more reference, please see this http://stackoverflow.com/questions/5082452/python-string-formatting-vs-format

Understood. Thanks. :+1:
Repository: Cloud-CV/EvalAI
Issue: Remove unused images from frontend directory
GitHub labels: enhancement

easy_to_fix

frontend

critical

There are a lot of images that are not being used at all and are there in the repository. Can we please get rid of those images? 

Sure @deshraj, will do it.

@deshraj @aka-jain mind if I take this up?

Sure @the-dagger. 

Can I take this up?

@xamfy sure. :) 


@deshraj  following are the images that should be removed:
```
challenge-preview.png
cloud-4.gif
cloud3.png
dash-challenge.png
owl.png
plane.png
rock.png
rocket.png
sat.png
sate.png
star.gif
```
Am I right?
Repository: Cloud-CV/EvalAI
Issue: Write integration tests for submission worker
Root cause: Unimplemented Feature Gap

Submission worker: Runs as a individual python process, has dependency on RabbitMQ and the main app server.

### How ??

I was thinking of starting submission worker via subprocess and then killing it later on via `process.kill`. This way the worker will start listening on queue and then we can publish a message which will be acted upon by the worker.

Also the only way to check if the submission has passed through the worker flow is to check its status in db. It should be either SUCCESS or FAILURE and nothing else.

@deshraj @dexter1691 : your thought here please

@deshraj : let me know if you have any development here ?

@TroJan sorry I have been busy with working on other stuff and hence didn't get time to work on this one. I am afraid that I won't be able to work on this for next 3 days. I will let you know once I have something. 

@deshraj @dexter1691 : let me know your thoughts please :)

I am not sure how this should be done exactly. I need some more time. Also, @RishabhJain2018 @TroJan please suggest if you have other thoughts about how to do this? 

@deshraj Is this issue still active?

@isht3 Yes, but can you please discuss the strategy here before implementing it.

@RishabhJain2018 I've specified two methods of implementation in #1457, could you give me feedback?

@deshraj @RishabhJain2018 : Can we close if not required?

Thanks @taranjeet for pointing it out. Closing it.
Repository: Cloud-CV/EvalAI
Issue: Change path of storage of Evaluation Scripts
Root cause: Architectural Constraint

## Current behaviour

Currently, the evaluation scripts are stored in the `submission_files` directory which should not happen ideally.

## Solution
Modify the `RandomFile()` method such that the evaluation scripts should be uploaded in the `evaluation_scripts` directory instead of the `submission_files` directory. 


@deshraj : Why is this required ?

The `RandomFile()` definition sets the path of the uploaded file to be `submission_files/blah` and hence the evaluation scripts being uploaded are also stored in the submission_files. I hope I am clear now.
Repository: Cloud-CV/EvalAI
Issue: Notification toasts misplaced on the auth pages
GitHub labels: bug

easy_to_fix

frontend

On the new Signup and login page, the notification toast is misaligned and is not shown at the correct position. 

@deshraj I would like to work on this, could you please explain a bit more on what the issue is?

@vedantrathore it seems like this has already been fixed by @aka-jain. Please feel free to take up any other issue. :)
Repository: Cloud-CV/EvalAI
Issue: Auth form input field sizes are contracted.
Root cause: Documentation Deficiency

Auth pages like login, signup etc have contracted input field. Need to expand them.

Hey @aka-jain has this been resolved? I'd love to take it if it hasn't yet. 

I think this is already resolved. Isn't it @aka-jain ?

@aka-jain @deshraj Is this still an issue ? I think it is resolved.

I am not sure. @aka-jain can you please reply to this thread? 

This issue is resolved. Closing this.
Repository: Cloud-CV/EvalAI
Issue: Change the default remote for reviewing pull requests
GitHub labels: enhancement

easy_to_fix

Currently, the default remote is set to `upstream` for reviewing pull requests. We would ideally want to set it to `origin` by default since this is mainly used by the project maintainers. 

@deshraj : this is done I suppose. Can we close it ?

Sure. Fixed in #1020
Repository: Cloud-CV/EvalAI
Issue: Create endpoint to update a particular ChallengeHost details

Details about the endpoint:

Url: /api/hosts/challenge_host_team/<challenge_host_team_id>/challenge_host/<challenge_host_id>
Header: Authorization: Token
Method: PUT/PATCH
Response:
status on success: HTTP_200_OK
status on exception: HTTP_406_NOT_ACCEPTABLE
status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

Fixed in #116
Repository: Cloud-CV/EvalAI
Issue: Create endpoint to delete a ChallengeHost

Details about the endpoint:

Url: /api/hosts/challenge_host_team/<challenge_host_team_id>/challenge_host/<challenge_host_id>
Header: Authorization: Token
Method: DELETE
Response:
status on success: HTTP_200_OK
status on exception: HTTP_406_NOT_ACCEPTABLE
status on Unauthorized access: HTTP_203_NON_AUTHORITATIVE_INFORMATION

Merged via #117
Repository: Cloud-CV/EvalAI
Issue: Add models for creating challenge using zip file.
Root cause: Unimplemented Feature Gap

The models must be defined in `apps/challenges/models.py`.

**Fields:**
- user: ForeignKey to User
- challenge : OneToOneField to Challenge
- zip_configuration: FileField
- is_created: Boolean field
- stdout_file: Filefield
- stderr_file: Filefield