Repository: beir-cellar/beir
Issue: Is re_ranking implementation wrong?
Root cause: Algorithmic Error

Either my understanding is wrong or the implementation of re-ranking is wrong. The [EvaluateRetrieval.rerank](https://github.com/beir-cellar/beir/blob/c3334fd5b336dba03c5e3e605a82fcfb1bdf667d/beir/retrieval/evaluation.py#L25) looks like:
```
def rerank(self, 
        corpus: Dict[str, Dict[str, str]], 
        queries: Dict[str, str],
        results: Dict[str, Dict[str, float]],
        top_k: int) -> Dict[str, Dict[str, float]]:

    new_corpus = {}

    for query_id in results:
        if len(results[query_id]) > top_k:
            for (doc_id, _) in sorted(results[query_id].items(), key=lambda item: item[1], reverse=True)[:top_k]:
                new_corpus[doc_id] = corpus[doc_id]
        else:
            for doc_id in results[query_id]:
                new_corpus[doc_id] = corpus[doc_id]
                
    return self.retriever.search(new_corpus, queries, top_k, self.score_function)
```

By re-ranking we mean, given the top-k (e.g., 100) retrieved doc for each query, a re-ranker will output scores for the top_k retrieved documents only and then they will be sorted based on the scores. However, according to the above implementation, the re-ranker is actually re-ranking a list of documents that may contain up to `top_k * len(queries)` documents, isn't it? Am I missing something?

@thakur-nandan need your help.

I agree, I noticed the same thing. The search seems to be done using the same pool of documents for each of the queries. The pool of documents is the joint set of the retrieved document for all the queries.

The search method seems completely agnostic to the doc_id associated with a given query and no filtration is done on the retrieved document from the call to search to only keep the doc_ids from the retrieval.

This is pretty important as it would mean the BM25 + Reranking results overstate its actual performance

Have a look at this code example:
https://github.com/beir-cellar/beir/blob/c3334fd5b336dba03c5e3e605a82fcfb1bdf667d/examples/retrieval/evaluation/reranking/evaluate_bm25_ce_reranking.py#L63

It doesn't use the [EvaluateRetrieval.rerank](https://github.com/beir-cellar/beir/blob/c3334fd5b336dba03c5e3e605a82fcfb1bdf667d/beir/retrieval/evaluation.py#L25) method, instead it uses the [Rerank.rerank](https://github.com/beir-cellar/beir/blob/c3334fd5b336dba03c5e3e605a82fcfb1bdf667d/beir/reranking/rerank.py#L14) method.

> Have a look at this code example:
> 
> https://github.com/beir-cellar/beir/blob/c3334fd5b336dba03c5e3e605a82fcfb1bdf667d/examples/retrieval/evaluation/reranking/evaluate_bm25_ce_reranking.py#L63
> 
> It doesn't use the [EvaluateRetrieval.rerank](https://github.com/beir-cellar/beir/blob/c3334fd5b336dba03c5e3e605a82fcfb1bdf667d/beir/retrieval/evaluation.py#L25) method, instead it uses the [Rerank.rerank](https://github.com/beir-cellar/beir/blob/c3334fd5b336dba03c5e3e605a82fcfb1bdf667d/beir/reranking/rerank.py#L14) method.

Ah great, thanks! 👍

That is for the cross-encoder example, but it doesn't affect the example using reranking for a dual-encoder (sentence-bert). It also impacts potential custom implementation that uses the `EvaluateRetrieval` rerank method.

![image](https://user-images.githubusercontent.com/71275712/193583697-c7136cae-485d-4c1a-9b09-7e2107226e00.png)

https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/reranking/evaluate_bm25_sbert_reranking.py#L44