Repository: beir-cellar/beir
Issue: Different results using DenseRetrievalExactSearch & DenseRetrievalParallelExactSearch
Root cause: Algorithmic Error

I'm using `DenseRetrievalExactSearch` & `DenseRetrievalParallelExactSearch` with https://huggingface.co/sentence-transformers/sentence-t5-xxl on SciFact and getting different results. I'm using them in mteb, which just wraps them [here](https://github.com/embeddings-benchmark/mteb/blob/4dabbaf94863e4f78e012481899bf2f7952f263b/mteb/abstasks/AbsTaskRetrieval.py#L47) so I think this is an issue with BEIR.
Did anyone confirm that they produce the same results? cc @NouamaneTazi 


`DenseRetrievalExactSearch`:
```json
{
  "test": {
    "evaluation_time": 787.86,
    "map_at_1": 0.41606,
    "map_at_10": 0.50777,
    "map_at_100": 0.51611,
    "map_at_1000": 0.51655,
    "map_at_3": 0.47967,
    "map_at_5": 0.49714,
    "ndcg_at_1": 0.44,
    "ndcg_at_10": 0.5538,
    "ndcg_at_100": 0.59487,
    "ndcg_at_1000": 0.60719,
    "ndcg_at_3": 0.50475,
    "ndcg_at_5": 0.5294,
    "precision_at_1": 0.44,
    "precision_at_10": 0.078,
    "precision_at_100": 0.01003,
    "precision_at_1000": 0.00111,
    "precision_at_3": 0.20333,
    "precision_at_5": 0.13933,
    "recall_at_1": 0.41606,
    "recall_at_10": 0.68456,
    "recall_at_100": 0.881,
    "recall_at_1000": 0.98,
    "recall_at_3": 0.55044,
    "recall_at_5": 0.61194
  }
}
```

`DenseRetrievalParallelExactSearch`:
```json
{
  "test": {
    "evaluation_time": 171.44,
    "map_at_1": 0.37106,
    "map_at_10": 0.45417,
    "map_at_100": 0.46099,
    "map_at_1000": 0.46138,
    "map_at_3": 0.43122,
    "map_at_5": 0.4442,
    "ndcg_at_1": 0.39333,
    "ndcg_at_10": 0.49823,
    "ndcg_at_100": 0.53358,
    "ndcg_at_1000": 0.54726,
    "ndcg_at_3": 0.45594,
    "ndcg_at_5": 0.47503,
    "precision_at_1": 0.39333,
    "precision_at_10": 0.06967,
    "precision_at_100": 0.00887,
    "precision_at_1000": 0.00101,
    "precision_at_3": 0.18444,
    "precision_at_5": 0.12333,
    "recall_at_1": 0.37106,
    "recall_at_10": 0.61889,
    "recall_at_100": 0.78967,
    "recall_at_1000": 0.90383,
    "recall_at_3": 0.50033,
    "recall_at_5": 0.54961
  }
}
```

Yes, I did confirm their equivalence a while ago **for some tasks only**. It could be the difference in the number of final results we return:
* For DRPES [it's exactly `top_k+1`.](https://github.com/beir-cellar/beir/blob/6ebe3eaf60774810f56fce7f11b1ef0ffe517753/beir/retrieval/search/dense/exact_search_multi_gpu.py#L145-L146)
* For DRES [it's `top_k+1` for each corpus chunk.](https://github.com/beir-cellar/beir/blob/6ebe3eaf60774810f56fce7f11b1ef0ffe517753/beir/retrieval/search/dense/exact_search.py#L69-L70)

cc @thakur-nandan


Hi @Muennighoff and @NouamaneTazi, thanks for pointing out the issue.

In theory, both are exact search methods so the results should be identical. I'm surprised we are getting a difference between the results. @NouamaneTazi we experimented and confirmed equivalence for which datasets?

Can we check the top-10 documents for a few sample queries and see the overlap in DRES and DRPES?

Kind regards,
Nandan Thakur

> Yes, I did confirm their equivalence a while ago **for some tasks only**. It could be the difference in the number of final results we return:
> 
> * For DRPES [it's exactly `top_k+1`.](https://github.com/beir-cellar/beir/blob/6ebe3eaf60774810f56fce7f11b1ef0ffe517753/beir/retrieval/search/dense/exact_search_multi_gpu.py#L145-L146)
> * For DRES [it's `top_k+1` for each corpus chunk.](https://github.com/beir-cellar/beir/blob/6ebe3eaf60774810f56fce7f11b1ef0ffe517753/beir/retrieval/search/dense/exact_search.py#L69-L70)
> 
> cc @thakur-nandan

I ran a test adding [this code](https://github.com/Muennighoff/sgpt/blob/127ae2599507996eb7f2793f7b3386aca503aaa6/biencoder/beir/custommodels/exact_search.py#L121) to DRES, which reduces the results to `top_k + 1` after each batch. So It's exactly `top_k+1` like in DPRES at the end. It produced the same results as without, but DPRES still produces different results. SciFact with [kom.](https://huggingface.co/sentence-transformers/average_word_embeddings_komninos) on 2 GPUs:


DRES with/w/o the modif:

```json
{
  "test": {
    "evaluation_time": 4.5,
    "map_at_1": 0.18667,
    "map_at_10": 0.25737,
    "map_at_100": 0.26597,
    "map_at_1000": 0.26716,
    "map_at_3": 0.23662,
    "map_at_5": 0.25047,
    "ndcg_at_1": 0.2,
    "ndcg_at_10": 0.29526,
    "ndcg_at_100": 0.34256,
    "ndcg_at_1000": 0.3779,
    "ndcg_at_3": 0.25674,
    "ndcg_at_5": 0.28073,
    "precision_at_1": 0.2,
    "precision_at_10": 0.04433,
    "precision_at_100": 0.00703,
    "precision_at_1000": 0.00102,
    "precision_at_3": 0.10778,
    "precision_at_5": 0.07867,
    "recall_at_1": 0.18667,
    "recall_at_10": 0.39944,
    "recall_at_100": 0.63361,
    "recall_at_1000": 0.91622,
    "recall_at_3": 0.30194,
    "recall_at_5": 0.35806
  }
```

DPRES
```json
{
  "test": {
    "evaluation_time": 15.18,
    "map_at_1": 0.18,
    "map_at_10": 0.24946,
    "map_at_100": 0.25776,
    "map_at_1000": 0.25898,
    "map_at_3": 0.22995,
    "map_at_5": 0.24255,
    "ndcg_at_1": 0.19333,
    "ndcg_at_10": 0.28628,
    "ndcg_at_100": 0.33218,
    "ndcg_at_1000": 0.36891,
    "ndcg_at_3": 0.25007,
    "ndcg_at_5": 0.27175,
    "precision_at_1": 0.19333,
    "precision_at_10": 0.043,
    "precision_at_100": 0.00683,
    "precision_at_1000": 0.00102,
    "precision_at_3": 0.10556,
    "precision_at_5": 0.076,
    "recall_at_1": 0.18,
    "recall_at_10": 0.38778,
    "recall_at_100": 0.61528,
    "recall_at_1000": 0.90956,
    "recall_at_3": 0.29528,
    "recall_at_5": 0.34639
  }
```

Interesting. Can you share the code you use for testing @Muennighoff , and your environment please?
For me I do get closer results between DRES and DRPES by running this code:
```python
import logging

from mteb import MTEB
from sentence_transformers import SentenceTransformer


logging.basicConfig(level=logging.INFO)

if __name__ == '__main__':

    model_name = "average_word_embeddings_komninos"
    model = SentenceTransformer(model_name)
    evaluation = MTEB(tasks="SciFact")
    evaluation.run(model, output_folder=None, eval_splits=["test"], corpus_chunk_size=50000)
```

```bash
(Output using DRPES)
INFO:root:NDCG@1: 0.2000
INFO:root:NDCG@3: 0.2567
INFO:root:NDCG@5: 0.2807
INFO:root:NDCG@10: 0.2953
INFO:root:NDCG@100: 0.3426
INFO:root:NDCG@1000: 0.3779
INFO:root:

INFO:root:MAP@1: 0.1867
INFO:root:MAP@3: 0.2366
INFO:root:MAP@5: 0.2505
INFO:root:MAP@10: 0.2574
INFO:root:MAP@100: 0.2660
INFO:root:MAP@1000: 0.2672
INFO:root:

INFO:root:Recall@1: 0.1867
INFO:root:Recall@3: 0.3019
INFO:root:Recall@5: 0.3581
INFO:root:Recall@10: 0.3994
INFO:root:Recall@100: 0.6336
INFO:root:Recall@1000: 0.9162
INFO:root:

INFO:root:P@1: 0.2000
INFO:root:P@3: 0.1078
INFO:root:P@5: 0.0787
INFO:root:P@10: 0.0443
INFO:root:P@100: 0.0070
INFO:root:P@1000: 0.0010
INFO:mteb.evaluation.MTEB:Evaluation for SciFact on test took 44.32 seconds
INFO:mteb.evaluation.MTEB:Scores: {'ndcg_at_1': 0.2, 'ndcg_at_3': 0.25674, 'ndcg_at_5': 0.28073, 'ndcg_at_10': 0.29526, 'ndcg_at_100': 0.34256, 'ndcg_at_1000': 0.3779, 'map_at_1': 0.18667, 'map_at_3': 0.23662, 'map_at_5': 0.25047, 'map_at_10': 0.25737, 'map_at_100': 0.26597, 'map_at_1000': 0.26716, 'recall_at_1': 0.18667, 'recall_at_3': 0.30194, 'recall_at_5': 0.35806, 'recall_at_10': 0.39944, 'recall_at_100': 0.63361, 'recall_at_1000': 0.91622, 'precision_at_1': 0.2, 'precision_at_3': 0.10778, 'precision_at_5': 0.07867, 'precision_at_10': 0.04433, 'precision_at_100': 0.00703, 'precision_at_1000': 0.00102, 'evaluation_time': 44.32}
--DONE--
```
```
(Output using DRES)

Time taken to retrieve: 2.40 seconds
INFO:root:

INFO:root:NDCG@1: 0.2000
INFO:root:NDCG@3: 0.2567
INFO:root:NDCG@5: 0.2807
INFO:root:NDCG@10: 0.2953
INFO:root:NDCG@100: 0.3426
INFO:root:NDCG@1000: 0.3779
INFO:root:

INFO:root:MAP@1: 0.1867
INFO:root:MAP@3: 0.2366
INFO:root:MAP@5: 0.2505
INFO:root:MAP@10: 0.2574
INFO:root:MAP@100: 0.2660
INFO:root:MAP@1000: 0.2672
INFO:root:

INFO:root:Recall@1: 0.1867
INFO:root:Recall@3: 0.3019
INFO:root:Recall@5: 0.3581
INFO:root:Recall@10: 0.3994
INFO:root:Recall@100: 0.6336
INFO:root:Recall@1000: 0.9162
INFO:root:

INFO:root:P@1: 0.2000
INFO:root:P@3: 0.1078
INFO:root:P@5: 0.0787
INFO:root:P@10: 0.0443
INFO:root:P@100: 0.0070
INFO:root:P@1000: 0.0010
INFO:mteb.evaluation.MTEB:Evaluation for SciFact on test took 2.55 seconds
INFO:mteb.evaluation.MTEB:Scores: {'ndcg_at_1': 0.2, 'ndcg_at_3': 0.25674, 'ndcg_at_5': 0.28073, 'ndcg_at_10': 0.29526, 'ndcg_at_100': 0.34256, 'ndcg_at_1000': 0.3779, 'map_at_1': 0.18667, 'map_at_3': 0.23662, 'map_at_5': 0.25047, 'map_at_10': 0.25737, 'map_at_100': 0.26597, 'map_at_1000': 0.26716, 'recall_at_1': 0.18667, 'recall_at_3': 0.30194, 'recall_at_5': 0.35806, 'recall_at_10': 0.39944, 'recall_at_100': 0.63361, 'recall_at_1000': 0.91622, 'precision_at_1': 0.2, 'precision_at_3': 0.10778, 'precision_at_5': 0.07867, 'precision_at_10': 0.04433, 'precision_at_100': 0.00703, 'precision_at_1000': 0.00102, 'evaluation_time': 2.55}
--DONE--
```




And after further investigation, it does seem that DRPES results depend on `corpus_chunk_size` 

For example, using:
```python
evaluation.run(model, output_folder=None, eval_splits=["test"])
```
would set `corpus_chunk_size=260` which greatly influence the final results:


```bash
(Output using DRPES corpus_chunk_size=None)
Time taken to retrieve: 11.89 seconds
INFO:root:

INFO:root:NDCG@1: 0.2000
INFO:root:NDCG@3: 0.2567
INFO:root:NDCG@5: 0.2807
INFO:root:NDCG@10: 0.2947
INFO:root:NDCG@100: 0.3419
INFO:root:NDCG@1000: 0.3773
INFO:root:

INFO:root:MAP@1: 0.1867
INFO:root:MAP@3: 0.2366
INFO:root:MAP@5: 0.2505
INFO:root:MAP@10: 0.2570
INFO:root:MAP@100: 0.2656
INFO:root:MAP@1000: 0.2668
INFO:root:

INFO:root:Recall@1: 0.1867
INFO:root:Recall@3: 0.3019
INFO:root:Recall@5: 0.3581
INFO:root:Recall@10: 0.3978
INFO:root:Recall@100: 0.6319
INFO:root:Recall@1000: 0.9146
INFO:root:

INFO:root:P@1: 0.2000
INFO:root:P@3: 0.1078
INFO:root:P@5: 0.0787
INFO:root:P@10: 0.0440
INFO:root:P@100: 0.0070
INFO:root:P@1000: 0.0010
INFO:mteb.evaluation.MTEB:Evaluation for SciFact on test took 12.02 seconds
INFO:mteb.evaluation.MTEB:Scores: {'ndcg_at_1': 0.2, 'ndcg_at_3': 0.25674, 'ndcg_at_5': 0.28073, 'ndcg_at_10': 0.29465, 'ndcg_at_100': 0.34194, 'ndcg_at_1000': 0.37729, 'map_at_1': 0.18667, 'map_at_3': 0.23662, 'map_at_5': 0.25047, 'map_at_10': 0.257, 'map_at_100': 0.2656, 'map_at_1000': 0.26679, 'recall_at_1': 0.18667, 'recall_at_3': 0.30194, 'recall_at_5': 0.35806, 'recall_at_10': 0.39778, 'recall_at_100': 0.63194, 'recall_at_1000': 0.91456, 'precision_at_1': 0.2, 'precision_at_3': 0.10778, 'precision_at_5': 0.07867, 'precision_at_10': 0.044, 'precision_at_100': 0.007, 'precision_at_1000': 0.00102, 'evaluation_time': 12.02}
--DONE--
```

This is, what I meant @thakur-nandan when I said that I got some small differences between DRES and DRPES. I thought it was because of how `trec` handled the number of documents we returned per query, but apparently It's just because I was experimenting with high `corpus_chunk_size=50000`

I'll try to debug this weird behaviour and make a more stable DRPES. But for now please refrain from using it for your experiments @Muennighoff. Thank you for raising the issue 

Yeah I was leaving corpus_size as the default. The code is [here](https://github.com/Muennighoff/mtebscripts/blob/d0dc59d8f6ee3cb64e5b81ad8489b8716dc1dbb6/run_array.py#L152).

Thanks for investigating it!

> I'll try to debug this weird behaviour and make a more stable DRPES. But for now please refrain from using it for your experiments @Muennighoff. Thank you for raising the issue

Do you already have an update on this? 😇

@Muennighoff @NouamaneTazi was this fixed in #107?

I think so; if you use mteb this issue is definitely no longer present as `DenseRetrievalParallelExactSearch` has been removed in favor of just doing the parallelism on the modeling side e.g. sth like https://github.com/ContextualAI/gritlm/blob/b89fdefa18731f1aa1d6111c3849c1e4c811b9d6/gritlm/gritlm.py#L75