Repository: beir-cellar/beir
Issue: BM25 reproducibility issues for ArguAna and FEVER datasets.
Root cause: External Dependency Breakage

I am trying to reproduce your research using the `evaluate_bm25.py` script located [there](https://github.com/beir-cellar/beir/tree/main/examples/retrieval/evaluation/lexical).

I have issues with two datasets - ArguAna and FEVER. These are the `NDCG@10` values returned by the script:
ArguAna - 47.16 vs 31.5 in your paper
FEVER - 64.9 vs 75.3 in your paper

The same inconsistency occurs in the repository with [BM25 evaluations](https://github.com/xhluca/bm25-benchmarks) prepared by the author of [BM25S](https://github.com/xhluca/bm25s), where he reports `NDCG@10` values similar to mine.

Has anything changed about these datasets over the last few years after your article was published?

Hi @dkoterwa - this might be due to how different text fields are treated. Look at:
https://dl.acm.org/doi/10.1145/3626772.3657862

"multifield" vs. "flat"

Hope this helps.

Thank you for a quick response @lintool 

To clarify, I am using the [evaluate_bm25.py](https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/lexical/evaluate_bm25.py) script, not [evaluate_anserini_bm25.py](https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/lexical/evaluate_anserini_bm25.py), because I thought that it would bring me exactly the same results as in the original BEIR paper.

Regarding the distinction between "multifield" and "flat":

In the source you linked in your message it is written that official BEIR paper used a multifield approach:
> The original BEIR paper presented a BM25 baseline using Elasticsearch. We refer to this as “multi-field” BM25 because it ingested  the title and body of documents into separate fields (called “title” and “contents”, respectively) in cases where the original corpus provided this information.
 
I think that this behavior is also visible in the source code of this repository:

```python
# index function of the BM25Search class
def index(self, corpus: Dict[str, Dict[str, str]]):
        progress = tqdm.tqdm(unit="docs", total=len(corpus))
        # dictionary structure = {_id: {title_key: title, text_key: text}}
        dictionary = {idx: {
            self.config["keys"]["title"]: corpus[idx].get("title", None), 
            self.config["keys"]["body"]: corpus[idx].get("text", None)
            } for idx in list(corpus.keys())
        }
        self.es.bulk_add_to_index(
                generate_actions=self.es.generate_actions(
                dictionary=dictionary, update=False),
                progress=progress
                )
```

Considering all I wrote before - shouldn't the results obtained by running  the [evaluate_bm25.py](https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/lexical/evaluate_bm25.py) script be the same as these reported in the original paper (if we assume that the datasets have been unchanged over the years)?

Below is a comparison for 5 data sets between the paper (Table 2) you linked and the script result. I am able to get close to the paper results for data sets other than ArguAna and FEVER, for which the difference is significant.

| Dataset  | Paper metric | Script output |
| ------------- | ------------- | ------------- |
| NQ  | 32.9 | 32.6 |
| SCIDOCS | 15.8 | 16.5 |
| FiQA-2018 | 23.6 | 25.4 |
| ArguAna | 41.4 | 47.2 |
| FEVER | 75.3 | 64.9 |



Hi @dkoterwa, please use the Pyserini to reproduce the BEIR BM25 scores. Elasticsearch updates its versions, which can cause reproducibility issues. On the other hand, pyserini has 2-click regressions using which you can reproduce exactly the nDCG@10 score.

Here is the 2CR table: https://castorini.github.io/pyserini/2cr/beir.html