zhiminy/EvalEng huggingface dataset:
Repository: beir-cellar/beir
Issue: Can't reproduce BM25 baselines
Root cause: Documentation Deficiency

Hi

I instal elacticsearch on debian 10:

```
{
  "name" : "xx",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "iE5FrFezRoWh0J-bp1Zb5g",
  "version" : {
    "number" : "7.17.10",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "fecd68e3150eda0c307ab9a9d7557f5d5fd71349",
    "build_date" : "2023-04-23T05:33:18.138275597Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
```
I run the [BM25](https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/lexical/evaluate_bm25.py) on BEIR MS-MARCO dataset, and I obtain NDCG@10: 0.4769 which is much bigger than the score 0.228 in table 2 of your paper. On scifact, I get NDCG@10 0.6906 which is also much higher than 0.665 in table 2. Any suggestions? Thanks




I re-run the [notebook](https://colab.research.google.com/drive/1HfutiEhHMJLXiWGT8pcipxT5L2TpYEdt?usp=sharing#scrollTo=yJcCpUCFClis) and get a very closed NDCG@10 score: 0.69064 on scifact. The two NDCG@10 scores I get are much bigger than 0.665 reported in table 2

For msmarco you have to run it on the dev split

@nreimers 

Thanks for your reply. On msmarco dev, I got a very closed NDGC@10 score: `0.22747`. I am confused about reporting the results on dev dataset instead of the test dataet. The training script of [sbert](https://github.com/beir-cellar/beir/blob/main/examples/retrieval/training/train_sbert.py) evaluates the model on dev dataset during the training. Why report the results on dev? BTW, could you make it clear that how to use your splits. like which split is for training, which split is for evaluation during the training and which file is for final testing. Thank your very much!
 

Msmarco doesn't have a test set.
here in BEIR the test split is TREC DL 2019. It is quite confusing.

People report for msmarco on dev set. This dev set shouldn't be used for training/stopping etc

@nreimers Thank you for the clarification! BTW, I can's reproduce your BM25 NDCG@10 on FEVER. I run your notebook and get NDCG@10 0.64938 which is much smaller than 0.753 reported in table 2. On dev, the NDCG@10 is 0.66363 which is also much smaller than 0.753.

update:
BEIR reports Anserini BM25. I will run it. Thanks.

To reproduce official BEIR scores, Pyserini is probably easier... https://github.com/castorini/pyserini/

Specifically, try: https://castorini.github.io/pyserini/2cr/beir.html

You'll be able to get the scores on the official BEIR leaderboard: https://eval.ai/web/challenges/challenge-page/1897/overview

cc/ @thakur-nandan 



@lintool Thanks. My reproduced NDCG@10 on FEVER is much closer to your result 0.6513. BEIR utilizes Anserini BM25. I will run it to reproduce the results.
Repository: beir-cellar/beir
Issue: Don't rebuild bm25 pyserini index every time when running the script
Root cause: Unimplemented Feature Gap

Hi there,

In the file `eval_bm25.py`, there is a parameter `initialize` to control whether or not to rebuild the index. However, I didn't find such a parameter in the file `evaluate_anserini_bm25.py`.  There is no need to rebuild the index for the same dataset each time I run the script. How can I skip the index building in file `evaluate_anserini_bm25.py` if I don't want to rebuild it. Thanks.

Best Regards,
Zhiyuan.
Repository: beir-cellar/beir
Issue: evaluate_anserini_bm25.py retrieves 1000 documents for each query no matter which k I set in payload
Root cause: Configuration Error

Hi there,

I set k=10 in `payload = {"queries": query_texts, "qids": qids, "k": 10` but the bm25 still retrieves 1000 documents for each query. I re-filter the 1000 documents to work around it. It will be great if you can fix this bug. Thanks.


Hi @zhiyuanpeng, 

Thanks for notifying this bug. I'll add it in my todo list to update it.

The `evaluate_anserini_bm25.py` actually uses an old docker version of Anserini BM25. If you wish to run the latest Anserini BM25, I would redirect you here:  https://github.com/castorini/anserini/tree/master#%EF%B8%8F-regression-experiments--reproduction-guides.

The Anserini repository lets you change more parameters and ensures reproducible BM25 regressions on BEIR.

Hope it helps!


@thakur-nandan File an issue to redirect the BM25 baselines over to Pyserini? Will save you from having to answer such queries again in the future...
Repository: beir-cellar/beir
Issue: Reproduce Signal-1M
Root cause: Documentation Deficiency

I have seen that to reproduce the Signal-1M dataset I should manually scrape the tweets with the scrape_tweets.py script you provided. 

After inserting my consumer_key and consumer_secret I am facing the following error:

`Forbidden: 403 Forbidden
453 - You currently have access to a subset of Twitter API v2 endpoints and limited v1.1 endpoints (e.g. media post, oauth) only. If you need access to this endpoint, you may need a different access level. You can learn more here: https://developer.twitter.com/en/portal/product`

Can you give me some indications? Or the only way is to pay for the Basic subscription in order to get the access level?



Hi @Freddavide, you can download directly the SIGNAL-1M preprocessed dataset here: https://drive.google.com/drive/folders/1CgDO-KmQQMpGEGeD3R20ZgTTM008xix9?usp=sharing.

Please make sure you have the necessary licenses required.

Thanks,
Nandan

Wow thanks!
Repository: beir-cellar/beir
Issue: Redirect Anserini BM25 baselines in BEIR to Pyserini
Root cause: Documentation Deficiency

The current Anserini-based BM25 implementation in BEIR uses an old docker image. 

Redirect authors to the Pyserini repository where they can evaluate the latest BM25 baselines and regressions.
 
- [ ] Add Anserini BM25 regressions in README.
- [ ] Update the `evaluation_anserini_bm25.py` with a note saying to check Pyserini.
- [ ] Update BEIR Wiki docs to add information about Pyserini Regressions.
- [ ] Link latest BEIR resources Paper.
Repository: beir-cellar/beir
Issue: Storing document embeddings index
Root cause: Documentation Deficiency

Is there a way to cache/load embedded documents and queries? That would help to save time on embedding big datasets such as ms marco and nq

Hi @svakulenk0,

Sadly at the moment, the only solution is to pickle save the document embeddings.

Recently, I have been working on getting faiss indexes to be integrated. That would allow caching or saving corpus embeddings as a faiss index. I can't say a fixed timeline of when this will be completely integrated into the repo, but I will let you know once it is done.

Kind Regards,
Nandan

Hi Nandan, thank you for the reply! I love the library :)

Hi @svakulenk0,

Update: In the latest version of the BEIR package, now you can save/load the corpus embeddings as a faiss index.
Check out: https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/dense/evaluate_faiss_dense.py

Kind Regards,
Nandan

nice!!! thank you

> Is there a way to cache/load embedded documents and queries? That would help to save time on embedding big datasets such as ms marco and nq

Thanks Svitlana for this question and Nandan for providing this feature, it is beneficial!

> nice!!! thank you

Hi @svakulenk0, have you tried this?

> Hi @svakulenk0,
> 
> Update: In the latest version of the BEIR package, now you can save/load the corpus embeddings as a faiss index. Check out: https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/dense/evaluate_faiss_dense.py
> 
> Kind Regards, Nandan

Hi @thakur-nandan, thank you for this. Can you please give me an example of how to store and load the embeddings?
Repository: beir-cellar/beir
Issue: Missing test.tsv in thakur/BEIR/datasets/nq-train.zip
Root cause: Documentation Deficiency

Hi Nandan,

@thakur-nandan I can download the original train.tsv and the corresponding queries.jsonl and corpus.jsonl from https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nq-train.zip. But, there is no original test.tsv file in nq-train. The BEIR NQ corpus.jsonl is a reduced one and so, I can't use BEIR NQ's test.tsv. Do you have the test.tsv that can work with the corpus.jsonl in nq-train? Thanks.

Best Regards,
Zhiyuan.

Hi, check https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nq.zip for the BEIR test evaluation.

The `nq-train` folder which you use is only meant for training on NQ which contains a different corpus.
Repository: beir-cellar/beir
Issue: org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024    
Root cause: Configuration Error

When I run the beir/pysirini-fastapi for the BM25 search algorithm on my customized dataset, there is an error on the terminal:
`org.apache.lucene.search.BooleanQuery$TooManyClauses: maxClauseCount is set to 1024    `
In this case, I would like to know if there is any way to change the parameter `maxClauseCount` to a larger number. I would be very grateful if you could help me with this.
Repository: beir-cellar/beir
Issue: Typing for retriever is too strict: implement abstract retriever base class 
Root cause: Architectural Constraint

One should be able to evaluate BEIR on other retrievers other than  `Union[Type[DRES], Type[DRFS], Type[BM25], Type[SS]`

The work around is the use the static method `evaluate`. It's slightly awkward. See SPLADE's eval
https://github.com/naver/splade/blob/06947de1da83e5443809f1b7e78937416d53c14b/splade/beir_eval.py#L85
Repository: beir-cellar/beir
Issue: Fine-tuned Cross-encoder scores are lower than ms-marco zero-shot scores ?
Root cause: Documentation Deficiency

Hi @thakur-nandan, @nreimers 

I am fine-tuning the either `cross-encoder/ms-marco-electra-base` or `cross-encoder/ms-marco-MiniLM-L-12-v2` models on other IR collections (tree-covid or NQ). But the fine-tuned model scores are lower than zero-shot scores. I wonder if there's a domain shift in custom datasets or am I doing the training wrong ? I am using sentence-transformer cross-encoder APIs for training.

Since these pre-trained models trained in certain settings (hyper-parameters and model architecture with loss), are these models sensitive to those settings during fine-tuning as well ?


Hey!
Could you give the code, so we can further help you diagnose the issue?
Repository: beir-cellar/beir
Issue: Unable to find pyserini docker
Root cause: Documentation Deficiency

Hi, I'm trying to experiment with beir, BM25 and pyserini, but I'm unable to find the docker image (beir/beir-pyserini). Looking at dockerhub (https://hub.docker.com/u/beir) it seems that the only image available is pyserini-fastapi. 

Is pyserini-fastapi the same as beir-pyserini or there is something that I'm doing wrong?

Hi @cadurosar,

Both are actually the same. So you can use ``docker pull beir/pyserini-fastapi`` to get the latest docker image.
``beir-pyserini`` was the original (old) docker image name but it was not a great name so I renamed it ``pyserini-fastapi``. 

Thanks for pointing this out, I missed updating the image names in the BEIR examples. I will do that. 

Kind Regards,
Nandan
Repository: beir-cellar/beir
Issue: Extras-require, base package without Faiss-CPU requirement
Root cause: Unimplemented Feature Gap

Hi,

I appreciate the work you're doing with the Beir benchmark – it's very useful.

I'm developing a library that enables training models like Splade, Sparsembed, BLP, and Colbert using PyTorch. I've included Beir in the requirements to facilitate evaluations during training. However, I'm facing an issue. I want to utilize Faiss GPU with my own retriever for evaluating my models on Beir benchmarks and I don't wan't to install Faiss CPU.

I'm wondering if it's possible to create a custom extras_requires section in your setup.py that would exclude Faiss for users who don't intend to use Beir retrievers.

Here is an example:

```sh
pip install "beir[no_faiss]"
```

```python
base_packages = [
      'sentence-transformers',
      'pytrec_eval',
      'faiss_cpu',
      'elasticsearch==7.9.1',
      'datasets'
 ]

no_faiss = [
      'sentence-transformers',
      'pytrec_eval',
      'elasticsearch==7.9.1',
      'datasets'
 ]

setuptools.setup(
    ...,
    packages=setuptools.find_packages(),
    install_requires=base_packages,
    extras_require={
        "no_faiss": no_faiss
    }
)
```

I'm fine with creating a PR if you are ok with this.

Thank you,

Raphaël

Hi @raphaelsty, Sounds like a meaningful suggestion. 

Feel free to create the PR. I can go ahead and merge the PR.

Regards,
Nandan
Repository: beir-cellar/beir
Issue: Malformed Chinese mMARCO dataset
Root cause: External Dependency Breakage

Thanks for this excellent tool firstly!

I'm goind to calculate metrics based on some Chinese datasets, like mMARCO. I downloaded Chinese mMARCO via the link (https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/mmarco.zip). When I opened the `queries.jsonl`, the content of `text` is malformed as follows:
```
// {\\fn华文楷体\\fs16\\1cHE0E0E0} and {\\fn华文楷体\\fs16\\1cHE0E0E0}"}
{"_id": "224811", "text": "{\\fn华文楷体\\fs16\\1cHE0E0E0}萤火虫怎么点亮的 {\\fn华文楷体\\fs16\\1cHE0E0E0}"}

// many repeated 每桶
{"_id": "473204", "text": "房客建房每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶每桶"}

// meaningless and ungrammatical
{"_id": "880877", "text": "何国籍何为姓甘?"}

//  meaningless and repeated $
{"_id": "319885", "text": "$$$$$$$$$$$$$$$ $$$ $$$ $$$ $$$ $$ $$ $ $$ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $"}

// ????
{"_id": "1035441", "text": "???????"}
```
Repository: beir-cellar/beir
Issue: Corpus encoding times for hotpotqa on A100 GPU
Root cause: Algorithmic Error

I'm trying to reproduce evaluate_sbert.py on the `hotpotqa` dataset on an A100 GPU (AWS ml.p4d.24xlarge instance), using `msmarco-distilbert-base-tas-b` model.
According to the progress, it seems to be taking about 8 minutes for ~ 10,000 corpus passages, implying it will take about 69 hours for the entire 5,233,329 passages. Is this normal, or am I doing something really wrong? If the latter, could anybody share some expected times, or any tips?
Thanks a lot!
Repository: beir-cellar/beir
Issue: TypeError: Cannot create a consistent method resolution order (MRO) for bases tqdm, tqdm_asyncio
Root cause: External Dependency Breakage

![image](https://github.com/beir-cellar/beir/assets/55336369/3aef1061-04d5-45f4-9add-be244d708b84)

When running the BEIR examples. The problem is coming from the haystack module when imported with the beir module. 
[haystack seems to be aware of this issue](https://github.com/deepset-ai/haystack/issues/5478). But even the latest haystack 1.x version gives the same error.
Repository: beir-cellar/beir
Issue: Why need check corpus_id != query_id in DenseRetrievalExactSearch.search()
Root cause: Documentation Deficiency

Why do we need [this line](https://github.com/beir-cellar/beir/blob/f062f038c4bfd19a8ca942a9910b1e0d218759d4/beir/retrieval/search/dense/exact_search.py#L81) to check corpus_id != query_id

for a query with id_q, the corpus with the same id id_q does not mean it is the positive corpus for it. So why do we need to avoid corpus_id == query_id

```
            for query_itr in range(len(query_embeddings)):
                query_id = query_ids[query_itr]                  
                for sub_corpus_id, score in zip(cos_scores_top_k_idx[query_itr], cos_scores_top_k_values[query_itr]):
                    corpus_id = corpus_ids[corpus_start_idx+sub_corpus_id]
                    if corpus_id != query_id:
                        if len(result_heaps[query_id]) < top_k:
                            # Push item on the heap
                            heapq.heappush(result_heaps[query_id], (score, corpus_id))
                        else:
                            # If item is larger than the smallest in the heap, push it on the heap then pop the smallest element
                            heapq.heappushpop(result_heaps[query_id], (score, corpus_id))

        for qid in result_heaps:
            for score, corpus_id in result_heaps[qid]:
                self.results[qid][corpus_id] = score
        
        return self.results 
```

Hi @mengyao00, thanks for asking the question.

We require this line for two datasets: ArguAna and Quora, where `corpus_ids` and `query_ids` are similar, i.e., the query is also present within the corpus. 

The line is used to avoid the edge case of self-retrieval where the query is self-retrieved at the top-1 position, which reduces the nDCG@10 score for ArguAna and Quora.

Hope it helps!

Regards,
Nandan Thakur
Repository: beir-cellar/beir
Issue: init_weights error when loading DPR models from HF modelhub
Root cause: Unimplemented Feature Gap

Congrats on this very well structured, documented and helpful framework for figuring out whats going on in IR - especially on OOD data. Keep up the good work!

When loading DPR models from HF modelhub like:

```python
model = DRES(models.SentenceBERT((
    "facebook/dpr-question_encoder-multiset-base",
    "facebook/dpr-ctx_encoder-multiset-base",
    " [SEP] "), batch_size=128))
```

I run into an `NotImplementedError: Make sure `_init_weigths` is implemented for <class 'transformers.models.dpr.modeling_dpr.DPRQuestionEncoder'>`

I know you converted the model to a sentencetransformers already and can be loaded like this but an interoperability with the HF hub would be slick - also for other DPR models in other languages like [French](https://huggingface.co/etalab-ia/dpr-question_encoder-fr_qa-camembert) or [German](https://huggingface.co/deepset/gbert-base-germandpr-ctx_encoder).

Thanks

Hi @Timoeller, 

Added support to evaluate HF DPR models, you can load using this snippet below:
```python
model = DRES(models.DPR((
    "facebook/dpr-question_encoder-multiset-base",
    "facebook/dpr-ctx_encoder-multiset-base"), batch_size=128))
```
Also works with deepset.ai GermanDPR model,
```python
model = DRES(models.DPR((
    "deepset/gbert-base-germandpr-question_encoder",
    "deepset/gbert-base-germandpr-ctx_encoder"), batch_size=128))
```
I would push the updates to the latest pip version hopefully soon. Until then you can download the repository and run it locally if required.

Kind Regards,
Nandan

Thanks for the fix, tested on current master and it works.
Repository: beir-cellar/beir
Issue: `retriever.retrieve(corpus, queries)` with `len(queries)==1` errors
Root cause: Validation Gap

I have a somewhat silly use case. I'm running `retriever.retrieve`, with a `queries` dict with only 1 entry. However, this causes an IndexError with pytorch due to how pytorch indexes 2D arrays where the first dim is of size 1:
```
File "<dir>/script.py", line 83, in <module>
    results = retriever.retrieve(para_d, one_query_d)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<env>/beir/retrieval/evaluation.py", line 20, in retrieve
    return self.retriever.search(corpus, queries, self.top_k, self.score_function, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<env>/beir/retrieval/search/dense/exact_search.py", line 73, in search
    scores_dim = cos_scores[1]
                 ~~~~~~~~~~^^^
IndexError: index 1 is out of bounds for dimension 0 with size 1
```

I was able to fix this by changing the below line:
https://github.com/beir-cellar/beir/blob/f062f038c4bfd19a8ca942a9910b1e0d218759d4/beir/retrieval/search/dense/exact_search.py#L73

And swapping out `cos_scores[1]` with `scores_dim`:
```
scores_dim = cos_scores[1] if cos_scores.shape[0] != 1 else cos_scores[0]
```

This monkey patch works for me, but I'm just curious if there's a more appropriate way to retrieve given only 1 query. Thanks!

As another dirty fix, I just added the same query twice (with the same relevant documents) so it does not complain about only one query.
Repository: beir-cellar/beir
Issue: How to Save Retrieved Documents in Retrieval Tasks
Root cause: Documentation Deficiency

I am currently working on an information retrieval task and I want to save the retrieved documents for bad case analysis.
I have printed the output of the search function in beir/retrieval/search/dense/exact_search.py, and the format is {q_id: {pid: score}}. such as:

![image](https://github.com/beir-cellar/beir/assets/109325850/9b4b1d23-4d5a-4080-86bc-9e99794b05f9)


However, it seems that the corpus ids are not sorted according to the scores. Should I sort the results by cosine similarity from high to low and save them so that the most relevant documents are placed at the beginning?

Thanks!

yes, the scores are not sorted, you can use this line to sort them:
```python
for query_id in results:
    scores_sorted = sorted(results[query_id].items(), key=lambda item: item[1], reverse=True)
```
Repository: beir-cellar/beir
Issue: Missing document from corpus ArguAna
Root cause: Validation Gap

### Dataset
ArguAna

### Problem
In the qrels, there is one instance where for the given query `test-free-speech-debate-yfsdfkhbwu-con03a` the relevant document had the following id `test-free-speech-debate-yfsdfkhbwu-con03b`. However, if you check the corpus and look up for this id, there is no such a document.

Yes, this is possible. This is likely an issue with the original dataset: ArguAna. Unfortunately, we would need to contact the original dataset authors in this case.
Repository: beir-cellar/beir
Issue: mt5-base model (unicamp-dl/mt5-base-en-msmarco) throws 'nan' outputs
Root cause: External Dependency Breakage

Hi,

I was running `unicamp-dl/mt5-base-en-msmarco`: ['▁no' , '▁yes'] model for both English and other My.TyDi languages, but the output scores are **nan**. When I switched to `unicamp-dl/mt5-13b-mmarco-100k`: ['▁', '▁true'] model, I get actual real logits. I wonder if there's any issues with underlying `unicamp-dl/mt5-base-en-msmarco` model.

Thanks.

I sadly am not aware of the model and what the issue might be here.
Repository: beir-cellar/beir
Issue: recommended pytrec-eval-terrier  rather than pytrec-eval
Root cause: External Dependency Breakage

It is recommended to use `pytrec-eval-terrier` instead of `pytrec-eval`.

During the installation of beir, pytrec-eval is required, but this library has not been maintained for 4 years, and installation failures often occur, which in turn causes beir failure.
`https://pypi.org/project/pytrec-eval-terrier/#files`
It is a substitute for pytrec-eval. I recommend that beir replace the dependency in setup.py

This is done!
Repository: beir-cellar/beir
Issue: can't reproduce DeepCT results with evaluate_deepct.py
Root cause: External Dependency Breakage

Hi!


We are trying to reproduce the beir results for DeepCT framework on NF Corpus for our university project, however, we get very different scores. The paper reports an NDCG@10 of 0.283 while we get 0.2146. What we did was:

- cloned this repository
- cloned [DeepCT repo](https://github.com/NThakur20/DeepCT.git)
- pulled the Docker container beir/pyserini-fastapi
- run the container:  docker run -p 8000:8000 -it --rm beir/pyserini-fastapi
- downgrade the tf version to 2.15 (tf.estimator is deprecated in 2.16)
- change the dataset to `nfcorpus`
- run evaluate_deepct.py

We didn't change anything inside the scripts, so we assume the model is exactly the same.

Were there any major changes to the repository code after the paper had been released that could cause such big mismatches? 

I sadly believe the DeepCT repository is quite outdated at this point. I apologise but I cannot help at this point.