zhiminy/EvalEng huggingface dataset:
Repository: beir-cellar/beir
Issue: NQ - File datasets/nq/qrels/train.tsv not present
Root cause: Documentation Deficiency

I downloaded the NQ dataset and the tsv file for the train set appears to be missing

Hi @GiacoL 
There's another zip file for train set - "nq-train"
All the available datasets - https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/

https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/,this url contains nq dataset,but I didn't find train.csv in nq.zip

> I downloaded the NQ dataset and the tsv file for the train set appears to be missing

please,where did you finally download train.csv?

Hi @Gerry-j  https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nq-train.zip


Thank you very much！




------------------&nbsp;原始邮件&nbsp;------------------
发件人:                                                                                                                        "beir-cellar/beir"                                                                                    ***@***.***&gt;;
发送时间:&nbsp;2024年8月15日(星期四) 上午10:31
***@***.***&gt;;
***@***.******@***.***&gt;;
主题:&nbsp;Re: [beir-cellar/beir] NQ - File datasets/nq/qrels/train.tsv not present (Issue #179)





 
Hi @Gerry-j  https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nq-train.zip
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***&gt;

Hi all, thanks for this info. Is the corpus set not the same between them? I see `18,060,996` lines in the corpus for this link but the BEIR NQ corpus for test has `2,681,468`? Perhaps the train has the unfiltered corpus while the test has the filtered version.

It seems like the qrels for train have documents up to 18 million also, so it appears one would have to index the train corpora separately to use these. 

@orionw yes that's true. The BEIR NQ test corpus is different from the train corpus (much larger, can contain duplicates), therefore for uniformity we choose to not complicate our BEIR structure and provide nq training pairs separately under `nq-train.zip`

Closing this issue as it has been resolved.
Repository: beir-cellar/beir
Issue: Training CrossEncoder using BEIR

How can I train cross encoder using the BEIR style example given here: https://github.com/beir-cellar/beir/blob/main/examples/retrieval/training/train_sbert.py

Hi @jmnian 
You may try the following : 

```
from sentence_transformers.cross_encoder.evaluation import CECorrelationEvaluator
from sentence_transformers.cross_encoder import CrossEncoder
from beir import util, LoggingHandler
from beir.datasets.data_loader import GenericDataLoader
from beir.retrieval.train import TrainRetriever
import pathlib, os
import logging

#### Just some code to print debug information to stdout
logging.basicConfig(format='%(asctime)s - %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S',
                    level=logging.INFO,
                    handlers=[LoggingHandler()])
#### /print debug information to stdout

#### Download nfcorpus.zip dataset and unzip the dataset
dataset = "nfcorpus"

url = "https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/{}.zip".format(dataset)
out_dir = os.path.join(pathlib.Path(__file__).parent.absolute(), "data")
data_path = util.download_and_unzip(url, out_dir)

#### Provide the data_path where nfcorpus has been downloaded and unzipped
corpus, queries, qrels = GenericDataLoader(data_path).load(split="train")
#### Please Note not all datasets contain a dev split, comment out the line if such the case
dev_corpus, dev_queries, dev_qrels = GenericDataLoader(data_path).load(split="dev")

#### Or provide pretrained sentence-transformer model
model = CrossEncoder("distilroberta-base", num_labels=1)
retriever = TrainRetriever(model=model, batch_size=16)

#### Prepare training samples
train_samples = retriever.load_train(corpus, queries, qrels)
dev_samples = retriever.load_train(dev_corpus, dev_queries, dev_qrels)
train_dataloader = retriever.prepare_train(train_samples, shuffle=True)

#### Prepare dev evaluator
ir_evaluator = CECorrelationEvaluator.from_input_examples(dev_samples, name="nfcorpus-dev")

#### Provide model save path
model_save_path = os.path.join(pathlib.Path(__file__).parent.absolute(), "output", "{}-v1-{}".format('distilroberta-base', dataset))
os.makedirs(model_save_path, exist_ok=True)

#### Configure Train params
num_epochs = 1
evaluation_steps = 5000
warmup_steps = int(len(train_samples) * num_epochs / retriever.batch_size * 0.1)

model.fit(
    train_dataloader=train_dataloader,
    evaluator=ir_evaluator,
    epochs=num_epochs,
    warmup_steps=warmup_steps,
    output_path=model_save_path,
)
```

Thank you very much! I will try that
Repository: beir-cellar/beir
Issue: how was the hotpot_qa dataset preprocessed?
Root cause: Documentation Deficiency

I am curious how you created the list of documents (the corpus). The original hotpot_qa does not come with that list of documents. Instead for each query it comes with a list of only 10 documents - 2 documents with the content for the gold answer and 8 distractor documents. My current assumption is the following. you took the distractor dataset and extracted the documents for all queries to build the corpus. The 2 gold documents in the original hotpotqa were then marked as the relevant documents for a specific query.

Please let me know how it works, since this confuses me quite a lot. Thank you very much!

If my assumption is correct you could also have a look at the multi-hop-rag dataset which was specifically created in that format already (corpus is seperated from the query and answer). The documents are also longer, which I think is a more realistic use case for a retrieval system, specially a RAG system.

We use the `full-wiki` setting in HotpotQA which provided us with the whole corpus in the website.
Link: https://hotpotqa.github.io/wiki-readme.html

Closing issue for now.
Repository: beir-cellar/beir
Issue: BUG - `EvaluateRetrieval`'s `evaluate` doesn't consider results dict order
Root cause: Algorithmic Error

I'm guessing it because it runs C code underneath, and in the translation to HashMap the iteration order because the lexical order of the keys instead of the original order of the dict.

I opened a pull request that fixes the issue:

https://github.com/beir-cellar/beir/pull/183



We do not need to order the dict to retrieve results, pytrec_eval can evaluate this automatically without ordering the dict.
Repository: beir-cellar/beir
Issue: evaluate_sbert_multi_gpu run error
Root cause: Documentation Deficiency

i got some errors when runing example evaluate_sbert_multi_gpu.py:

```bash
File "/root/miniconda3/envs/qe/lib/python3.9/site-packages/beir/retrieval/evaluation.py", line 20, in retrieve
    return self.retriever.search(corpus, queries, self.top_k, self.score_function, **kwargs)
  File "/root/miniconda3/envs/qe/lib/python3.9/site-packages/beir/retrieval/search/dense/exact_search_multi_gpu.py", line 127, in search
    pool = self.model.start_multi_process_pool(self.target_devices)
  File "/root/miniconda3/envs/qe/lib/python3.9/site-packages/beir/retrieval/models/sentence_bert.py", line 35, in start_multi_process_pool
    p.start()
  File "/root/miniconda3/envs/qe/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/root/miniconda3/envs/qe/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/root/miniconda3/envs/qe/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/root/miniconda3/envs/qe/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/root/miniconda3/envs/qe/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "/root/miniconda3/envs/qe/lib/python3.9/multiprocessing/spawn.py", line 154, in get_preparation_data
    _check_not_importing_main()
  File "/root/miniconda3/envs/qe/lib/python3.9/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
		An attempt has been made to start a new process before the
		        current process has finished its bootstrapping phase.
		
		        This probably means that you are not using fork to start your
		        child processes and you have forgotten to use the proper idiom
		        in the main module:
		
		            if __name__ == '__main__':
		                freeze_support()
		                ...
		
		        The "freeze_support()" line can be omitted if the program
		        is not going to be frozen to produce an executable.
```

Hi @ET-yzk can you try again now? It should work!
Repository: beir-cellar/beir
Issue: 你们的IDCG计算是什么方式
Root cause: Documentation Deficiency

你好，我发现你们的IDCG方式的结果偏低，想问一下你们是怎么计算的

I google translated: `Hello, I found that the results of your IDCG method are low. I would like to ask how you calculated them.`

I do not calculate the IDCG metric in the BEIR repository, refer to [pytrec_eval](https://github.com/cvangysel/pytrec_eval) if you have concerns with the metric.
Repository: beir-cellar/beir
Issue: Question about the document index during evaluation on the test split of a dataset
GitHub labels: question

Hi,
I have a question about evaluation on the test splits of the datasets. How do we create a document index during this evaluation, for example in the case of MSMARCO? Do we construct it only from the corpus of documents from the test split, or do we combine train, eval, and test splits together to build a large document index?

If the second option is applied, don't we create some kind of a bias by showing specific documents to the model during training and then evaluating with an index which contains them?



This is a good question on what is information retrieval. A corpus is usually a large dump of documents/webpages which we consider as our source for retrieving information. Retrieval is only meaningful with larger corpora, which makes it challenging to evaluate. So, ideally all retrieval evaluation datasets have a single corpus with queries generated from multiple splits.

No, there is no bias to the as long as you are not seeing the test queries and qrels and start to fine-tune on the relevance judgments. If you have training dataset for a particular source, e.g., MSMARCO and evaluate on the same domain, you are doing an in-domain evaluation, instead of zero-shot which is main focus on the BEIR benchmark.

@thakur-nandan Thank you for the explanation. To be honest, when I was writing this issue in December, I thought that there is some division of the corpus into splits. I downloaded some of BEIR datasets and saw that this is not the case - there is one corpus for all the splits. Since my intuition is correct now, I am closing this issue :)
Repository: beir-cellar/beir
Issue: Multigpu dense eval return errors after corpus embedding
Root cause: External Dependency Breakage

Hi and thanks for you work. I'm trying to run the example for evaluation with multigpu ([https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/dense/evaluate_sbert_multi_gpu.py](https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/dense/evaluate_sbert_multi_gpu.py)) but i continuosly receive this error:

  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: Expected to be able to read 2594632 bytes for message body, got 2594620
[W110 10:20:47.593865407 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
/opt/conda/envs/campeses/lib/python3.11/multiprocessing/resource_tracker.py:254: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
  
  
  I tried to both downgrading and upgrading beir/sentence trasnformers and also datasets and pyarrow. 
  
  Did you have suggestion on how to solve this?

It should work now with the latest BEIR version. Can you retry @sirCamp?

Yes! Now It seems working! 
I would like to report that in the "evaluate_sbert_multi_gpu.py, line 69 there is a predefined cache Path which Is personal.

Thanks to fixing the issue!🎉
I love this repo!

Ah thanks for telling me this, I missed to remove this while testing! I'll update the repo!
Repository: beir-cellar/beir
Issue: Add BM25+RM3 as a baseline to re-ranking leaderboard
Root cause: Documentation Deficiency

We see this approach in the "[dense leaderboard](https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns/edit#gid=0)", but it would be good to add a row to the "[re-ranking leaderboard](https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns/edit#gid=867044147)" for easier comparisons against baselines.


Hi @joshdevins,
Nice suggestion! I would copy and paste the row to the reranking leaderboard.

Kind Regards,
Nandan Thakur

This has been updated here: https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns/edit#gid=867044147

Kind Regards,
Nandan
Repository: beir-cellar/beir
Issue: The BeIR/climate-fever dataset seems to be the wrong dataset
Root cause: External Dependency Breakage

Hi, thanks a lot for providing this great project!

I noticed that the published BeIR/climate-fever dataset does not align with the original CLIMATE-FEVER dataset (https://www.sustainablefinance.uzh.ch/en/research/climate-fever.html)

If I download it using...

load_dataset("BeIR/climate-fever", "corpus")

I see that it contains different features...

  "download_size": 2015475662,
  "features": {
    "_id": {
      "dtype": "string",
      "_type": "Value"
    },
    "title": {
      "dtype": "string",
      "_type": "Value"
    },
    "text": {
      "dtype": "string",
      "_type": "Value"
    }
  },


claim_label seems to be missing and the dataset size significantly differs from the original CLIMATE-FEVER dataset.

It seems to be a different dataset?







However, the queries file seems to be correct. The corpus file seems to be unrelated. Any advice?

(Same applies for this source: https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/climate-fever.zip)

Yes, the Climate-FEVER corpus was reused from FEVER, which contained abstracts of Wikipedia pages with a few additional abstracts manually inserted.

We need a newer Climate-FEVER version & corpus that can map the original corpus texts (shared in the original dataset) into a corpus for retrieval.
Repository: beir-cellar/beir
Issue: Unable to run colab notebook BEIR.ipynb
Root cause: Validation Gap

I run the cells sequentially and eventually get stuck on cell 7 with:

NameError: name 'faiss' is not defined

```python
from beir.retrieval.evaluation import EvaluateRetrieval
from beir.retrieval import models
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES # ERROR HERE

#### Dense Retrieval using SBERT (Sentence-BERT) ####
#### Provide any pretrained sentence-transformers model
#### The model was fine-tuned using cosine-similarity.
#### Complete list - https://www.sbert.net/docs/pretrained_models.html

model = DRES(models.SentenceBERT("msmarco-distilbert-base-v3"), batch_size=128)
retriever = EvaluateRetrieval(model, score_function="cos_sim")

#### Retrieve dense results (format of results is identical to qrels)
results = retriever.retrieve(corpus, queries)
```

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
[<ipython-input-8-3b3030b759cf>](https://localhost:8080/#) in <cell line: 0>()
      1 from beir.retrieval.evaluation import EvaluateRetrieval
      2 from beir.retrieval import models
----> 3 from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES
      4 
      5 #### Dense Retrieval using SBERT (Sentence-BERT) ####

2 frames
[/usr/local/lib/python3.11/dist-packages/beir/retrieval/search/dense/faiss_search.py](https://localhost:8080/#) in PQFaissSearch()
    195         num_of_centroids: int = 96,
    196         code_size: int = 8,
--> 197         similarity_metric=faiss.METRIC_INNER_PRODUCT,
    198         use_rotation: bool = False,
    199         **kwargs,

NameError: name 'faiss' is not defined


Hi @DhananjayAshok, thanks for reporting this! I should have fixed it now locally.
The codebase is updated in the main branch -- so try the local installation of the GitHub repository.

```python
$ git clone https://github.com/beir-cellar/beir.git
$ cd beir
$ pip install -e .
```

Thanks,
Nandan

Checked now that it works! Closing the issue.
Repository: beir-cellar/beir
Issue: Separate average score for zero-shot only
Root cause: Unimplemented Feature Gap

Between the two [leaderboard](https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns/edit#gid=867044147) tabs, they are laid out slightly differently. It would be nice to normalize these or put them on a single tab. If anything, just separating the "average" score to only include zero-shot methods in the "re-ranking leaderboard" would help compare across tabs. Basically the same view as is in the paper is preferred.

Hi @joshdevins, 

Yes, I need to change the layout of the reranking table comparable to the dense one, it has been on my todo list now since forever. Will do that soon and update you once it has been done!

Kind Regards,
Nandan



This has been updated here: https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns/edit#gid=867044147

Kind Regards,
Nandan

Look amazing, thank you!
Repository: beir-cellar/beir
Issue: NFCorpus dataset missing queries
Root cause: Validation Gap

It seems that some queries are missing from the `queries.jsonl` file in the NFCorpus dataset. For example, the query `PLAIN-2` in `test.tsv` cannot be found in `queries.jsonl`.

Additionally, in the paper "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models", could you confirm which dataset split (train, dev, or test) was used to calculate the scores in Table 2? I am trying to reproduce some results using BM25. Thank you!
Repository: beir-cellar/beir
Issue: Importlib.util needs to be directly imported
Root cause: External Dependency Breakage

On Python 3.12, attempting to import `DenseRetrievalExactSearch`,
```
from beir.retrieval.search.dense.exact_search import DenseRetrievalExactSearch as DRES
```
 throws an AttributeError:
```
Traceback (most recent call last):
  File "/workspace/exa-bin-quant/bin_quant/encode.py", line 3, in <module>
    from beir.retrieval.search.dense.exact_search import DenseRetrievalExactSearch as DRES
  File "/workspace/exa-bin-quant/.venv/lib/python3.12/site-packages/beir/retrieval/search/dense/__init__.py", line 3, in <module>
    from .exact_search import DenseRetrievalExactSearch
  File "/workspace/exa-bin-quant/.venv/lib/python3.12/site-packages/beir/retrieval/search/dense/exact_search.py", line 8, in <module>
    if importlib.util.find_spec("faiss") is not None:
       ^^^^^^^^^^^^^^
AttributeError: module 'importlib' has no attribute 'util'
```

It seems that `importlib` doesn't expose `util` by default. See [this Python stdlib thread](https://discuss.python.org/t/python3-11-importlib-no-longer-exposes-util/25641), for example.
Repository: beir-cellar/beir
Issue: Possibility of including TREC DL 2019 and TREC DL 2020
Root cause: Documentation Deficiency

Hi,
As the BEIR library sees wider adoption, I’m wondering if it would be possible to add more datasets to better support it. A typical candidate would be TREC DL 2019 and TREC DL 2020. Since we already have MS MARCO, integrating these two datasets shouldn’t be too difficult. 

I greatly appreciate your help. If you don’t have time, maybe I could process them into the same format, and then you could just upload them to the website.

Many thanks.

I’ve created a [code repository](https://github.com/liyongkang123/extended_beir_datasets) to deal with this. 

Anyone who needs it can use it.

Hi @liyongkang123, the TREC-DL 2019 is present in the `msmarco` dataset under the `test` split.

Regards,
Nandan

Hi @thakur-nandan  When using the BEIR msmarco for benchmarking. Should we use "dev" or "test"?
Repository: beir-cellar/beir
Issue: HFDataLoader returns HuggingFace Dataset, but EvaluateRetrieval expects dict (AttributeError: 'Dataset' object has no attribute 'keys')
Root cause: Interface Contract Mismatch

### Description

I’m running into an internal incompatibility when using `HFDataLoader` together with
`EvaluateRetrieval` for dense retrieval.

This is confusing because the setup closely follows BEIR’s own example using the HF
loader, but it fails at runtime due to mismatched data structures.

### Code snippet

```python
from beir.datasets.data_loader_hf import HFDataLoader
from beir.retrieval.models import SentenceBERT
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES
from beir.retrieval.evaluation import EvaluateRetrieval

corpus, queries, qrels = HFDataLoader(
    hf_repo=REPO_ID,
    hf_repo_qrels=REPO_ID,
    corpus_file="corpus.jsonl",
    query_file="queries.jsonl",
    streaming=False
).load(split="test")

beir_model = SentenceBERT(
    "sentence-transformers/static-similarity-mrl-multilingual-v1"
)

model = DRES(beir_model, batch_size=128)
retriever = EvaluateRetrieval(model, score_function="dot")

results = retriever.retrieve(corpus, queries)
```

### Error

```text
AttributeError: 'Dataset' object has no attribute 'keys'
```

Traceback points to:

```python
# beir/retrieval/search/dense/exact_search.py
query_ids = list(queries.keys())
```

### What’s going wrong

- `HFDataLoader.load()` returns Hugging Face `Dataset` objects for `corpus` and `queries`
- `EvaluateRetrieval` / `DenseRetrievalExactSearch` assumes dict-like inputs
  (`queries.keys()`, indexing by ID)
- This results in an immediate runtime failure

This is especially surprising because the HF loader is explicitly provided to reduce
RAM usage, and its usage is demonstrated in the repository itself:

https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/dense/evaluate_sbert_hf_loader.py

### Expected behavior

One of the following:

- `EvaluateRetrieval` should natively support Hugging Face `Dataset` objects returned
  by `HFDataLoader`, or
- The HF loader example should clearly document that users must convert datasets to
  dicts before retrieval, or
- `HFDataLoader` should optionally return dicts in the format expected by the
  retrieval pipeline

Right now the API contracts don’t line up, even though they are presented as compatible. Or I am missing something

### Why this matters

The HF loader is very useful for large corpora, but at the moment it’s not actually
usable with the standard dense retrieval pipeline without custom glue code. This makes
the example misleading and the failure mode non-obvious.