zhiminy/EvalEng huggingface dataset:
Repository: beir-cellar/beir
Issue: No Quora & CQADupstack in leaderboard?
Root cause: Documentation Deficiency

Hey,
why are Quora & CQADupstack excluded from the [leaderboard](https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns/edit#gid=0)?

Hi @Muennighoff 
Thanks for reporting this. The spreadsheet was just an intermediate solution, not sure why Quora & CQA is missing.

I currently work on an updated leaderboard that will be launched soon, which will include these datasets

Hi @Muennighoff, Thanks for pointing it out. It has been there in my TODO list, More models started to pour in and I almost forgot about it. I will update the table soon and add the new numbers.

Kind Regards,
Nandan Thakur
Repository: beir-cellar/beir
Issue: Question about ideal architecture for deep IR

Hi again @NThakur20! I've got an interesting search project which consists of a golden set of search queries and their results, for a financial services domain search application. One of the search types is for financial analysts, e.g. a partial analyst name search query which is answered by the search engine with the full contact particulars for that analyst. Using a fine tune trained T5.1 large parameter model I am achieving 97%+ classification accuracy for observed searches but the issue here is generalization to new searches where the analyst exists only in the database and the model needs to generate the response based on unsupervised contact data that's in the contact analyst database. So the thought was to either train an MS MARCO T5 model in an unsupervised fashion on the contact database in hopes that it generalizes to unobserved search queries, or to populate a deep IR pipeline with those records and use that for the analyst contact retrieval. Is this a reasonable use case with BEIR?
Repository: beir-cellar/beir
Issue: Error in `beir/beir/retrieval/search/sparse/sparse_search.py`
Root cause: Algorithmic Error

https://github.com/UKPLab/beir/blob/568f4c34fa0be1901e2aaa8479978c9e54a1e377/beir/retrieval/search/sparse/sparse_search.py#L28-L34

Hello, 

I think there is an error in `sparse_search.py`; only the first query in each batch is evaluated. 

Thank you!

Indeed! An easy fix would be to remove the step size `self.batch_size`?
Repository: beir-cellar/beir
Issue: Missing datasets
Root cause: Documentation Deficiency

Hi, it seems that some datasets that are on the leaderboard are missing from https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/

For example:
* `bioasq`
* `signallm`
* `trec-news` 

Is there a reason for removing those datasets?

Hi @thigm85, 

I do not have permissions to freely open-source and provide these datasets due their individual licence policies. You can check on how to replicate them yourself from the README file here: https://github.com/UKPLab/beir/tree/main/examples/dataset.

Kind regards,
Nandan

Thanks for the quick response. I got it :)
Repository: beir-cellar/beir
Issue: ValueError when running `evaluate_bm25.py`
Root cause: Validation Gap

Hi, I was trying to run your `evaluate_bm25.py` baseline, but I got the following error. There may be some problem with `elasticsearch`. Could you please help me fix it?
```
2022-02-17 02:38:34 - Loading Queries...
2022-02-17 02:38:34 - Loaded 300 TEST Queries.
2022-02-17 02:38:34 - Query Example: 0-dimensional biomaterials show inductive properties.
2022-02-17 02:38:34 - Activating Elasticsearch....
2022-02-17 02:38:34 - Elastic Search Credentials: {'hostname': 'localhost', 'index_name': 'scifact', 'keys': {'title': 'title', 'body': 'txt'}, 'timeout': 100, 'retry_on_timeout': True, 'maxsize': 24, 'number_of_shards': 1, 'language': 'english'}
Traceback (most recent call last):
  File "evaluate_bm25.py", line 64, in <module>
    model = BM25(index_name=index_name, hostname=hostname, initialize=initialize, number_of_shards=number_of_shards)
  File "/anaconda/envs/beir/lib/python3.8/site-packages/beir/retrieval/search/lexical/bm25_search.py", line 22, in __init__
    self.es = ElasticSearch(self.config)
  File "/anaconda/envs/beir/lib/python3.8/site-packages/beir/retrieval/search/lexical/elastic_search.py", line 34, in __init__
    self.es = Elasticsearch(
  File "/anaconda/envs/beir/lib/python3.8/site-packages/elasticsearch/_sync/client/__init__.py", line 312, in __init__
    node_configs = client_node_configs(
  File "/anaconda/envs/beir/lib/python3.8/site-packages/elasticsearch/_sync/client/utils.py", line 101, in client_node_configs
    node_configs = hosts_to_node_configs(hosts)
  File "/anaconda/envs/beir/lib/python3.8/site-packages/elasticsearch/_sync/client/utils.py", line 141, in hosts_to_node_configs
    node_configs.append(url_to_node_config(host))
  File "/anaconda/envs/beir/lib/python3.8/site-packages/elastic_transport/client_utils.py", line 198, in url_to_node_config
    raise ValueError(
ValueError: URL must include a 'scheme', 'host', and 'port' component (ie 'https://localhost:9200')
```

As hostname I think you must use http://localhost (or http://localhost:9200), not just localhost

Thank you so much! I change the hostname to `http://localhost:9200` and it works. But when I run it to evaluate BM25, I get different scores at different runs. For example, the NDCG@10 score ranges from 0.64~0.67 on scifact dataset. Do you know why? Is there any randomness in the BM25 algorithm?

This was addressed in https://github.com/UKPLab/beir/issues/58

Not sure if the latest release already includes this. You can either update BEIR to use the latest version from the GIT. Or you add a sleep after you index the documents in your code.

I see. It's fixed in the `beir` code but not yet included in the `examples`. I add a sleep time and eventually get a consistent score.

Hi @jordane95,

Yes soon with our next pip update, hopefully, this should not be an issue anymore and consistent scores should be visible with Elasticsearch BM25. Thanks for notifying me!

Kind regards,
Nandan Thakur
Repository: beir-cellar/beir
Issue: Script to generate leaderboard metrics
Root cause: Documentation Deficiency

Can I find the complete script used to generate [this leaderboard](https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns/edit#gid=867044147) somewhere? I saw snippets such as [benchmark_bm25.py](https://github.com/UKPLab/beir/blob/main/examples/benchmarking/benchmark_bm25.py) but not a full-scale script that includes elastic search config and all.

I am implementing a [BEIR compatible Vespa version](https://github.com/thigm85/beir/blob/main/examples/benchmarking/benchmark_lexical_vespa.py) that I plan to submit as a PR soon. I am, however, finding different results between my BM25 metrics and the elastic BM25 results from the leaderboard. 

Generating results side by side would be great to debug my implementation.

Hi @thigm85, 

I used this python script here: [evaluate_bm25.py](https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/lexical/evaluate_bm25.py) to generate scores for Elasticsearch BM25 in the leaderboard.

However as notified recently in #58, some metrics might be different from what I got in the leaderboard. I can rerun them soon and update the latest metrics for the leaderboard. However, since these changes have not been currently reflected in the latest pip version, you can download the development branch locally and use  [evaluate_bm25.py](https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/lexical/evaluate_bm25.py) to get the accurate Elasticsearch BM25 scores. 

Kind Regards,
Nandan Thakur

Thanks for the reply

> Hi @thigm85,
> 
> I used this python script here: [evaluate_bm25.py](https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/lexical/evaluate_bm25.py?rgh-link-date=2022-02-18T21%3A33%3A31Z) to generate scores for Elasticsearch BM25 in the leaderboard.
> 
> However as notified recently in #58, some metrics might be different from what I got in the leaderboard. I can rerun them soon and update the latest metrics for the leaderboard. However, since these changes have not been currently reflected in the latest pip version, you can download the development branch locally and use [evaluate_bm25.py](https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/lexical/evaluate_bm25.py?rgh-link-date=2022-02-18T21%3A33%3A31Z) to get the accurate Elasticsearch BM25 scores.
> 
> Kind Regards, Nandan Thakur

Hi @NThakur20,

I want to confirm if you have updated the metrics. I fail to reproduce them with the latest code.
Repository: beir-cellar/beir
Issue: TypeError when running Quick Example
Root cause: Interface Contract Mismatch

Thanks for the great work.

When I run the Quick Example, I got a TypeError:

```
Traceback (most recent call last):
  File "demo.py", line 34, in <module>
    ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)
  File "/home/renruiyang/anaconda3/envs/beir/lib/python3.6/site-packages/beir/retrieval/evaluation.py", line 63, in evaluate
    evaluator = pytrec_eval.RelevanceEvaluator(qrels, {map_string, ndcg_string, recall_string, precision_string})
TypeError: Unable to resolve all measures.
```

Could you help me solve this error? 

Thanks a lot!

Hi @ReyonRen, 
I'm unsure why the error occurs! When I seem to run the custom example, it works fine for me. 
Here is a Google colab demo: https://colab.research.google.com/drive/1JKySGIMPRAaryVoL3BKkkfO2IRWtkxwB?usp=sharing

Let me know if you still find any issues!

Kind Regards,
Nandan Thakur
Repository: beir-cellar/beir
Issue: Training configuration for ColBERT

Thank you for sharing this work!

Could you share the training configuration for ColBERT retriever?
- batch size, optimizer, and learning rate used for training
- training triples (Did you use official train triples of MSMARCO without ANCE-style hard negative mining?)

Thanks in advance

Hi @jihyukkim-nlp, 

For training the ColBERT retriever, we used the same training configurations as the original default training code mentioned here: https://github.com/stanford-futuredata/ColBERT#training, with just one change of ``--doc_maxlen 300`` instead of ``180``. 

Our training triplets were official MSMARCO train triplets.

Kind Regards,
Nandan Thakur

Thanks for the heads-up :) 

If it helps, you can find my ColBERT model checkpoint here: https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/models/ColBERT/msmarco.psg.l2.zip

Kind Regards,
Nandan Thakur

Thank you for sharing. 
I found the url in the paper. This helped me a lot. 

But, I wanted to further analyze training process of ColBERT under the same training configuration.
Now, I can do this. Thanks!

Best regards,
Jihyuk Kim

Hi @thakur-nandan , is there a reference on how many NUM_PARTITIONS for ColBERT faiss search is used for each BEIR datasets ? The default is set to 32768 in their original repo, but 96 was given in your evaluation script (https://github.com/thakur-nandan/beir-ColBERT/blob/91190882deac1792c78b3c33d51be9edaa9c6805/evaluate_beir.sh#L26) . I wonder if that was changed for each datasets.
Repository: beir-cellar/beir
Issue: Question on the BM25 implementation
Root cause: Documentation Deficiency

Hi, I read your code regarding BM25. I find that it's backed by `elasticsearch`. But I'm not so familiar with it. May I ask you a few questions about it? 
1. What tokenizer does `elasticsearch` use? A BERT-like subword tokenizer, or a custom one?
2. Can `elasticsearch` in your implementation handle BERT-tokenized text? For example, assume a sentence "I like beir" corresponds to "101 146 1176 1129 3161 102" after BERT tokenizer (still in string format), then we feed the string of IDs to BM25, can it be splited to ["101", "146", "1176", "1129", "3161", "102"] where each word ID is treated as a single word to be indexed?
Repository: beir-cellar/beir
Issue: Datasets without Download Links
Root cause: Documentation Deficiency

Hello,
 Thanks for your great work  !
 I am just curious how can we reproduce some results and get the bioasq, signal1m etc  ...  datasets to work with beir.
 (currently testing our own model on beir  ! )



Hello @sclincha ,

I need to still add instructions for datasets not publicly available yet in the benchmark. 
Few are a bit complicated (Signal1M) would add details on how to implement them hopefully soon in the benchmark.

If you want them immedeately, please write me an email (nandant@gmail.com) and I can personally share instructions for reproduction.

ok Great I will write to you !
Repository: beir-cellar/beir
Issue: Eval script breaks for custom evaluations when query does not have any hits
Root cause: Validation Gap

Steps to reproduce:

1. Setup:
```
git clone git@github.com:UKPLab/beir.git
cd beir
conda create --name beir python=3.7
conda activate beir
pip install -e .
```

2. Run script `sample.py`:
```
"""
Sourced from examples/retrieval/evaluation/lexical/evaluate_bm25.py
"""

from beir import util
from beir.datasets.data_loader import GenericDataLoader
from beir.retrieval.evaluation import EvaluateRetrieval
from beir.retrieval.search.lexical import BM25Search as BM25

import pathlib, os

dataset = "nfcorpus"
url = "https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/{}.zip".format(dataset)
out_dir = os.path.join(pathlib.Path(__file__).parent.absolute(), "datasets")
data_path = util.download_and_unzip(url, out_dir)

corpus, queries, qrels = GenericDataLoader(data_path).load(split="test")
hostname = "http://0.0.0.0:9200" 
index_name = "nfcorpus_bug" 
initialize = True

number_of_shards = 1
model = BM25(index_name=index_name, hostname=hostname, initialize=initialize, number_of_shards=number_of_shards)
retriever = EvaluateRetrieval(model)
results = retriever.retrieve(corpus, queries)

for metric in ["mrr", "recall_cap", "hole", "accuracy"]:
    retriever.evaluate_custom(qrels, results, retriever.k_values, metric=metric)
```

Stack trace:
```
Traceback (most recent call last):
  File "sample.py", line 78, in <module>
    retriever.evaluate_custom(qrels, results, retriever.k_values, metric=metric)
  File "/Users/narayan/OSS/beir/beir/retrieval/evaluation.py", line 92, in evaluate_custom
    return mrr(qrels, results, k_values)
  File "/Users/narayan/OSS/beir/beir/retrieval/custom_metrics.py", line 22, in mrr
    for rank, hit in enumerate(top_hits[query_id][0:k]):
KeyError: 'PLAIN-510'
```

Notes:
I am running ElasticSearch v7.17.0. via docker locally.

Preliminary Investigation:
I think when a query does not have any hits in ES we should add an empty `scores` dict to our `self.results` in the below code so our eval scripts do not raise KeyError if they don't find a query in the `results`.

https://github.com/UKPLab/beir/blob/a55552db70f37102352fd5c21b4e811516659a55/beir/retrieval/search/lexical/bm25_search.py#L49-L54

Hi @narayanacharya6,

Thanks for reporting this issue. Indeed looking at the error in your stack trace, I believe what you mentioned is happening. Elasticsearch may not return hits for a query it did not find at all. This is by default taken care of in evaluation using ``pytrec_eval`` for Recall, Precision, NDCG, etc. but not in my custom definition of MRR and possibly others. One could add zero-scores to ``self.results``, but a better solution would be to handle it at the evaluation step of each metric. Will work on it and update the dev branch soon!

Kind Regards,
Nandan Thakur
Repository: beir-cellar/beir
Issue: On the dependency `faiss`
Root cause: Architectural Constraint

Hi, I noticed that your `beir==0.2.3` depends on `faiss-cpu`. While in your NeurIPS'2021 paper, you benchmarked several dense retrieval models on GPU. Did you use `faiss-gpu` for that? Have you considered update the dependency of `beir` to `faiss-gpu`?

Hi @jordane95,

We benchmark using exact-search and not faiss for the NeurIPS paper which used the GPU. You can check this example: https://github.com/UKPLab/beir/blob/main/examples/benchmarking/benchmark_sbert.py

Updating faiss-cpu to faiss-gpu could be interesting, however last time I checked we find issues in downloading faiss-gpu using PyPI. We decided to go with faiss-cpu for now for others to work on it easily. It will be interesting to integrate faiss-gpu in the future!

Kind regards,
Nandan
Repository: beir-cellar/beir
Issue: Why index the queries together with the corpus?
Root cause: Documentation Deficiency

I noticed that some datasets (e.g., Arguana) include the queries on the documents fed to the search engine. In addition, the evaluation code ignores the scores of the indexed queries. Is there a reason for such a choice?

Hi @thigm85, yes this would be there for ArguAna and Quora datasets. This is due to the inherent nature of these tasks and how originally the datasets were created. 

ArguAna task involves retrieval of a counterargument (of passage length) for an input argument (also of passage length). Whereas also for Quora, the task involves retrieval of duplicate query (usually of sentence length) for an input query (also of sentence length). So only for both of these datasets, it is possible a query (i.e. a passage for ArguAna and query for Quora) can be a possible answer for a different query in the dataset and hence can be used interchangeably.

The reason for ignoring the scores is because for an input query you might retrieve itself i.e. the same query present within the collection. So we move ahead and remove these queries.

Except, for these two datasets you would not find this happening in other datasets, as the query is usually of sentence length whereas the corpus involves documents of (passage length) which cannot be used interchangeably.

Kind Regards,
Nandan Thakur

Thanks for the context @NThakur20.
Repository: beir-cellar/beir
Issue: End-Task Labels
Root cause: Unimplemented Feature Gap

Thank you for the great repository! 

I noticed that for datasets where there is an end-task (e.g. classification for QQP or QA for Natural Questions), that the data files do not have these end-task annotations (e.g. the answer for NQ instances). Is there any easy way to recover these end-task labels for the different datasets?



Hi @anthonywchen,

Sadly BEIR was focused on IR, hence we do not have these end-task annotations. For a few datasets, we might have some additional detail on their end-task in the metadata fields. However, I believe this is not present for NQ.

Kind Regards,
Nandan Thakur

Thank you for the help! closing the issue.
Repository: beir-cellar/beir
Issue: Strange NDCG@10 for Touche-2020 on the BEIR leaderboard
Root cause: Configuration Error

I noticed that the NDCG@10 for Touche-2020 on the BEIR leaderboard is around 0.60 for elastic bm25. 

Is it correct to assume that Touche-2020 is represented by the dataset named "webis-touche2020"? If yes, I just ran the elastic search bm25 for it, and I found NDCG@10 at around 0.35, which is similar to what I got with Vespa.

Any thoughts?

Recently I had the same problem as you. It is linked to the fact that there are two versions of the webis-touche2020, with the newer one being the one used now (around 0.35) and the older one having better scores (0.6). In issue #11 it seems that the older version was kept, but then in issue #40 the new versions seems to have taken over as the default one, making some numbers of the benchmark obsolete (reranking page mostly, sparse and dense seem to be using the new version).

Got it. Thanks for the reply @cadurosar. 

Does the same issue happen with msmarco? I just ran the Elastic Search BM25 with `msmarco` and the NDCG@10 is around 0.45 instead of the 0.22 as reported on the BEIR leaderboard. Is that correct @NThakur20? 

Hi, @thigm85 and @cadurosar, 

Yes, the webis-touche authors contacted us with problems in their version v1 dataset. So we kept scores on the v2 version (with no annotation errors). Some scores in the leaderboard might not be changed like @cadurosar mentioned. The leaderboard is getting revamped and soon will have the latest updated scores on it.
 
Regarding MSMARCO, I think you would have evaluated the ``test`` set. That's why probably you get NDCG@10 of around 0.45, however, you should evaluate the ``dev`` set instead where you should get an identical score mentioned in the leaderboard.

Kind Regards,
Nandan Thakur

Hi @NThakur20, thanks for the clarification. Does the leaderboard use the `dev` set for all the datasets or only for MS MARCO?

``dev`` set only for MSMARCO, rest are the ``test`` sets.
Repository: beir-cellar/beir
Issue: Question about details/scripts for generating the datasets
Root cause: Documentation Deficiency

Given that the framework tries to get all datasets into a single consistent structure, I am trying to find details/scripts about how some of the available datasets were generated from their raw form. I could not find it in this repo.

Also, somewhat related to datasets can you please clarify this paragraph from the paper:

```
Finally, we notice that there can be a strong lexical bias present in datasets included within the
benchmark, likely as lexical models are pre-dominantly used during the annotation or creation of
datasets. This can give an unfair disadvantage to non-lexical approaches. We analyze this for the
TREC-COVID [63] dataset: We manually annotate the missing relevance judgements for the tested
systems and see a significant performance improvement for non-lexical approaches.
```

Is the TREC-COVID datasets available via BEIR with these manual annotations or not?
If yes, is there a raw version of the TREC-COVID dataset available via BEIR?
If no, are these manual annotations available somewhere I can access?

Hi @narayanacharya6, 

The raw form of the datasets is not made available as it defeats the purpose. BEIR provides a single unified format for all datasets. As these all datasets are available publicly, you can go to their original repository for each dataset and get the raw form of the datasets.

Yes, the TREC-COVID annotated dataset is also available here: https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/trec-covid-beir.zip as mentioned in Table 7 (Appendix) in the arxiv version of the paper: https://arxiv.org/abs/2104.08663.

Kind Regards,
Nandan Thakur

I was trying to look for how you converted the raw datasets into the single unified format. Essentially, are there are scripts that convert these datasets from their raw format into the processed datasets available via BEIR?

Also, thanks for pointing me to the `trec-covid-beir` version of the dataset!



Hi @narayanacharya6, 

Sadly I do not have conversion scripts anymore. Essentially, the conversion is easy as I take the original dataset and wrote simple preprocessing scripts to convert them into a corpus ``dict`` with ``doc_id`` as key and title and text fields and similar for the query as well with ``query_id`` as the key. 

Kind Regards,
Nandan Thakur

No worries. Closing this issue.
Repository: beir-cellar/beir
Issue: How did you evaluate ColBERT on BEIR benchmark?
Root cause: Documentation Deficiency

Hello,

I have a little question regarding the ColBERT evaluation. ColBERT is trained on MS MARCO, where the passage in `collection.tsv` only contains one field. They prepend it with a special [D] token before feeding into the model to get token level dense representation.

However, for datasets in BEIR, passage in `corpus.jsonl` also contains a `title` field besides the `text` field.

May I know how did you evaluate ColBERT on your BEIR benchmark? In other words, how did you handle the title and text field in your BEIR datasets to fit the ColBERT input?

Hi @jordane95, I just released the ColBERT evaluation code on the BEIR benchmark today, you can find more details in this repository: https://github.com/NThakur20/beir-ColBERT.

I simply concatenate the title and text with a single whitespace and treat them as passages. More details can be found within the repository!

Kind Regards,
Nandan Thakur