zhiminy/EvalEng huggingface dataset:
Repository: beir-cellar/beir
Issue: New Dataset inclusion to BEIR benchmarks
Root cause: Unimplemented Feature Gap

Hi, 
     I would like to request the inclusion of [RELiC](https://aclanthology.org/2022.acl-long.517/) dataset in the BEIR benchmark for retrieval. I work with the lab at UMass Amherst from which the Authors of RELiC are and requesting the inclusion of a subset of the dataset on their behalf.

## Task Overview
> We collect a large-scale dataset (RELiC) of 78K literary quotations and surrounding critical analysis and use it to formulate the novel task of literary evidence retrieval, in which models are given an excerpt of literary analysis surrounding a masked quotation and asked to retrieve the quoted passage from the set of all passages in the work.

## Specifications
We make a subset of the test set available to check the zero-shot retrieval capability of retrievers. The test set consists of 5 books formatted in accordance with BEIR. The dataset differs from the complete RELiC dataset in the following ways

1. The RELiC dataset has a combination of continuous masked quotations of different lengths, ranging from 1-5. We include only masked quotations of 1 in this dataset for retrieval. 
2. By construction, the dataset has both a preceding context and a succeeding context for retrieval. However, the norm with retrievers and benchmarks is to have a single input sentence for retrieval. To accommodate for this, we drop the succeeding contexts and pose RELiC as a retrieval problem using only the preceding context of a maximum of 4 sentences. This can be referred to as the 4/0 setting. 

## Dataset 
The dataset can be found [here](https://drive.google.com/file/d/1KXeEGVKHQR2-XnPtQc4LFuzcfRNHU2BK/view?usp=sharing). At the top level, there are 5 folders, each corresponding to a book from the test set. 

## LeaderBoard
A public Leader board is [available](https://relic.cs.umass.edu/leaderboard.html) for this task for both the zero-shot and trained settings.
Repository: beir-cellar/beir
Issue: HFDataLoader is incompatible with CQADupstack
Root cause: Unimplemented Feature Gap

If I see correctly cqadupstack is the only dataset not uploaded to the Hub, i.e. there's [only its qrels](https://huggingface.co/datasets/BeIR/cqadupstack-qrels).
This prevents using it with DPRES :(
Would be great if it could be uploaded!
Repository: beir-cellar/beir
Issue: Support for revisions in HFDataLoader
Root cause: Unimplemented Feature Gap

Would be great if the [HFDataLoader](https://github.com/beir-cellar/beir/blob/c3334fd5b336dba03c5e3e605a82fcfb1bdf667d/beir/datasets/data_loader_hf.py#L89) would support dataset revisions.

Also imo it would be much better to merge the qrels HF repos with the regular ones & load the entire dataset via `.load_dataset()` using a dataset script on the repos.
Repository: beir-cellar/beir
Issue: Add example notebook for benchmarking BEIR using an Sentence Transformer(SBERT) Custom Model 
Root cause: Documentation Deficiency

Sentence Transformers is one of the easiest libraries to train models for STS/other tasks.Given the fact that people would consequently use SBERT with BEIR given the authors involvement in both it would be natural to look for an example implementation as to how to use them together for benchmarking.

@nreimers
Repository: beir-cellar/beir
Issue: Error loading model in `query_gen_and_train.py`
Root cause: Interface Contract Mismatch

Hello,

I'm working on the GenQ setting and find that the model loading part seems incorrect. In `query_gen_and_train.py`, the model loading part is like:

```python
#### Provide any sentence-transformers model path
model_path = "bert-base-uncased" # or "msmarco-distilbert-base-v3"
retriever = TrainRetriever(model_path=model_path, batch_size=64, max_seq_length=350)
```

However, the `TrainRetriever` class doesn't have the argument `model_path`. It seems that the error was introduced in [this commit](https://github.com/UKPLab/beir/commit/78cb3b03f4f709149e3bdf6b332020be7c2ed628).

Shi Yu



Hi @Yu-Shi,

Thanks for bringing this to my notice. 
The example has been updated.

Kind Regards,
Nandan
Repository: beir-cellar/beir
Issue: Quora titles are empty on Huggingface
Root cause: Configuration Error

When I load the `BeIR/quora` dataset through huggingface, it looks like all of the queries are empty:

```python
>>> import datasets
>>> d = datasets.load_dataset('BeIR/quora', 'corpus')
>>> d['corpus'][0]
{'_id': '1', 'title': '', 'text': 'What is the step by step guide to invest in share market in india?'}
```



Update: I found some queries here, but they don't seem to match the corpus:

```datasets.load_dataset('BeIR/quora', 'queries')```

Ah I figured it out.
Repository: beir-cellar/beir
Issue: Training script/configuration for DocT5query

Hi all, thank you for sharing this awesome repo!

I am wondering if the training script and training configurations (e.g., batch size, learning rate, iterations, etc) for 
* [BeIR/query-gen-msmarco-t5-base-v1](https://huggingface.co/BeIR/query-gen-msmarco-t5-base-v1)
* [BeIR/query-gen-msmarco-t5-large-v1](https://huggingface.co/BeIR/query-gen-msmarco-t5-large-v1)
can be shared in addition to the checkpoints.

(maybe via model card's `training_args.bin`?)

Best regards,
Jihyuk
Repository: beir-cellar/beir
Issue: supporting chinese dataset retrieval?
Root cause: Documentation Deficiency

Hello, Does Beir support chinese dataset retrieval？

Hi @yajingsunno, 

BEIR is agnostic of the language of the task. You can store the documents in the format which you specify and provide a multilingual model for chinese retrieval.

Kind Regards,
Nandan Thakur

哈喽，
请问你安装上了吗？我为什么安装的时候安装到pytrec_eval总会报错呢

> 哈喽， 请问你安装上了吗？我为什么安装的时候安装到pytrec_eval总会报错呢

hello想问问兄弟你解决了嘛，我这边pytrec_eval也安装报错了
Repository: beir-cellar/beir
Issue: Proposal: Describe metrics in Wiki with more details

I would like to [understand the metrics](https://github.com/beir-cellar/beir/wiki/Metrics-available) better and therefore it would be nice if you could add more infos to the Wiki, e.g. a formula and explanation of how each metric is calculated.

I will research them for myself now and add a proposal to this issue. If anyone knows a good source please add it here.

I decided to turn it into a blog post: https://fabianhertwig.com/blog/information-retrieval-metrics/

If you want to pull anything from that to put it into the wiki then feel free to do so.

Thank you! This is quite informative.

Will update the Wiki Metrics to include portions from the blogpost for each metrics! Credits would be given as well! :)

Kind Regards,
Nandan Thakur
Repository: beir-cellar/beir
Issue: MonoT5 .predict gives NaN
Root cause: Algorithmic Error

Trying to use MonoT5 3B on some custom reranking tasks, the gist of the code is 

```
model_name = "castorini/monot5-3b-msmarco-10k"
cross_encoder_model = MonoT5(model_name, token_false='▁false', token_true='▁true')

query, doc = "some query", "some doc"
cross_encoder_model.predict([(query, doc)], batch_size=1)
```

and returns `[nan]`

is this an expected case? should we treat this as a score of `0` in this case?

Hi, have you found the solution for this?

Hi, It's not exactly the same issue, but I've noticed similar issues on the _reranking_ task.
For example, I'm using the following code.
```
def do_evaluation(queries, qrels, corpus, results=None):
    k_values = [1,5,10,20,50,100]
    retriever = EvaluateRetrieval()
    from beir.reranking.models import CrossEncoder, MonoT5
    from beir.reranking import Rerank
    cross_encoder_model = MonoT5(mode, token_false='▁false', token_true='▁true')
    print(f"Loading cross-encoder model from: {cross_encoder_model.model.config._name_or_path}")
    reranker = Rerank(cross_encoder_model, batch_size=256)
    results = reranker.rerank(corpus, queries, results, top_k=100) # outputs nan scores to results
    results = remove_nan(results) # manually assign score due to bug
    ndcg, _map, recall, precision = retriever.evaluate(qrels, results, k_values)
```

It may be too late, but the **order** inside the `results` **is preserved**, so the following code can be used as a quick workaround:

```
def remove_nan(results):
    new_res = {}
    for query_key in results.keys():
        out = {}
        for i, corpus_key in enumerate(results[query_key].keys()):
            out[corpus_key] = 100 - i
        new_res[query_key] = out
    return new_res
```
Using this code, it output correct scores for ndcg, recall.. and so on.
Repository: beir-cellar/beir
Issue: Reproducability for all-minilm-l6-v2
Root cause: Documentation Deficiency

Hello,
had a look at the results of comparing gpt-3, google and sentence transformer embedding models against each other on a few benchmarks from BEIR (https://docs.google.com/spreadsheets/d/1cbkJAinXVKIzf6oZ1lAR_feD4rw1vXFSjCjb0AJ83jI/edit#gid=0). I'm just wondering how the results for the sentence transformer models could be reproduced.

I used the evaluation SBERT [script](https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/dense/evaluate_sbert.py) with [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) and end up with a score of 0.18 for SciDocs and around 0.45 in average for CQADupStack. While also the [leaderboard](https://eval.ai/web/challenges/challenge-page/1897/leaderboard) shows results in the range of 0.14 and 0.21 for SciDocs both results are far away from the scores mentioned in the spreadsheet above.

Can someone help me and point me into the right direction?



Hi @Manikseam, The [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) is sentence-level model which suffers on retrieval tasks in BEIR, which require passage-level understanding for retrieval of relevant passages for a given query. 

The first spreadsheet which you shared contains tasks only for sentence-level where all-MiniLM-L6-v2 perform well. 

I would suggest you can have a look at models such as [TAS-B](https://huggingface.co/sentence-transformers/msmarco-distilbert-base-tas-b) to achieve scores mentioned in the BEIR leaderboard.

Kind Regards,
Nandan Thakur
Repository: beir-cellar/beir
Issue: Robust04 preprocessing
Root cause: Documentation Deficiency

Hi, I saw from https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns/edit#gid=0 that Robust04 has been added to the BEIR leaderboard. Thanks for providing the results! Meanwhile, I was wondering what preprocessing is made for the dataset. For example, which fields of the topics are used for constructing the queries (title, desc, or narr)? Which parts of the documents are included (Headline, Text, Date, etc)? I'm asking because I tried to evaluate ANCE (the publicly released checkpoint) on Robust04 with minor preprocessing, but the ndcg@10 score is around 0.33, which is much lower than 0.39 as reported in the leaderboard. Thanks a lot!

Hi @Ji-Xin,
I specifically followed the AllenAI package on``ir_datasets`` for loading Robust04. They easily scrape the relevant HTML content into text. For more details, see here - https://ir-datasets.com/trec-robust04.html#trec-robust04

Kind Regards,
Nandan

Thanks for the quick response! Just a follow-up question: which field of the topic should we use? For example:

> TrecQuery(query_id='301', title='International Organized Crime', description='Identify organizations that participate in international criminal\nactivity, the activity, and, if possible, collaborating organizations\nand the countries involved.', narrative='A relevant document must as a minimum identify the organization and the\ntype of illegal activity (e.g., Columbian cartel exporting cocaine).\nVague references to international drug trade without identification of\nthe organization(s) involved would not be relevant.')

Do we use title, description, or narrative?

 I used the description of the query for the results in the leaderboard.

Kind Regards,
Nandan

Thanks!
Repository: beir-cellar/beir
Issue: Can't explain low results for some models
Root cause: Validation Gap

Hey,

it is the second time I encounter low results for specific models. In short, I once trained `deepset/gbert-base` with `train_msmarco_v3_margin_MSE.py` and it worked like a charm. Then I tried the large version (`deepset/gbert-large`) and all results created with `evaluate_sbert.py `were almost zero (NDCG@1/5/10/100/1000 = 0.001...). Again, the base model created good results.

Now I did the same with `xlm-roberta-base` which again created good results. Using` microsoft/xlm-align` results in bad results again. What do I miss here? Are some models not technically feasible?

Are you suspecting a problem in training or evaluation?
Repository: beir-cellar/beir
Issue: Can't download data
Root cause: External Dependency Breakage

The website: https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/ is down. Any idea on when we might expect it to be online?


Hi @TahaBinhuraib, Seems to be online now. 

Sadly, I am not affiliated with UKP anymore. I'm unable to anticipate the downtimes of the dataset. I will try to replicate the datasets and host them on another server, in incase the UKP one is found to be down.

Regards,
Nandan Thakur

@thakur-nandan Thanks a lot!
Repository: beir-cellar/beir
Issue: evaluate_sbert_multi_gpu - metrics.compute() unable to read cache file
Root cause: Resource Mishandling

I'm trying to run beir/examples/retrieval/evaluation/dense/evaluate_sbert_multi_gpu.py. Doing do I end up with the below error.

Traceback (most recent call last):
  File "evaluate_sbert_multi_gpu.py", line 62, in <module>
    results = retriever.retrieve(corpus, queries)
  File "/data/user/beir/beir/retrieval/evaluation.py", line 23, in retrieve
    return self.retriever.search(corpus, queries, self.top_k, self.score_function, **kwargs)
  File "/data/user/beir/beir/retrieval/search/dense/exact_search_multi_gpu.py", line 150, in search
    cos_scores_top_k_values, cos_scores_top_k_idx, chunk_ids = metric.compute()
  File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/evaluate/module.py", line 433, in compute
    self._finalize()
  File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/evaluate/module.py", line 390, in _finalize
    self.data = Dataset(**reader.read_files([{"filename": f} for f in file_paths]))
  File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/datasets/arrow_reader.py", line 260, in read_files
    pa_table = self._read_files(files, in_memory=in_memory)
  File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/datasets/arrow_reader.py", line 195, in _read_files
    pa_table: Table = self._get_table_from_filename(f_dict, in_memory=in_memory)
  File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/datasets/arrow_reader.py", line 331, in _get_table_from_filename
    table = ArrowReader.read_table(filename, in_memory=in_memory)
  File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/datasets/arrow_reader.py", line 352, in read_table
    return table_cls.from_file(filename)
  File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/datasets/table.py", line 1065, in from_file
    table = _memory_mapped_arrow_table_from_file(filename)
  File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/datasets/table.py", line 52, in _memory_mapped_arrow_table_from_file
    pa_table = opened_stream.read_all()
  File "pyarrow/ipc.pxi", line 750, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: Expected to be able to read 80088040 bytes for message body, got 80088032

--
command used: CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python evaluate_sbert_multi_gpu.py

@thakur-nandan Any idea how to proceed?

The reason for this error is insufficient host memory (CPU ram). I would suggest evaluating on a larger GPU cluster or try reducing the batch size.
Repository: beir-cellar/beir
Issue: Allow option to disable progress bar when using `GenericDataLoader`
Root cause: Unimplemented Feature Gap

Right now, when using the data loader:
```python
corpus, queries, qrels = GenericDataLoader(data_dir).load(split=split)
```

Tqdm will always show up. there should be a way to disable it, e.g.:
```python
corpus, queries, qrels = GenericDataLoader(data_dir).load(split=split, show_progress_bar=False)
```

Hey xhluca,

I also was not fan of tqdm. It was really annoying sometimes.
When I do not need it, I disabled it by using the environment variable.
os.environ["TQDM_DISABLE"] = "1"

I had the same problem and would like to see this be optional, too. 
For anyone coming across this in the meantime, if you are using a Jupyter Notebook you can suppress the data loader's output with this without disabling tqdm:
```Python
from IPython.utils import io

with io.capture_output() as captured:
    corpus, queries, qrels = GenericDataLoader(data_dir).load(split=split)
```
Repository: beir-cellar/beir
Issue: Loosen Elasticsearch requirement
Root cause: Architectural Constraint

Hi!

I'm currently using beir via MTEB in another project, and noticed that beir was installing a very specific version of the elasticsearch package, overwriting my (looser) requirements. I was wondering if there was a reason this was pinned so specifically, and whether it could be loosened. I think `install_requires` should be as loosely pinned as possible, to reduce installation errors.

Thanks! Stéphan

Hi @stephantul, 

The reason for specifying this version was due to Elasticsearch updating its client mentioned in https://github.com/beir-cellar/beir/issues/58.
Repository: beir-cellar/beir
Issue: No module named 'tensorflow_text'
Root cause: Environment Incompatibility

Hi I am trying to run your Colab example and I get the following error:

No module named 'tensorflow_text' with the cell that starts training the model.

Could you please help solve the issue?

run `!pip install tensorflow_text` in a code cell

if you wish to run BEIR with `tensorflow_text`. 

run `pip install beir[tf]`, these will install additional tensorflow packages.

Thanks a lot for the response.

Do you have code in the repo for preparing the JSON files for the qrels and
the queries?
Thanks

On Tue, Jun 20, 2023 at 1:43 PM Nandan Thakur ***@***.***>
wrote:

> if you wish to run BEIR with tensorflow_text.
>
> run pip install beir[tf], these will install additional tensorflow
> packages.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/beir-cellar/beir/issues/138#issuecomment-1598700450>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ANOIMY2EVWIG4ZJU5NGLXV3XMGLHJANCNFSM6AAAAAAWRMKTH4>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


Looks interesting. Let me work on it!
Repository: beir-cellar/beir
Issue: [Question] Using T5 as a retriveal model in BeIR
Root cause: Unimplemented Feature Gap

Hi there,

Is there any way or approach to use the T5 model as a retrieval model in BeIR?

@rahmanidashti, 

Do you have any specific T5 retrieval model you wish to use? Are the model weights publicly available on Huggingface?

Thanks,
Nandan 

@thakur-nandan 

Thanks, Nandan! No, I do not have any specific model in mind, can we use the same T5 model that we use to generate the synthetic queries for the ranking? Can we use it to estimate the joint probability of passage and query?

Thanks,
Saeed
Repository: beir-cellar/beir
Issue: AssertionError: Elastic-Search Window too large, Max-Size = 10000
Root cause: Validation Gap

Using BM25, for sparse embedding in a pretty big datasets (eg. FiQA), I get the following assertion error:
```AssertionError: Elastic-Search Window too large, Max-Size = 10000```

The function that call BM25 is the next one:
```
def sparse_embeddings_bm25(dataset_name, corpus, queries, qrels, k_primes):
  '''
  PURPOSE: compute the sparse embedding using the BM25 implementation from beir and elastichsearch
  ARGUMENTS:
    - dataset_name: string describing the dataset name
    - corpus: sequence of documents 
    - queries: sequence of queries
    - qrels: ground truth of query document relevance
    - k_primes: list of number of top k prime documents to return
  RETURN: see embeddings return values
  '''
  hostname = 'localhost' 
  index_name = dataset_name
  initialize = True # Delete existing index with same name and reindex all documents

  print(f'{dataset_name} - BM25')
  model = BM25(index_name=index_name, hostname=hostname, initialize=initialize) # Defining the BM25
  return embeddings('Sparse', model, corpus, queries, qrels, k_primes)
```

I've already tryed to create the index before running BM25 and set initialize = False, but doing so I need somewhat to pass to the index the corpus and the queries.

Note that I'm running all the application in Google Colab Pro, I don't know if this is important or not.
Repository: beir-cellar/beir
Issue: Training script/configuration of T5 generator for GenQ
Root cause: Documentation Deficiency

Hello, thank you for sharing this repo and useful hugging face model cards!

I am interested in T5 generators for query generation, and trying to extend this to other datasets/tasks.
For doing so, I would like to reproduce T5 generators, specifically [BeIR/query-gen-msmarco-t5-large-v1](https://huggingface.co/BeIR/query-gen-msmarco-t5-large-v1).

I am wondering if the training script and training configurations for the generators can be shared,
including
* T5 initial checkpoint (Did you use [google/t5-v1_1-large](https://huggingface.co/google/t5-v1_1-large) or [google/flan-t5-large](https://huggingface.co/google/flan-t5-large)?),
* maximum source/target length,
* batch size,
* optimizer,
* learning rate,
* learning rate scheduling,
* warmup steps,
* and total training steps.

Best regards,
Jihyuk Kim

HI @jihyukkim-nlp, 

Sadly, I do not remember these exact training details for training the question generator models. @nreimers could you help me here?

Regarding the first two points:

- We use the T5-Large model as the initial checkpoint (Not the Flan T5 large model).
- I believe the maximum length for the target question is 64 tokens and the source is 350 tokens.

Hi @thakur-nandan, 

Thank you for the information!
I have tried a few different configurations, e.g., different learning rates (1e-5, 3e-5, 5e-5, 1e-4) either with or without warmup steps. But I have failed to reproduce. 

It worked relatively well for MS MARCO, but not for BEIR. 
At this point, I am also wondering if other datasets, such as NQ, were used for training the generator by any chance?

It will be really helpful if @nreimers can help.