zhiminy/EvalEng huggingface dataset:
Repository: beir-cellar/beir
Issue: Allow GPU parallelization to take advantage of multi-GPU instances
Root cause: Unimplemented Feature Gap

I'm running [evaluate_anserini_docT5query.py](https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/sparse/evaluate_anserini_docT5query.py) and it's currently only able to utilize a single GPU. I'm using GCP instances with 4x V100 GPUs and I'd like to finish more experiments in less time by using more GPUs. Is there a simple way to parallelize batches or configure using multiple GPUs that I'm not aware of?


Hi @joshdevins, 

Nice question! I, unfortunately, don't know how to implement this. I searched online but couldn't find a direct solution to generate using multi-GPUs. I will keep searching if I find something relevant will get back to you. 

If you have suggestions on implementation, I would happy to have a look at them!

Unfortunately for now I can only suggest hosting the individual question-generation models separately in the GPUs.

Kind Regards,
Nandan

Thanks @NThakur20. The obvious place for me would be to add data parallelization. So the corpus is chunked right now (I think into size 80 examples?), and we could easily parallelize that loop with a [multiprocessing](https://docs.python.org/3/library/multiprocessing.html) Pool of size `n` (one per GPU). When we call `generate` we can also pass a GPU/core number and CUDA can use that GPU as specified. I can take a stab at that one script for doc2query and any downstream changes, and maybe it's applicable to processing other datasets at scale as well.

So here we can pass a GPU number and make `n` models, one per GPU
https://github.com/UKPLab/beir/blob/8d8061934e2c33374c075412da2093a5f403521a/beir/generation/models/auto_model.py#L6

Last thought, I'm not familiar enough with Transformers+PyTorch innards but there should be a native way to do data parallel inference since we provide batches already. For example https://github.com/huggingface/transformers/issues/3936

Thanks for the pointers @joshdevins, I will have a look at them!

Kind Regards,
Nandan

Hi @joshdevins,

Finally, after having no success with DataPallalel, I'm happy to share that I have added support now using multiprocessing to generate multiple process pools which are able to take advantage of multiple GPUs. Hope it helps!

Here is a sample code on how to use it: https://github.com/UKPLab/beir/blob/development/examples/generation/query_gen_multi_gpu.py

I hope now you are able to utilize multiple GPUs now for a faster generation. The updates are yet in the development branch. I would suggest if it's urgent, you can clone the development branch and start working. I would soon push the changes to master and update the PyPI version.

Kind Regards,
Nandan
Repository: beir-cellar/beir
Issue: Processing CQADupstack does not work out-of-the-box
Root cause: Documentation Deficiency

I'm using [evaluate_anserini_docT5query.py](https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/sparse/evaluate_anserini_docT5query.py). In that script it's using `util.download_and_unzip` then `GenericDataLoader` and fails with the following message when using dataset `cqadupstack `:

```
ValueError: File /home/josh/source/beir/examples/retrieval/evaluation/sparse/datasets/cqadupstack/corpus.jsonl not present! Please provide accurate file.
```

The CQADupstack dataset is divided up into sub-categories which is causing the above error since it contains an extra sub-directory per-category.

```
 ls -las ~/source/beir/examples/retrieval/evaluation/sparse/datasets/cqadupstack/
total 56
4 drwxr-xr-x 14 josh josh 4096 Jul  1 08:38 .
4 drwxr-xr-x  3 josh josh 4096 Jul  1 08:35 ..
4 drwxr-xr-x  3 josh josh 4096 Jul  1 08:38 android
4 drwxr-xr-x  3 josh josh 4096 Jul  1 08:37 english
4 drwxr-xr-x  3 josh josh 4096 Jul  1 08:35 gaming
4 drwxr-xr-x  3 josh josh 4096 Jul  1 08:37 gis
4 drwxr-xr-x  3 josh josh 4096 Jul  1 08:37 mathematica
4 drwxr-xr-x  3 josh josh 4096 Jul  1 08:37 physics
4 drwxr-xr-x  3 josh josh 4096 Jul  1 08:37 programmers
4 drwxr-xr-x  3 josh josh 4096 Jul  1 08:37 stats
4 drwxr-xr-x  3 josh josh 4096 Jul  1 08:36 tex
4 drwxr-xr-x  3 josh josh 4096 Jul  1 08:37 unix
4 drwxr-xr-x  3 josh josh 4096 Jul  1 08:37 webmasters
4 drwxr-xr-x  3 josh josh 4096 Jul  1 08:37 wordpress
```

How was CQADupstack used in the benchmarking for the paper and leaderboard? Was each category processed separately or was everything somehow combined into a single evaluation?

Hi @joshdevins,

Yes, CQADupstack contains 12 StackExchange forums and you should process each category separately. 
The scores in the paper are the average of the individual forum scores. 
Also see: https://github.com/UKPLab/beir/issues/9

Kind Regards,
Nandan
Repository: beir-cellar/beir
Issue: Training ColBERT on a new test collection
Root cause: Unimplemented Feature Gap

Hi, 

Thanks for the wonderful package!

I want to train a ColBERT model on a new test collection which I have. I couldn't find any example related to it. 

Could you please point out how to train a ColBERT model with the package? 

Thanks

Hi @shashankg7,

Sadly there is no technique currently implemented in BEIR to train ColBERT on a new test collection.
I would suggest having a look at the original ColBERT repo: https://github.com/stanford-futuredata/ColBERT

Kind Regards,
Nandan Thakur
Repository: beir-cellar/beir
Issue: DPR on MSMARCO
Root cause: Unimplemented Feature Gap

Hi! Thanks for sharing BEIR - this is a wonderful project!

I wonder if you can include DPR model trained on MSMARCO on the [leaderboard](https://docs.google.com/spreadsheets/d/1L8aACyPaXrL8iEelJLGqlMqXKPX2oSP_R10pZoy77Ns/edit#gid=0)? The current DPR (Multi) and DPR (KILT) are not really comparable with other models which are trained on MARCO.

Thanks!

Hi @JetRunner,

Unfortunately, I do not have a DPR model trained on the MS MARCO dataset. The original DPR code requires 8 V100 GPUs for training and I personally have access to only a few GPUs at max in my lab. 

What is really interesting is the Gradient Caching Technique (https://arxiv.org/abs/2101.06983) introduced by @luyug which I plan to hopefully integrate soon within BEIR. This will allow you to train a DPR system using a single V100 GPU within 11GB.

Once we have GradCache (https://github.com/luyug/GradCache) included within the repository, I could share a trained DPR model on the MSMARCO dataset.

 Kind Regards,
Nandan

Thanks, @NThakur20!
Repository: beir-cellar/beir
Issue: Adding TREC ad-hoc collections
Root cause: Unimplemented Feature Gap

Hi,
Very useful work, thanks ! 
I was wondering why you did not include standard ad-hoc retrieval collections in the benchmark (like Robust04) ? Is it intended ? 
For people working on neural IR, it would be interesting to see how models trained on MS MARCO systematically generalize to these collections too 

Hi @thibault-formal, It definitely wasn't intended. Unfortunately, there are a high number of interesting datasets out there and we went ahead and chose a few of them. I also find standard ad-hoc collections interesting for analysis, I am currently in process of collecting the Robust04 dataset (sadly, requires a week to get the collections) and hopefully would have the analysis soon for the dataset. 

Kind Regards,
Nandan

Hi @NThakur20 
great, I think a lot of people will be interested in this -- including myself. :) 
cheers, 
Thibault 

Hi @thibault-formal, apologies for the delay. But the NIST website was down and finally got hold of the Robust04 dataset. Here below I mention the **nDCG@10** scores over the dataset - 

TREC-Robust04
-----------------------
Test Queries: 249
Number of Documents: 528155

|  BM25  |  DPR (Multi)  | SBERT (MiniLM-L6) | SBERT (DistilBERT-v3) | ANCE | ColBERT (100) | BM25 (100) + ELECTRA |  BM25 (100) + MiniLM-L6 | 
| :-------: |   :---------------:  | :--------------------------: | :-------: | :------:   | :------: | :-------:  | :--------: |
|   0.387 |        0.252        |                       0.293   |  0.318 |  0.392 |  0.391  | 0.438    |  0.467 | 

hi @NThakur20, great thank you !
Repository: beir-cellar/beir
Issue: The QRel file rank for BioASQ
Root cause: Documentation Deficiency

How would you assign numbers in the QRel file for BioASQ?
Could you please provide an example. Let's say for a question Q with id ID the ideal documents are [D1,D2,D3]. How would the QRel entries look like?

Thank you

Hi @ari9dam,

For BioASQ, we have relevant documents which are binary i.e. relevant or not relevant. So for your example your qrels would look something like below for ``(qrels/test.tsv)``

```
qid   cid     score
ID    D1      1
ID    D2      1
ID    D3      1
```

Hope it helps!
Repository: beir-cellar/beir
Issue: Unable to create Index in Elastic Search
Root cause: External Dependency Breakage

I am getting an error when running the bm25 evaluation file. 

`Unable to create Index in Elastic Search. Reason: The client noticed that the server is not a supported distribution of Elasticsearch
`
The error was not there last week when I ran the file. I am using Colab to run the file. Any idea what maybe the issue here?

Found the reason for the error. The latest versions of elasticsearch does not work with the 7.9.2 version. The version of elasticsearch should be downgraded to 7.9.1 which is compatible with version 7.9.2.
In other words just use this code

`!pip install elasticsearch==7.9.1`
Repository: beir-cellar/beir
Issue: Training data for NQ?
Root cause: Documentation Deficiency

Thanks for the great contribution!

I found that the downloaded data of NQ only contains test files and corpus, where can I get the training files?

Thank you!

Hi @ReyonRen,

You can use this dataset for NQ training: https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/datasets/paraphrases/NQ-train_pairs.jsonl.gz

Kind Regards,
Nandan

Thank you very much. Is the query of this training set a subset of the passage in your open-source NQ corpus?

Hi @ReyonRen, 

Actually, the evaluation corpus is a subset of the training set. Because in NQ (original dataset) often you can have duplicated pages present, i.e. identical Wikipedia pages from let's say 2014, 2015, etc. 

While creating the BEIR NQ evaluation corpus, we only evaluate a single question for a Wikipedia passage, because if we add other passages with the same title but from a different year let's say 2014 or 2015, you introduce duplicates within your dataset. 

However, during training, you do not care about duplicates and train with all passage and question combinations!

Kind Regards,
Nandan Thakur

Thank you for the kind reply!

Hi @NThakur20, is it possible to make the preprocessing code from jsonl to TSV available for the NQ dataset? Or if the `train.tsv` for NQ is available for download, that'd be helpful too. 

The tsv format is here: https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/nq-train.zip

Discussed here: https://github.com/beir-cellar/beir/issues/108
Repository: beir-cellar/beir
Issue: Can't understand a certain line

https://github.com/UKPLab/beir/blob/933b349bf300718cd6a2d285c51fe78f48fdec85/beir/retrieval/search/dense/exact_search.py#L77

I couldn't understand what this line is trying to do... `corpus_id` and `query_id` are from completely different groups and it's fine that they are the same, right? Removing this `if` statement has a huge impact on ndcg score (tested with ANCE@arguana).

Hi @ji-xin, 

In Few corpora such as Quora as Arguana, both query and document are similar (both passage or sentence), for eg. in Quora both query and document are a single sentence question and in Arguana both are passages. Now, when you retrieve a question in quora (for eg. q1: how many states are present in India?) it retrieves the identical input question itself as the most similar document (i.e q1 is returned as top-result) as the input query is present within the corpus! 

To avoid such a situation, we check whether the returned result (``corpus_id``) has the same id as the (``query_id``), if yes, we explicitly remove the document from the returned results. This doesn't have an effect on other datasets as query is a sentence, while documents are passages.

Kind Regards,
Nandan Thakur

Thanks for the response! Sounds like a reasonable move.
Repository: beir-cellar/beir
Issue: Additional annotations for Trec-covid
Root cause: Documentation Deficiency

Just saw the submission to Neurips (https://openreview.net/forum?id=wCu6T5xFjeJ), great work!
I was just wondering whether you plan to release the additional annotations for Trec-covid’s hole filling? Thanks!

Hi @ji-xin 
That was quick to notice.

In the appendix (sadly a separate file for the NeuRIPS submission) the link is provided:
https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/trec-covid-beir.zip

Thanks!
Repository: beir-cellar/beir
Issue: About webis-touche2020 qrels
Root cause: Documentation Deficiency

Thanks for your great works. I notice that average number of docs per query changes from 49.9 to 19.0 in your latest version of the paper. To keep the same setting as your, what should I need to do?

Thanks.

Hi @guoday,

Nice observation! I was wondering when I will get this issue. I used the latest version of webis-touche2020 qrels which the dataset authors recently provided for the paper. I was waiting for the numbers on the leaderboard (exps which I need to rerun), and then upload the latest dataset.

Anyways, I will upload the latest dataset soon and update you back here once its done!

Kind Regards,
Nandan Thakur

Thanks a lot for your reply. 😊 

All done, you can find the latest dataset here: https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/webis-touche2020.zip

Kind Regards,
Nandan

Thanks for your quick action!!!

Best,
Daya
Repository: beir-cellar/beir
Issue: How to evaluate on Trec-Covid
Root cause: Documentation Deficiency

Hi, I have questions regarding the Trec-Covid datasets. 
(1) I should retrieve the document in the entire corpus (171K) right? 
(2) The qrel test.tsv has three labels 0,1,2, when I get the prediction from the BM25 baseline, how should I assign the values to them? all 1 ? 
Thank you

Hi @luomancs,
(1) Yes, you should retrieve the document from the entire corpus.
(2) When you get predictions from the model, a higher score denotes a higher rank. So, my advice would be to use the BM25 scores as it is. 

Hope it helps!

Kind Regards,
Nandan Thakur

Hi Nandan,

Thanks for the response, as you suggested, I retrieve relevant document from the entire corpus and assign the bm25 score to each retrieved document. And I used the pytrec_eval to evaluate bm25 results, however, I got much lower ndcg@10 score as you had in the paper. Specifically, my script looks like follows, I wonder should I do any processing of the "qrel" which is the test.tsv that is given.

import pytrec_eval
import json

% from qrels/test.tsv, where each q_i has more than 10 documents from test.tsv
qrel = {
    'q1': {
        'd1': 0,
        'd2': 1,
        'd3': 2,
        'd4':0
    },
    'q2': {
        'd2': 1,
        'd3': 1,
    },
}

% prediction from bm25, where each q_i has 10 relevant documents and the score is assigned by bm25. 

run = {
    'q1': {
        'd4': 1.0,
        'd10': 0.0,
        'd9': 1.5,
    },
    'q2': {
        'd1': 1.5,
        'd6': 0.2,
    }
}

evaluator = pytrec_eval.RelevanceEvaluator(
    qrel, {'ndcg'})

ndcg = 0
for i, item in evaluator.evaluate(run).items():
    ndcg += item['ndcg']
print(round(ndcg/len(qrel),3))

Hi @luomancs,

Use this easy example for BM25 scores, it should work automatically for you!
https://github.com/UKPLab/beir/blob/main/examples/retrieval/evaluation/lexical/evaluate_bm25.py

Only replace L33 in the example as ``dataset = 'trec-covid'``

Hi Nandan, got it! Thank you !
Repository: beir-cellar/beir
Issue: Error in accuracy function
Root cause: Algorithmic Error

In the custom_metrics.py file, top_accuracy function:
you write:
top_hits[query_id] = sorted(doc_scores.keys(), key=lambda item: item[1], reverse=True)[0:k_max]
but I think that you should instead wrote [elem[0] for elem in
                              sorted(doc_scores.items(), key=lambda item: item[1], reverse=True)[0:k_max]]
indeed, in your code, you are sorting by the key and not by the score

Hi @wolhandlerdeb,

Thank you very much for pointing out this mistake! 
Wow, I did a mistake there and I am soon going to push a new pypi version with the updated changes.
Repository: beir-cellar/beir
Issue: create qrel.tsv file

Hi, Nandan,

When we create the qrel.tsv file for custom preprocessed dataset, can it just innclude relevant documents, i.e., score=1 or non-relevant documents should also be included?

Cheers
Xiang

Hi @xiangpku,

Only the relevant ones are necessary, i.e. with a score > 0. Everything else is automatically assumed with a score of zero if not present within the ``qrels.tsv`` file.

Hope it helps!

Kind Regards,
Nandan Thakur 

Great! 
Thanks again for your awesome work!

Xiang
Repository: beir-cellar/beir
Issue: Leaderboard results of USE
Root cause: Documentation Deficiency

Hi,
Thank you for publicizing this great benchmark!
I am interested in the retrieval performance of models that are not trained on retrieval datasets (e.g., MS MARCO). I think this benchmark already supports USE, but it is not listed on the leaderboard. Did you already test the performance of the model on this benchmark? I think it would be very helpful if the leaderboard also provides results of general sentence embedding models (e.g., USE, SimCSE).

Hi @ikuyamada,

Yes, in the previous version of the BEIR benchmark we included the USE model. I do have the numbers available and will add them to the leaderboard soon.

Do you have a list of specific models you would like to see the results of? I can evaluate these models and add their scores on the leaderboard.

Kind Regards,
Nandan

Hi @NThakur20,

Thanks for your prompt reply! I look forward to see the results of USE.
I think people want to see the results of recent unsupervised models such as SimCSE and CT.
Also, I personally see the results of BPR trained with MS MARCO :)

Best regards,
Ikuya

We will report results for them in an upcoming paper. All unsupervised methods don't perform at all on retrieval tasks, even when trained on in-domain data. 

@nreimers Thanks for your answer! The paper sounds really interesting. I close this issue for now.
Repository: beir-cellar/beir
Issue: Training a T5 model
Root cause: Interface Contract Mismatch

Hi @NThakur20  I was wondering if we can train a T5 model as when I was loading a T5 model from HF there seems to be an error. 

Hi @pritamdeka, 

Could you give a bit more context mentioned below:
1. Which model exactly did you use? LInk to the model, if available?
2. Which code did you use to train the model?
3. What error did you encounter during training?
4. If possible, a sample code snippet to reproduce your error.

Thank you!

Kind Regards,
Nandan Thakur

Hi @NThakur20  Thanks for the reply.

I used the **train_sbert_BM25_hardnegs.py** file for training using the SCIFACT dataset. Made a few changes for the T5 model such as changed the following line 

```word_embedding_model = models.Transformer(model_name, max_seq_length=300)``` to 
```word_embedding_model = models.T5.T5(model_name, max_seq_length=300)```

Also the model I used is **castorini/monot5-base-msmarco**

The error is something like this:

```
Epoch:   0% 0/1 [00:00<?, ?it/s]
Iteration:   0% 0/1121 [00:00<?, ?it/s]
Epoch:   0% 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/content/beir/examples/retrieval/training/train_sbert_BM25_hardnegs.py", line 132, in <module>
    use_amp=True)
  File "/usr/local/lib/python3.7/dist-packages/beir/retrieval/train.py", line 148, in fit
    callback=callback, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/sentence_transformers/SentenceTransformer.py", line 682, in fit
    data = next(data_iterator)
  File "/usr/local/lib/python3.7/dist-packages/sentence_transformers/datasets/NoDuplicatesDataLoader.py", line 41, in __iter__
    yield self.collate_fn(batch) if self.collate_fn is not None else batch
  File "/usr/local/lib/python3.7/dist-packages/sentence_transformers/SentenceTransformer.py", line 534, in smart_batching_collate
    tokenized = self.tokenize(texts[idx])
  File "/usr/local/lib/python3.7/dist-packages/sentence_transformers/SentenceTransformer.py", line 311, in tokenize
    return self._first_module().tokenize(texts)
  File "/usr/local/lib/python3.7/dist-packages/sentence_transformers/models/T5.py", line 55, in tokenize
    return self.tokenizer.encode(self.task_identifier+text)
TypeError: can only concatenate str (not "list") to str

```

Have you found a solution for this? Is there any T5-based models currently implemented in BeIR?
Repository: beir-cellar/beir
Issue: BaseException with rerank - 'dict' object has no attribute 'strip'
Root cause: Interface Contract Mismatch

Hi there,

I've been working on a dense IR pipeline with BEIR including a custom dataloader, which works fine for dense IR runs but throws an exception whenever I add a cross encoder for reranking.

Rerank:

```
cross_encoder_model = CrossEncoder('cross-encoder/ms-marco-electra-base')
reranker = Rerank(cross_encoder_model, batch_size=128)
```

Dataloader:

```
corpus = {}

for index, item in corpusdf.iteritems():
    corpus.update({
        "doc"+(str(index)): {
            "title": "",
            "text": item,
            },
    })

queries = {}

for index, row in queriesdf.iterrows():
    queries.update({
        "q"+str(index): {
            "doc"+(str(index)): row[0],
            },
    })

qrels = {}

for i in range(len(df)):
    qrels.update({
        "q"+str(i): {
            "doc"+(str(i)): 1,
        },
    })
```

Exception:
```
Traceback (most recent call last):
  File "C:\Users\costco\venv\lib\site-packages\sentence_transformers\cross_encoder\CrossEncoder.py", line 273, in predict
    for features in iterator:
  File "C:\Users\costco\venv\lib\site-packages\tqdm\std.py", line 1180, in __iter__
    for obj in iterable:
  File "C:\Users\costco\venv\lib\site-packages\torch\utils\data\dataloader.py", line 521, in __next__
    data = self._next_data()
  File "C:\Users\costco\venv\lib\site-packages\torch\utils\data\dataloader.py", line 561, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\costco\venv\lib\site-packages\torch\utils\data\_utils\fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "C:\Users\costco\venv\lib\site-packages\sentence_transformers\cross_encoder\CrossEncoder.py", line 93, in smart_batching_collate_text_only
    texts[idx].append(text.strip())
AttributeError: 'dict' object has no attribute 'strip'
```
Seems like a simple fix but I am trying to avoid modifying BEIR sources, any ideas would be greatly appreciated!
Repository: beir-cellar/beir
Issue: Providing BM25 top-100 for all datasets
Root cause: Unimplemented Feature Gap

First of all, thanks for this amazing benchmark. I'd like to evaluate a re-ranking model on several datasets. If I got I correctly, I will have to download and index all datasets independently to get the top-100 BM25 rank lists. Could you please provide those for each dataset for easier evaluation?

Thanks a lot!

I would echo this request since I cannot reproduce the BM25 (Elasticsearch, default setting) results on certain datasets, particularly the ones with fewer docs like NFCorpus and SciFact. 

Besides, I found that the scores may change a bit in different runs. Is there any randomness inside Elasticsearch? Maybe @thakur-nandan can provide some insights? Thanks!