zhiminy/EvalEng huggingface dataset:
Repository: beir-cellar/beir
Issue: Title needed when using retriever.retrieve after using GenericDataLoader(...)load_custom(...) 
Root cause: Validation Gap

I have noticed an error when using EvaluateRetrieval.retrieve(model) when using the exact_search combined load_custom() method from the GenericDataLoader. 

https://github.com/UKPLab/beir/blob/main/beir/datasets/data_loader.py -> GenericDataLoader:
It says in the document that the title is optional, but when feeding a custom corpus without title, it goes through the _load_corpus() method where the following code is executed:

                self.corpus[line.get("_id")] = {
                    "text": line.get("text"),
                    "title": line.get("title"),
                }

more precisely: 
                    "title": line.get("title"),

By default, if the title is not present, it returns a None.

Then in the following file:
https://github.com/UKPLab/beir/blob/main/beir/retrieval/evaluation.py

It uses the method retrieve, which uses the search method:

return self.retriever.search(corpus, queries, self.top_k, self.score_function, **kwargs)


When using [search.dense.exact_search.py](https://github.com/UKPLab/beir/blob/main/beir/retrieval/search/dense/exact_search.py) as the retriever, the following code is executed:

corpus_ids = sorted(corpus, key=lambda k: len(corpus[k].get("title", "") + corpus[k].get("text", "")), reverse=True)

The problem is the following: When using GenericDataLoader(...)load_custom(...), if there is no title field, It creates a None type as title, then when trying to run this code : 

corpus[k].get("title", "") + corpus[k].get("text", "")

It returns a TypeError: unsupported operand type(s) for +: 'NoneType' and 'str'

What I suggest to resolve this problem (It worked in my case) is replacing the code like the following, for the  _load_corpus() method:

                self.corpus[line.get("_id")] = {
                    "text": line.get("text"),
                    "title": line.get("title", ""),
                }
It creates a string, then it is possible to run the following code:

corpus[k].get("title", "") + corpus[k].get("text", "")

Olivier 





I stumbled upon the same problem and totally support this. To make the code work I modified my dataset to have "title": "" for each corpus entry. 

However, I find it ambiguous that "title" is used by default in evaluation if present while default behavior with no title will be to use the "text" only (after the proposed fix).
I would also add a parameter to enable or disable title by default. It should probably go in the SentenceBERT.encode_corpus(use_title_if_present)
Repository: beir-cellar/beir
Issue: Elasticsearch connection error for BM25
Root cause: Documentation Deficiency

For all `BM25Search` instatiations that use Elastic search, I'm getting the following error on both Ubuntu 18.04 and 20.04:
```Traceback (most recent call last):
  File "/home/jaxball/anaconda3/envs/pytorch/lib/python3.7/site-packages/elastic_transport/_transport.py", line 334, in perform_request
    request_timeout=request_timeout,
  File "/home/jaxball/anaconda3/envs/pytorch/lib/python3.7/site-packages/elastic_transport/_node/_http_urllib3.py", line 199, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fb774790ed0>: Failed to establish a new connection: [Errno 111] Connection refused)
```
I simply replaced the `hostname` with `http://localhost:9200` and index name with `scifact`. This fails for multiple files i.e. `evaluate_bm25.py` or `train_bm25_hardnegs.py` so I suspect my ES is not setup correctly. Any ideas how to fix this? 

Looks like this is a duplicate of #4. For those with the same problem, check out the example of starting elasticsearch in `BEIR.ipynb`. Closing issue.
Repository: beir-cellar/beir
Issue: Low BM25 baselines?
Root cause: Validation Gap

Hi there, thanks for providing this nice resource!

Looking at your paper, I think your BM25 baselines are a bit low? You report 0.218 nDCG@10 on MS MARCO, if I'm not mistaken - from Table 2.

With Pyserini https://github.com/castorini/pyserini/blob/master/docs/experiments-msmarco-passage.md - we can get, and this has been widely reproduced:

```
$ tools/eval/trec_eval.9.0.4/trec_eval -c -mrecall.1000 -mmap -m ndcg_cut.10   collections/msmarco-passage/qrels.dev.small.trec runs/run.msmarco-passage.bm25tuned.trec
map                   	all	0.1957
recall_1000           	all	0.8573
ndcg_cut_10           	all	0.2340
```

So, 1.6 points higher?

I suspect all the BM25 results should be a bit higher, based on our experience: https://arxiv.org/abs/2104.05740

For many of the other datasets with dense labels, a competitive baseline - and widely acknowledged in the IR community - would be something like BM25+RM3.

We would be happy to work with you on building out Pyserini as the competitive baseline for this task... Please reach out!
Repository: beir-cellar/beir
Issue: Question about MS MARCO dataset in BEIR
Root cause: Documentation Deficiency

Hello, 

I see that you processed the raw MS MARCO dataset to include it in your unified BEIR-style dataset. After downloading and checking it, I have several questions.
1. Where does the data in `test.tsv` come from? Given that the test set of MS MARCO passage retrieval task is not publicly available.
2. What does the scores in `test.tsv` stand for? I see there is only score 1 in `train.tsv` and `dev.tsv`, while there are 0, 1, 2, 3 in `test.tsv`.

Hi @jordane95, thank you for your questions!

1. The data in the ``test.tsv`` comes from the passage ranking task in TREC-DL 2019 (https://microsoft.github.io/msmarco/TREC-Deep-Learning-2019.html#passage-ranking-dataset).
2. Yes, the scores in ``train.tsv`` only will be binary and 1 as we only care about relevant passages (i.e. with score = 1) while training. However in evaluation for a few datasets, we may have multiple levels of relevancy, i.e. a score of 3 denotes a higher similarity between the query and passage in contrast to a score of 1.

Hope it helps!

Kind Regards,
Nandan Thakur


Thank you for your answer! That helps a lot!
Repository: beir-cellar/beir
Issue: Marco triplet in train_msmarco_v2.py
Root cause: Documentation Deficiency

Thanks for sharing your work! I fould the triplets_url [here](https://github.com/beir-cellar/beir/blob/a55552db70f37102352fd5c21b4e811516659a55/examples/retrieval/training/train_msmarco_v2.py#L53) in train_msmarco_v2 is stale. Could you please share this ms marco triplets file? Thanks!

@WenzhengZhang You can find an updated version of the file here:
https://huggingface.co/datasets/sentence-transformers/embedding-training-data/resolve/main/msmarco-triplets.jsonl.gz

Not that his file is in jsonl format, while the linked script expects tsv. Hence, you need to update you script how the triplets are loaded

Better (updated) training scripts can be found here:
https://github.com/UKPLab/sentence-transformers/tree/master/examples/training/ms_marco

Thanks for your help!
Repository: beir-cellar/beir
Issue: Support for custom retrievers: common base type for `EvaluateRetrieval`'s `retriever` param
Root cause: Architectural Constraint

The current design of `EvaluateRetrieval` is already supporting multiple different retrievers. See https://github.com/beir-cellar/beir/blob/bb8a0582d8a82d6f4fddee8b668b1ebaa42985ef/beir/retrieval/evaluation.py#L14 
However according to typings the supplied retrievers are limited to those implemented in BEIR. If we want to integrate BEIR in other tools (like e.g. [haystack](https://github.com/deepset-ai/haystack)) a custom retriever is basically easy to implement (see https://github.com/deepset-ai/haystack/pull/2333). This is due to the fact that all retrievers share a common interface of course. However the typings on `EvaluateRetrieval` do not account for this explicitly. A common abstract base class instead of a Tuple of all implemented retrievers would make this extensibility more comprehensive.



Hi @tstadel, 

Thank you for bringing up this issue. I agree with your point. Could you provide an implementation that you have in mind for a common abstract base class and provide a pull request for that? I can check and merge it in the future.

Also, happy to see BEIR integrated within [haystack](https://github.com/deepset-ai/haystack). 

Kind Regards,
Nandan Thakur

Hey @NThakur20,

sure. I'll submit a PR in the next days.

Shouldn't the line:

```
Union[Type[DRES], Type[DRFS], Type[BM25], Type[SS]]
```

Be:

```
Union[DRES, DRFS, BM25, SS]
```

In the examples you use this code:

```
model = DRES(models.SentenceBERT("msmarco-roberta-base-ance-fristp"))
retriever = EvaluateRetrieval(model, score_function="dot")
```

In this case, `model` is not of type `Type[DRES]`, it is of type `DRES`
Repository: beir-cellar/beir
Issue: Pre-processing code for missing datasets
Root cause: Documentation Deficiency

Hi there,

I wonder if the code for preprocessing unprovided datasets could be released? For example, I downloaded bioasq and signal1m following the instruction. But it's not clear to me how to convert the raw dataset to corpus.jsonl, queries.jsonl, qrels/{train/dev/test}.tsv the same way as you did. I think it's critical for reproducing your results and fair benchmarking.

Thank you,
Rui

Hi @memray,

You can send out an email to nandant@gmail.com. I can send you the datasets privately. 
Please ensure you are responsible for accepting the licenses for all private datasets.

Kind Regards,
Nandan Thakur

Thanks @NThakur20 ! I'll reach out to you via email later.

Hi @NThakur20,
I sent an email as well.
It would be great if you could share the details for the html2text initialization for TREC News, and which Anserini tweet indexing options were used for Signal AI.
Thanks!

Hi @thakur-nandan,

I also sent an email to kindly request access to the TREC-News dataset TGZ files. I assure you that the dataset will be used solely for academic purposes. I greatly appreciate your assistance and support!

Thanks!
im.jzfeng@gmail.com

Hi @gsgoncalves and @Cyril-JZ, you can find the private BEIR datasets here (all preprocessed): https://drive.google.com/drive/folders/1CgDO-KmQQMpGEGeD3R20ZgTTM008xix9?usp=sharing.

Hope it helps!

Thanks!

Thanks for your prompt reply! It helps me a lot!
Repository: beir-cellar/beir
Issue: url to msmarco triples do not exists anymore
Root cause: External Dependency Breakage

Hi,

I'm trying to run a bi-encoder on msmarco dataset. I used your `train_msmarco_v2.py` script but it appears that the [url](https://github.com/beir-cellar/beir/blob/main/examples/retrieval/training/train_msmarco_v2.py#L53) to download msmarco triples do not exists anymore...



You can find the working scripts here:
https://github.com/UKPLab/sentence-transformers/tree/master/examples/training/ms_marco
Repository: beir-cellar/beir
Issue: Difference between SBERT and DPR?

Hello, 

I recently read your BEIR paper and your SBERT lib doc, as well as the DPR paper. I find that DPR and SBERT share nearly the same architecture, except that SBERT tie the two encoders while DPR not. They also have the same training objective when performing dense retrieval task. 

So is there any other difference between the two?

They are quite similar:
- SBERT uses in most cases tied encoders, as this gives a better performance. DPR uses two independent encoders.
- SBERT mainly uses mean pooling, as it is better than the used CLS pooling in DPR
- DPR is fixed to bert-base, sentence-transformers can work with all encoder models.
- DPR used dot-product. In SBERT, there are models with cosine similarity as well as with dot product. Both measures have their pros and cons, so no way to decide which is better.
- The sentence-transformers framework comes with a lot more flexibility how to design your models and how to train them. So you can create a DPR like model also with SBERT.

Thank you for your help !
Repository: beir-cellar/beir
Issue: CQAdupstack evaluation

First I would like to thank you for this incredible framework, it has been of incredible help to me. 

I have a question concerning the cqadupstack results. As there are 12 different corpus for that dataset, which mean did you used to compute your results? The mean of means (i.e. each dataset has equal weight) or the mean over all the queries (i.e. each query has equal weight)?

Thanks in advance.

Hi @cadurosar, Thank you for your kind words!

We considered the mean of means for cqadupstack with each domain/corpus having equal weight. 
In short, the final ``NDCG@10`` score becomes``= sum(NDCG@10 for each domain) / 12`` in cqadupstack.

Kind Regards,
Nandan

Great, thanks for your quick answer! :)

Can/should we add a link to this ticket from the README somewhere? Seems people will run into this again.
Repository: beir-cellar/beir
Issue: Support multi-gpu training?

Hello, 

I'm trying to use your [code](https://github.com/beir-cellar/beir/blob/main/examples/retrieval/training/train_msmarco_v3.py) to train a bi-encoder on msmarco, but it seems that using a comparable batch size exceeds my GPU memory. Does your code support multi-gpu training?

@jordane95 how are you moving the data to GPU. 
  Did you move data to GPU?
 train_dataset.to(device) 


I don't see torch in that  [code](https://github.com/beir-cellar/beir/blob/main/examples/retrieval/training/train_msmarco_v3.py) , and that is the only way i know to use GPU

My bad, cuda was not enabled and its seems BeIR uses it automatically if available 

[2022-06-06 10:29:03] INFO [beir.generation.models.auto_model.__init__:16] Use pytorch device: cuda

_Now back to your multi-Gpu question :)_

Hi @jordane95 and @ahadda5, sadly the code present in the repository uses ``Sentence-Transformers`` as a framework where multi-GPU training is yet not available. 

You can have a look at [Tevatron](https://github.com/texttron/tevatron) in case you are interested in multi-GPU training. 

Kind Regards,
Nandan Thakur

Thanks! I will check it out.

> jordane95

Hi @jordane95, could you solve this issue? I am also running dense models on MSMARCO but it exceeds memory during encoding corpus.

> > jordane95
> 
> Hi @jordane95, could you solve this issue? I am also running dense models on MSMARCO but it exceeds memory during encoding corpus.

Hi @rahmanidashti , may be you could try the [tevatron](https://github.com/texttron/tevatron) toolkit which supports chunked corpus encoding
Repository: beir-cellar/beir
Issue: Thoughts on reorganizing BEIR main landing page

tl;dr - IMO the current README landing page tries to do too much... I think we should trim down.

For example, why do we have these?
- https://github.com/beir-cellar/beir#beers-course-material-on-ir
- https://github.com/beir-cellar/beir#beers-multilingual-datasets 

People come to the page for the dataset itself? So these should come first:
- https://github.com/beir-cellar/beir#beers-quick-example
- https://github.com/beir-cellar/beir#beers-available-datasets

All the examples and baselines should come later?

Thoughts @NThakur20 ?

Hi @lintool, 

Sounds good to me. I think we can reposition the quick example and available datasets on top and push course material and multilingual datasets further down on the landing page. 

I kept the course material on the IR page, in case someone is not familiar with retrieval or dense retrieval, they can refer to these course materials available online. Similarly, BEIR can also support the evaluation of multilingual datasets, that's why we have a multilingual datasets section.

What do you think @nreimers?

Kind Regards,
Nandan Thakur

The README has been organized accordingly in the latest BEIR version v1.0.1. 
I am closing the issue.

Kind Regards,
Nandan Thakur
Repository: beir-cellar/beir
Issue: the number of queries in MSMARCO
Root cause: Documentation Deficiency

Hi,

I have a question regarding the number of queries in MSMARCO.
According to the paper and the readme, the number of test queries in MSMARCO is 6,980.

However, when I ran the following codes, I was only able to get 43 queries.
```
>> corpus, queries, qrels = GenericDataLoader(data_folder='msmarco').load(split="test")
>> print(len(queries))
43
```

Instead, I got 6,980 queries from the dev set.
Should I use the dev queries when evaluating MSMARCO instead of the test queries?

Thanks!

Hi,

The split you are looking for is the "dev" split (so split="dev"). BEIR considers MSMARCO test to be one of the TREC-DL competitions.

@cadurosar 

Thanks! It's clear now.

Thanks for the clarification. However, this is very misleading, because for other BEIR datasets, we should use the "test" split. And suprisingly for MsMarco, we should use "dev". I used to use "test" split also for MsMarco and produced unexpected high accuracy. It took me a while to figure this out. So, it is ideal to mention this particularity in the homepage so as to save other researcher's time. @thakur-nandan
Repository: beir-cellar/beir
Issue: Re-using same GenericDataLoader for different splits raises KeyError
Root cause: Validation Gap

Here is the script:
```python
gdl = GenericDataLoader(data_folder='.local/datasets/scifact')

corpus, queries, qrels = gdl.load(split="test")
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5183/5183 [00:00<00:00, 195184.62it/s]

corpus, queries, qrels = gdl.load(split="train")
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Input In [24], in <cell line: 1>()
----> 1 corpus, queries, qrels = gdl.load(split="train")

File ~/.cache/pypoetry/virtualenvs/sparse-attention-05J_nGF--py3.9/lib/python3.9/site-packages/beir/datasets/data_loader.py:78, in GenericDataLoader.load(self, split)
     76 if os.path.exists(self.qrels_file):
     77     self._load_qrels()
---> 78     self.queries = {qid: self.queries[qid] for qid in self.qrels}
     79     logger.info("Loaded %d %s Queries.", len(self.queries), split.upper())
     80     logger.info("Query Example: %s", list(self.queries.values())[0])

File ~/.cache/pypoetry/virtualenvs/sparse-attention-05J_nGF--py3.9/lib/python3.9/site-packages/beir/datasets/data_loader.py:78, in <dictcomp>(.0)
     76 if os.path.exists(self.qrels_file):
     77     self._load_qrels()
---> 78     self.queries = {qid: self.queries[qid] for qid in self.qrels}
     79     logger.info("Loaded %d %s Queries.", len(self.queries), split.upper())
     80     logger.info("Query Example: %s", list(self.queries.values())[0])

KeyError: '0'
```

The workaround at the moment is:
```python
gdl.queries = {}
```
Obviously the code does not deal with reloading.
Since the corpus is using so much memory, it would be a pity to create a new DataLoader for each split.

I would be happy to make a PR, but there will be a delay.
I'll check again soon.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Multilingual evaluation benchmarks
Root cause: Unimplemented Feature Gap


Added:
* Code generation (few-shot generation with BLEU evaluation): Concode (Java ), Spider (SQL), CoNaLa(Python)
* Code summarization (few-shot generation with BLEU evaluation): code-to-text benchmark from CodeXGLUE (Java, JavaScript, PHP,Python, Go and Ruby)
* Classification tasks (fine-tuning): complexity prediction (Java), clone detection (Java), defect detection(C)
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Library seems unnecessarily hardcoded
Root cause: Architectural Constraint

This library seems overly hard-coded, to the point where building off of it would be difficult. Is there a particular reason you decided to use this architecture for your code? Or why a fork of https://github.com/EleutherAI/lm-evaluation-harness or https://github.com/BigScience-Workshop/lm-evaluation-harness wouldn't suit your needs?

This repository started out to just combine 2-3 evaluation benchmarks and has since grown a bit. It is just a few weeks old and we might adapt and refactor it a bit when it makes sense as we are continue to develop it.

That makes sense! I've been involved with the development of the EleutherAI harness and lead the BigScience eval workshop so if you want to chat about stuff :)

Hi @StellaAthena, I made this [PR](https://github.com/bigcode-project/bigcode-evaluation-harness/pull/19) to refactor the codebase to build tasks in separate files like you do in lm-evaluation-harness, it should make it easier to build off of. 
I also updated the acknowledgment section to EleutherAI, and now also mention it in the top of the readme, as I used a similar architecture. Please let me know if you have any comments.

@loubnabnl Apologies for not replying more promptly, but this revamp looks great! I very much look forward to using the library and potentially helping out with the BigCode project in general.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: What does 'bs' in LANGUAGES list mean?
Root cause: Documentation Deficiency

https://github.com/bigcode-project/bigcode-evaluation-harness/blob/5b7e7235d97a45ce7881d8be478c58609dfe93fb/lm_eval/tasks/multiple.py#L39

If "bs" in the Language list means bash, I think it should be changed to "sh".
Even if you look at the dataset name in hf, it is sh, so the current code does not load the dataset properly when working with All_task.
Repository: bigcode-project/bigcode-evaluation-harness
Issue: Support `Salesforce/codet5p-220m` and other `T5ForConditionalGeneration` models
Root cause: Unimplemented Feature Gap

`AutoModelForCausalLM.from_pretrained(Salesforce/codet5p-220m)` [raises](https://app.warp.dev/block/D7QCQSbO1p7dnq9U2qEQG7) `ValueError: Unrecognized configuration class <class 'transformers.models.t5.configuration_t5.T5Config'> for this kind of AutoModel: AutoModelForCausalLM.`. Consider utilizing `AutoModelForSeq2SeqLM`. [Here](https://github.com/keyboardAnt/bigcode-evaluation-harness/commit/856959a105792d066286a091e96c197b013a948a) is a quick draft example.

This was added but only tested for instructcodet5p and HumanEvalPack here https://github.com/bigcode-project/bigcode-evaluation-harness/blob/9d17482e38eaeddee5e27488229a893dc59d42e1/main.py#L274 If more tests are done for other tasks we can generalize it
Repository: bigcode-project/bigcode-evaluation-harness
Issue: how to use --instruction_tokens?


This argument is only used for InstructHumanEval benchmark and its use is explained in this [documentation](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main/docs#instructhumaneval). For example:
```
accelerate launch  main.py \
  --model <MODEL_NAME> \
  --max_length_generation <MAX_LENGTH> \
  --tasks instruction-humaneval \
  --instruction_tokens <user_token>,<end_token>,<assistant_token>\
  --temperature 0.2 \
  --n_samples 50 \
  --batch_size 50 \
  --allow_code_execution
```