Repository: beir-cellar/beir
Issue: Question about ArguAna dataset - same content for the `text` field of queries.jsonl and corpus.jsonl
Root cause: Documentation Deficiency

Hello, 

Thanks for releasing the benchmark datasets. These are great work!

After downloading the ArguAna dataset you provided, we found that the `text` field of queries.jsonl and corpus.jsonl are the same.

Is that on purpose or a bug?

Looking forward to hearing from you, thanks!

Hi @nijianmo,

Yes, this is correct. The Arguana task is to retrieve a counterargument for a given argument. 
The task is different as the queries are also passage-level in the Arguana dataset.

Kind Regards,
Nandan

Thanks for the quick reply.

I'm still a bit confused though. How can it be a counterargument if the text is the same? Here're two examples:

One from from queries.jsonl:
```
{"_id": "test-environment-aeghhgwpe-pro04a", "text": "Being vegetarian reduces risks of food poisoning  Almost all dangerous types of food poisoning are passed on through meat or eggs. So Campylobacter bacteria, the most common cause of food poisoning in England, are usually found in raw meat and poultry, unpasteurised milk and untreated water. Salmonella come from raw meat, poultry and dairy products and most cases of escherichia coli (E-Coli) food poisoning occur after eating undercooked beef or drinking unpasteurised milk. [1]  Close contact between humans and animals also leads to zoonosis \u2013 diseases such as bird \u2018flu which can be passed on from animals to humans. Using animal brains in the processed feed for livestock led to BSE in cattle and to CJD in humans who ate beef from infected cows.  [1] Causes of food poisoning, nhs.co.uk, 23rd June 2009", "metadata": {}}
```

The other from corpus.jsonl:
 ```
{"_id": "test-environment-aeghhgwpe-pro04a", "title": "animals environment general health health general weight philosophy ethics", "text": "Being vegetarian reduces risks of food poisoning  Almost all dangerous types of food poisoning are passed on through meat or eggs. So Campylobacter bacteria, the most common cause of food poisoning in England, are usually found in raw meat and poultry, unpasteurised milk and untreated water. Salmonella come from raw meat, poultry and dairy products and most cases of escherichia coli (E-Coli) food poisoning occur after eating undercooked beef or drinking unpasteurised milk. [1]  Close contact between humans and animals also leads to zoonosis \u2013 diseases such as bird \u2018flu which can be passed on from animals to humans. Using animal brains in the processed feed for livestock led to BSE in cattle and to CJD in humans who ate beef from infected cows.  [1] Causes of food poisoning, nhs.co.uk, 23rd June 2009", "metadata": {}}
```

If the `text` for both query and document are the same, what do we expect the model to know? I think the counterarguments (which are the `text` in queries.jsonl) should be different from the arguments (the `text` in corpus.jsonl).

WDYT?

Thanks!


Hi @nijianmo,

This is indeed a good question. The two examples with the same ``_id`` can be present in both corpus and queries in Arguana.
As the total number of arguments was quite less in Arguana, we included all arguments i.e. the complete dataset as the corpus. This lead to query arguments being present in the corpus. However, there are two things we should note here:

1. The identical or exact document corresponding to a query is not marked as relevant. If you check ``qrels/test.tsv`` for arguana, you will find ``test-environment-aeghhgwpe-pro04a	test-environment-aeghhgwpe-pro04b	1`` in which the counterargument is labelled positive.
2. If the retrieved doc-id is identical to the query-id, it is automatically removed in BEIR. This ensures we are not evaluating on the exact identical document retrieved given the query.

Hope this helps! Let me know incase something else is also required.


Thanks for the explanation. These are helpful!