Title: data leakage in `webis/conclugen` dataset
Author: xflashxx (NONE)
State: closed
Created: 2022-04-30T17:43:37Z
Labels: dataset bug

## Describe the bug
Some samples (argument-conclusion pairs) in the *training* split of the `webis/conclugen` dataset are present in both the *validation* and *test* splits, creating data leakage and distorting model results.
Furthermore, all splits contain duplicate samples.

## Steps to reproduce the bug
```python
from datasets import load_dataset

training = load_dataset("webis/conclugen", "base", split="train")
validation = load_dataset("webis/conclugen", "base", split="validation")
testing = load_dataset("webis/conclugen", "base", split="test")

# collect which sample id's are present in the training split
ids_validation = list()
ids_testing = list()

for train_sample in training:
    train_argument = train_sample["argument"]
    train_conclusion = train_sample["conclusion"]
    train_id = train_sample["id"]
    
    # test if current sample is in validation split
    if train_argument in validation["argument"]:
        for validation_sample in validation:
            validation_argument = validation_sample["argument"]
            validation_conclusion = validation_sample["conclusion"]
            validation_id = validation_sample["id"]
            if train_argument == validation_argument and train_conclusion == validation_conclusion:
                ids_validation.append(validation_id)
    
    # test if current sample is in test split
    if train_argument in testing["argument"]:
        for testing_sample in testing:
            testing_argument = testing_sample["argument"]
            testing_conclusion = testing_sample["conclusion"]
            testing_id = testing_sample["id"]
            if train_argument == testing_argument and train_conclusion == testing_conclusion:
                ids_testing.append(testing_id)
```

## Expected results
Length of both lists `ids_validation` and `ids_testing` should be zero.

## Actual results
Length of `ids_validation` = `2556`
Length of `ids_testing` = `287`

Furthermore, there seems to be duplicate samples in (at least) the *training* split, since:
`print(len(set(ids_validation)))` = `950`
`print(len(set(ids_testing)))` = `101`

All in all, around 7% of the samples of each the *validation* and *test* split seems to be present in the *training* split.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.4
- Platform: macOS-12.3.1-arm64-arm-64bit
- Python version: 3.9.10
- PyArrow version: 7.0.0

Comments:
Hi @xflashxx, thanks for reporting.

Please note that this dataset was generated and shared by Webis Group: https://huggingface.co/webis

We are contacting the dataset owners to inform them about the issue you found. We'll keep you updated of their reply.

i'd suggest just pinging the authors here in the issue if possible?

Thanks for reporting this @xflashxx. I'll have a look and get back to you on this.

Hi @xflashxx and @albertvillanova,

I have updated the files with de-duplicated splits. Apparently the debate portals from which part of the examples were sourced had unique timestamps for some examples (up to 6%; updated counts in the README) without any actual content updated that lead to "new" items. The length of `ids_validation` and `ids_testing` is zero.

Regarding impact on scores:
1. We employed automatic evaluation (on a separate set of 1000 examples) only to justify the exclusion of the smaller models for manual evaluation (due to budget constraints). I am confident the ranking still stands (unsurprisingly, the bigger models doing better than those trained on the smaller splits). We also highlight this in the paper. 

2. The examples used for manual evaluation have no overlap with any splits (also because they do not have any ground truth as we applied the trained models on an unlabeled sample to test its practical usage). I've added these two files to the dataset repository.

Hope this helps!

Thanks @shahbazsyed for your fast fix.

As a side note:
- Your email appearing as Point of Contact in the dataset README has a typo: @uni.leipzig.de instead of @uni-leipzig.de
- Your commits on the Hub are not linked to your profile on the Hub: this is because we use the email address to make this link; the email address used in your commit author and the email address set on your Hub account settings.