rIsHu009/github-issues-updated huggingface dataset:
Title: add One Million Posts Corpus
Author: aseifert (CONTRIBUTOR)
State: closed
Created: 2020-12-08T00:50:08Z

- **Name:** One Million Posts Corpus
- **Description:** The “One Million Posts” corpus is an annotated data set consisting of user comments posted to an Austrian newspaper website (in German language).
- **Paper:** https://dl.acm.org/doi/10.1145/3077136.3080711
- **Data:** https://github.com/OFAI/million-post-corpus
- **Motivation:** Big German (real-life) dataset containing different annotations around forum moderation with expert annotations.

### Checkbox

- [X] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [X] Fill the `_DESCRIPTION` and `_CITATION` variables
- [X] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [X] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [X] Generate the metadata file `dataset_infos.json` for all configurations
- [X] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [X] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [X] Both tests for the real data and the dummy data pass.

Comments:
merging since the CI is fixed on master
Title: oclar-dataset
Author: alaameloh (CONTRIBUTOR)
State: closed
Created: 2020-12-07T23:56:45Z

Opinion Corpus for Lebanese Arabic Reviews (OCLAR) corpus is utilizable for Arabic sentiment classification on reviews, including hotels, restaurants, shops, and others. : [homepage](http://archive.ics.uci.edu/ml/datasets/Opinion+Corpus+for+Lebanese+Arabic+Reviews+%28OCLAR%29#)

Comments:
merging since the CI is fixed on master
Title: Created wiki_movies dataset.
Author: aclifton314 (CONTRIBUTOR)
State: closed
Created: 2020-12-07T23:38:54Z

First PR (ever). Hopefully this movies dataset is useful to others!

Comments:
looks like your PR includes changes about many other files than the ones for wiki_movies

Can you create another branch and another PR please ?

I'm happy to. What's the best way to do that (sorry, I'm new to PRs etc.)?

Sure !

First please save your new dataset files somewhere.
Then you can do in this order:
```
git checkout master
git fetch upstream
git rebase upstream/master
git push
git checkout -b my-new-branch-name
```
This will create a new branch from the updated master branch.
Then you can re-add your files and commit + push them

Once it's done you should be able to create a new PR using your new branch :) 

Done!

closing in favor of #1485
Title: add DFKI SmartData Corpus
Author: aseifert (CONTRIBUTOR)
State: closed
Created: 2020-12-07T23:03:48Z

- **Name:** DFKI SmartData Corpus
- **Description:** DFKI SmartData Corpus is a dataset of 2598 German-language documents which has been annotated with fine-grained geo-entities, such as streets, stops and routes, as well as standard named entity types.
- **Paper:** https://www.dfki.de/fileadmin/user_upload/import/9427_lrec_smartdata_corpus.pdf
- **Data:** https://github.com/DFKI-NLP/smartdata-corpus
- **Motivation:** Contains fine-grained NER labels for German.

### Checkbox

- [X] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [X] Fill the `_DESCRIPTION` and `_CITATION` variables
- [X] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [X] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [X] Generate the metadata file `dataset_infos.json` for all configurations
- [X] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [X] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [X] Both tests for the real data and the dummy data pass.

Comments:
Title: Adding OneStopEnglish corpus dataset
Author: purvimisal (CONTRIBUTOR)
State: closed
Created: 2020-12-07T22:05:11Z

This PR adds OneStopEnglish Corpus containing texts classified into reading levels (elementary, intermediate, advance) for automatic readability assessment and text simplification. 

Link to the paper: https://www.aclweb.org/anthology/W18-0535.pdf

Comments:
Hi @lhoestq, thanks for the review.
I have made all the changes, PTAL! :)
Title: new pr for Turkish NER
Author: merveenoyan (CONTRIBUTOR)
State: closed
Created: 2020-12-07T21:40:26Z


Comments:
Can you run `make style` to fix the code format ?

Also it looks like the file `file_downloaded/TWNERTC_TC_Coarse Grained NER_DomainIndependent_NoiseReduction.zip/TWNERTC_TC_Coarse Grained NER_DomainIndependent_NoiseReduction.DUMP` is missing inside the dummy_data.zip


(note that `TWNERTC_TC_Coarse Grained NER_DomainIndependent_NoiseReduction.zip` is a directory name, not an actual zip file)

Hi Quentin, thank you for your patience with me. I've fixed the preprocessing pipeline, got this very weird error that Yacine told me to push. I've pushed it and after I'll find out that it will work, I will have my final pr on styling.

looks like you removed the dataset script file in your latest commit, is it expected ?
Title: Add CovidQA dataset
Author: olinguyen (CONTRIBUTOR)
State: closed
Created: 2020-12-07T17:06:51Z

This PR adds CovidQA, a question answering dataset specifically designed for COVID-19, built by hand from knowledge gathered from Kaggle’s COVID-19 Open Research Dataset Challenge.

Link to the paper: https://arxiv.org/pdf/2004.11339.pdf
Link to the homepage: https://covidqa.ai

Comments:
It seems to share the same name as this dataset: https://openreview.net/forum?id=JENSKEEzsoU

> It seems to share the same name as this dataset: https://openreview.net/forum?id=JENSKEEzsoU

you're right it can be confusing. I'll add the organization/research group for clarity: `covid_qa_castorini`. I added the dataset you shared as `covid_qa_deepset` in another PR (#1182) 

Thanks for avoiding the name collision !
Title: enriched webnlg dataset rebase
Author: TevenLeScao (CONTRIBUTOR)
State: closed
Created: 2020-12-07T17:05:45Z

Rebase of #1206 !

Comments:
I've removed the `en` within `de` and reciprocally; but I don't think I will be able to thin it more than this. (Edit: ignore the close, I missclicked !)
Title: Added NewsPH Raw Dataset
Author: jcblaisecruz02 (NONE)
State: closed
Created: 2020-12-07T15:17:53Z

Added the raw version of the NewsPH dataset, which was used to automatically generate the NewsPH-NLI corpus. Dataset of news articles in Filipino from mainstream Philippine news sites on the internet. Can be used as a language modeling dataset or to reproduce the NewsPH-NLI dataset.

Paper: https://arxiv.org/abs/2010.11574
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

Comments:
looks like this PR has changes to many files other than the ones for `NewsPH`

Can you create another branch and another PR please ?
Title: Add KorQPair dataset
Author: jaketae (CONTRIBUTOR)
State: closed
Created: 2020-12-07T14:33:57Z

This PR adds a [Korean paired question dataset](https://github.com/songys/Question_pair) containing labels indicating whether two questions in a given pair are semantically identical. This dataset was used to evaluate the performance of [KoGPT2](https://github.com/SKT-AI/KoGPT2#subtask-evaluations) on a phrase detection downstream task.

Comments:
dummy data is missing

Hey @cceyda, thanks for pointing that out. I thought I'd added it, but seems like that wasn't the case. Just pushed a new commit with the dummy data.
Title: Added WikiText-TL-39
Author: jcblaisecruz02 (NONE)
State: closed
Created: 2020-12-07T13:43:48Z

This PR adds the WikiText-TL-39 Filipino Language Modeling dataset.

Paper: https://arxiv.org/abs/1907.00409
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

Comments:
looks like this PR also includes changes about another dataset `covid_qa_deepset`

Could you create another branch and another PR that only includes the changes for the wikitext-tl-39 dataset ?
Title: add thainer
Author: cstorm125 (CONTRIBUTOR)
State: closed
Created: 2020-12-07T13:41:54Z

ThaiNER (v1.3) is a 6,456-sentence named entity recognition dataset created from expanding the 2,258-sentence
[unnamed dataset](http://pioneer.chula.ac.th/~awirote/Data-Nutcha.zip) by
[Tirasaroj and Aroonmanakun (2012)](http://pioneer.chula.ac.th/~awirote/publications/).
It is used to train NER taggers in [PyThaiNLP](https://github.com/PyThaiNLP/pythainlp).
The NER tags are annotated by [Tirasaroj and Aroonmanakun (2012)]((http://pioneer.chula.ac.th/~awirote/publications/))
for 2,258 sentences and the rest by [@wannaphong](https://github.com/wannaphong/).
The POS tags are done by [PyThaiNLP](https://github.com/PyThaiNLP/pythainlp)'s `perceptron` engine trained on `orchid_ud`.
[@wannaphong](https://github.com/wannaphong/) is now the only maintainer of this dataset.

Comments:
Title: Add Naver sentiment movie corpus
Author: jaketae (CONTRIBUTOR)
State: closed
Created: 2020-12-07T13:33:45Z

Supersedes #1168 

> This PR adds the [Naver sentiment movie corpus](https://github.com/e9t/nsmc), a dataset containing Korean movie reviews from Naver, the most commonly used search engine in Korea. This dataset is often used to benchmark models on Korean NLP tasks, as seen in [this paper](https://www.aclweb.org/anthology/2020.lrec-1.199.pdf).

Comments:
Title: Add doc2dial dataset
Author: KMFODA (CONTRIBUTOR)
State: closed
Created: 2020-12-07T12:39:09Z

### Doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset v0.9

Once complete this will add the [Doc2dial](https://doc2dial.github.io/data.html) dataset from the generic data sets list.

Comments:
It not always practical to use nested `Sequence`. If you have troubles with sequence you can use lists instead. 

For example
```python

features=datasets.Features(
    {
        "dial_id": datasets.Value("string"),
        "doc_id": datasets.Value("string"),
        "domain": datasets.Value("string"),
        "turns": [
            {
                "turn_id": datasets.Value("int32"),
                "role": datasets.Value("string"),
                "da": datasets.Value("string"),
                "reference": [
                    {
                        "keys" : datasets.Value("string"),
                        "values": datasets.Value("string"),   
                    }

                ],
                "utterance": datasets.Value("string"),
            }
        ],
    }
),
```

this way `turns` will be a list of dict, and the "reference" key of `turns` will be a list of dict as well

No problem thanks for all your help getting this to the final stages! Added .gitignore, removed .lock and applied the changes you asked for.
Title: Adding indonlu dataset
Author: yasirabd (CONTRIBUTOR)
State: closed
Created: 2020-12-07T11:38:45Z

IndoNLU benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems for Bahasa Indonesia. It contains 12 datasets.

Comments:
looks like this PR includes changes about many files other than the ones for IndoNLU
Could you create another branch and another PR please ?

> looks like this PR includes changes about many files other than the ones for IndoNLU
> Could you create another branch and another PR please ?

Okay I'll make it
Title: Add Google Turkish Treebank Dataset
Author: abhishekkrthakur (CONTRIBUTOR)
State: closed
Created: 2020-12-07T11:09:17Z
Labels: dataset contribution


Comments:
Thanks for your contribution, @abhishekkrthakur. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Title: Add Google Noun Verb Dataset
Author: abhishekkrthakur (CONTRIBUTOR)
State: closed
Created: 2020-12-07T10:26:05Z
Labels: dataset contribution


Comments:
Thanks for your contribution, @abhishekkrthakur. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Title: adding bprec
Author: kldarek (CONTRIBUTOR)
State: closed
Created: 2020-12-07T10:15:49Z


Comments:
looks like this PR includes changes to many files other than the ones related to bprec
Can you create another branch and another PR please ?

> looks like this PR includes changes to many files other than the ones related to bprec
> Can you create another branch and another PR please ?

Yes, I realized I messed this one up, learning my way :) I'll close this one and open another hopefully clean PR :) Thanks!
Title: Multi Domain Sentiment Analysis Dataset (MDSA)
Author: abhishekkrthakur (CONTRIBUTOR)
State: closed
Created: 2020-12-07T09:57:15Z
Labels: dataset contribution


Comments:
can you also run `make style` to format the code ?

I'll come back to this one in sometime :) @lhoestq 

Also if you would use `xml.etree.ElementTree` to parse the XML it would be awesome, because right now you're using an external dependency `xmltodict `

> Also if you would use xml.etree.ElementTree to parse the XML it would be awesome, because right now you're using an external dependency xmltodict

Its pseudo xml so elementtree fails. xmltodict seems to be working quite good for this. do we have examples of pseudo xml datasets?

for the other pseudo xml the text is parsed manually

Can you add `xmltodict` to the test dependencies in setup.py please to fix the CI please ?

Also can you add the dataset card with the tags and run `make style` ?

Hi :) have you had a chance to fix the test dependency and apply `make style` ?

Feel fee to ping me when it's ready for a review

Thanks for your contribution, @abhishekkrthakur. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Title: Add AmbigQA dataset
Author: cceyda (CONTRIBUTOR)
State: closed
Created: 2020-12-07T09:07:19Z

# AmbigQA: Answering Ambiguous Open-domain Questions Dataset
Adding the [AmbigQA](https://nlp.cs.washington.edu/ambigqa/) dataset as part of the sprint 🎉 (from Open dataset list for Dataset sprint)

Added both the light and full versions (as seen on the dataset homepage)
The json format changes based on the value of one 'type' field, so I set the unavailable field to an empty list. This is explained in the README -> Data Fields

```py
train_light_dataset = load_dataset('./datasets/ambig_qa',"light",split="train")
val_light_dataset = load_dataset('./datasets/ambig_qa',"light",split="validation")
train_full_dataset = load_dataset('./datasets/ambig_qa',"full",split="train")
val_full_dataset = load_dataset('./datasets/ambig_qa',"full",split="validation")


for example in train_light_dataset:
    for i,t in enumerate(example['annotations']['type']):
        if t =='singleAnswer':
            # use the example['annotations']['answer'][i]
            # example['annotations']['qaPairs'][i] - > is []
            print(example['annotations']['answer'][i])
        else:
            # use the example['annotations']['qaPairs'][i]
            # example['annotations']['answer'][i] - > is []
            print(example['annotations']['qaPairs'][i])

```

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)

Comments:
Title: Wino bias
Author: akshayb7 (CONTRIBUTOR)
State: closed
Created: 2020-12-07T07:12:42Z

The PR will fail circleCi tests because of the requirement of manual loading of data. Fresh PR because of messed up history of the previous one.

Comments:
Closing this PR because of messed up history and opening another one after discussion with Quentin Lhoest.
Title: Added ade_corpus_v2, with 3 configs for relation extraction and classification task
Author: Nilanshrajput (CONTRIBUTOR)
State: closed
Created: 2020-12-07T07:05:14Z

Adverse Drug Reaction Data: ADE-Corpus-V2 dataset added configs for different tasks with given data

Comments:
@lhoestq I have added the tags they are in separate files for 3 different configs

@lhoestq  thanks for the review I added your suggested changes.

merging since the CI is fixed on master
Title: Add Curiosity Dialogs Dataset
Author: vineeths96 (CONTRIBUTOR)
State: closed
Created: 2020-12-07T06:01:00Z

Add Facebook [Curiosity Dialogs](https://github.com/facebookresearch/curiosity) Dataset.

Comments:
@lhoestq I tried manually creating the dummy files. But unfortunately it was raising an error during testing the dummy data (regarding JSON parsing).

The JSONs are pretty big so I cannot actually open it without crashing the text editor.

 Do you have any suggestions?

@lhoestq I have made all the changes you mentioned.
Title: Muchocine - Spanish movie reviews dataset
Author: mapmeld (CONTRIBUTOR)
State: closed
Created: 2020-12-07T02:23:29Z


Comments:
Hi @mapmeld !
have you had a chance to take a look at my suggestions ?

Feel free to ping me if you have questions or when you're ready for a review

@lhoestq unfortunately I don't have any more information about where the dataset comes from

It's fine, you can just add the sections titles back and leave the content with `[More Information Needed]`



added missing sections, updated the Python code ✅
Title: adding conceptnet5
Author: huu4ontocord (CONTRIBUTOR)
State: closed
Created: 2020-12-06T21:06:53Z

Adding the conceptnet5 and omcs txt files used to create the conceptnet5 dataset. Conceptne5 is a common sense dataset. More info can be found here: https://github.com/commonsense/conceptnet5/wiki

Comments:
Thank you. I'll make those changes. but I'm having problems trying to push my changes to my fork


Hi, I've removed the TODO, and added a README.md. How do I push these changes?


Also, what docstring are you recommending?


> Hi, I've removed the TODO, and added a README.md. How do I push these changes?

you can just commit and push your changes to the same branch as your first commit.

@ghomasHudson I've tried it but still getting code quality error.  I've removed all blank lines, etc. required by flake8. Don't know what else to do

> @ghomasHudson I've tried it but still getting code quality error. I've removed all blank lines, etc. required by flake8. Don't know what else to do

Did you run `make style` before committing? When I run it, it fixes some things (e.g. Splitting line 96 which is currently too long).

I think @yjernite is looking into this. I did "make style" but nothing happens

looks like your PR includes changes about many other files than the ones related to conceptnet5

could you create another branch and another PR please ?

@lhoestq I'm not sure what I did wrong. What did I push that wasn't conceptnet5? How do I see this?

 did this

make style
flake8 datasets
git add datasets/<your_dataset_name>
git commit
git fetch upstream
git rebase upstream/master
git pull
git push -u origin conceptnet5

Thanks for rebasing and force push :) 

Yeah! Thank you @lhoestq, @ghomasHudson  and @yjernite !
Title: Add numeric fused head dataset
Author: ghomasHudson (CONTRIBUTOR)
State: closed
Created: 2020-12-06T20:46:53Z

Adding the [NFH: Numeric Fused Head](https://nlp.biu.ac.il/~lazary/fh/) dataset.

Everything looks sensible and I've included both the identification and resolution tasks. I haven't personally used this dataset in my research so am unable to specify what the default configuration / supervised keys should be.

I've filled out the basic info on the model card to the best of my knowledge but it's a little tricky to understand exactly what the fields represent.

Dataset author: @yanaiela

Comments:
> Thanks for adding this @ghomasHudson!
> I added some comments for some of the fields.
> 
> Also, I'm not sure about this since I haven't used the library yet, but maybe it's worth adding the identification and resolution as two separate datasets?

Thanks for replying @yanaiela - I hope this will make your dataset more accessible to a wider audience - I've added the changes to the model card you suggested.

In terms of the identification and resolution tasks, I've currently added them as separate `splits` in huggingface/datasets so you can load identification like this:

```
import datasets
dataset = datasets.load_dataset("numeric_fused_head", "identification")
print(dataset["train"][0])
>> {"tokens": ["The", "quick", "brown", "fox",....], "start_index": 11, "end_index": 12, "label": 0}
```
And resolution like this:

```
import datasets
dataset = datasets.load_dataset("numeric_fused_head", "resolution")
print(dataset["train"][0])
>> {"tokens": ["The", "quick", "brown", "fox",....], "head": ["AGE"], "anchors_indices": [12], ...}
```

I hope so too, thanks!

Re the splits, that makes sense to me.
Title: Add HKCanCor
Author: j-chim (CONTRIBUTOR)
State: closed
Created: 2020-12-06T20:32:07Z

This PR adds the [Hong Kong Cantonese Corpus](http://compling.hss.ntu.edu.sg/hkcancor/), by [Luke and Wong 2015](http://compling.hss.ntu.edu.sg/hkcancor/data/LukeWong_Hong-Kong-Cantonese-Corpus.pdf). 

The dummy data included here was manually created, as the original dataset uses a xml-like format (see a copy hosted [here](https://github.com/fcbond/hkcancor/blob/master/sample/d1_v.txt) for example) that requires a few processing steps.

Comments:
Title: add Korean HateSpeech dataset
Author: stevhliu (MEMBER)
State: closed
Created: 2020-12-06T20:31:29Z


Comments:
It looks like you forgot to `make style` (I forget it a lot too 🤦 )
+ add dummy data

hi @cceyda 👋, thanks for the hint! it looks like i've run into some other errors though in `_split_generators` or `_generate_examples`.  do you have any idea of what's wrong here? 😅

I get the same errors on another pr too, so it probably has something to do with circleci, waiting on help.

the `RemoteDatasetTest ` error on the CI is fixed on master so it's fine

merging since the CI is fixed on master
Title: Add Korean NER dataset
Author: jaketae (CONTRIBUTOR)
State: closed
Created: 2020-12-06T20:19:06Z

Supersedes #1177 

> This PR adds the [Korean named entity recognition dataset](https://github.com/kmounlp/NER). This dataset has been used in many downstream tasks, such as training [KoBERT](https://github.com/SKTBrain/KoBERT) for NER, as seen in this [KoBERT-CRF implementation](https://github.com/eagle705/pytorch-bert-crf-ner).

Comments:
Title: Add WMT20 MLQE 3 shared tasks
Author: VictorSanh (CONTRIBUTOR)
State: closed
Created: 2020-12-06T19:59:12Z

3 tasks for the WMT 20 MLQE shared tasks -> 3 different datasets

(I re-created #1137 because it was too messy).

Note that in L199 `task3.py`, I used `logging.warning` to print some missing data in the train set.

Comments:
Thanks for the comments Quentin!
I integrated them

It should be ok now!
Sorry I wasn't attentive enough.
(tests are currently failing, I understand it's from other datasets)

merging since the CI is fixed on master
Title: Add limit
Author: j-chim (CONTRIBUTOR)
State: closed
Created: 2020-12-06T19:46:18Z

This PR adds [LiMiT](https://github.com/ilmgut/limit_dataset), a dataset for literal motion classification/extraction by [Manotas et al., 2020](https://www.aclweb.org/anthology/2020.findings-emnlp.88.pdf).

Comments:
My bad, didn't see this on the open dataset list. Closing this since it overlaps with PR#1256
Title: Add irc disentanglement
Author: dhruvjoshi1998 (CONTRIBUTOR)
State: closed
Created: 2020-12-06T19:30:46Z

added files for irc disentanglement dataset
was unable to test dummy data as a result of vpn/proxy issues

Comments:
looks like this PR includes changes about many files other than the ones for irc_disentanglement

Could you please create a new branch and create another PR please ?

closing in favor of #1586
Title: adding medical-questions-pairs dataset
Author: tuner007 (CONTRIBUTOR)
State: closed
Created: 2020-12-06T19:30:12Z

This dataset consists of 3048 similar and dissimilar medical question pairs hand-generated and labeled by Curai's doctors.
Dataset : https://github.com/curai/medical-question-pair-dataset
Paper : https://drive.google.com/file/d/1CHPGBXkvZuZc8hpr46HeHU6U6jnVze-s/view

Comments:
Title: add taskmaster3
Author: patil-suraj (CONTRIBUTOR)
State: closed
Created: 2020-12-06T17:56:03Z

Adding Taskmaster-3 dataset
https://github.com/google-research-datasets/Taskmaster/tree/master/TM-3-2020.

The dataset structure almost same as original dataset with these two changes

1. In original dataset, each `apis` has a `args` filed which is a `dict`  with variable keys, which represent the name and value of the args. Here converted that to a `list` of `dict` with keys `arg_name` and `arg_value`. For ex.

```python
args = {"name.movie": "Mulan", "name.theater": ": "Mountain AMC 16"}
```
becomes 
```python
[
  {
    "arg_name": "name.movie",
    "arg_value": "Mulan"
  },
  {
    "arg_name": "name.theater",
    "arg_value": "Mountain AMC 16"
  }
]
```

2. Each `apis` has a `response` which is also a `dict` with variable keys representing response name/type and it's value. As above converted it to `list` of `dict` with keys `response_name` and `response_value`.

Comments:
(you were unlucky, my rule of thumb for reducing the dummy data is to check whether they're above 50KB and you're at 52KB ^^')

> (you were unlucky, my rule of thumb for reducing the dummy data is to check whether they're above 50KB and you're at 52KB ^^')

Oops :(

Thanks for the suggestion, will reduce the size
Title: Add large spanish corpus
Author: lewtun (MEMBER)
State: closed
Created: 2020-12-06T17:06:50Z

Adds a collection of Spanish corpora that can be useful for pretraining language models. 

Following a nice suggestion from @yjernite we provide the user with three main ways to preprocess / load either 

* the whole corpus (17GB!)
* one specific sub-corpus
* the whole corpus, but return a single split. this is useful if you want to cache the whole preprocessing step once and interact with individual sub-corpora

See the dataset card for more details.

Ready for review!

Comments:
Title: [AfriBooms] Dataset exists already
Author: patrickvonplaten (CONTRIBUTOR)
State: closed
Created: 2020-12-06T16:35:13Z

When trying to add "AfriBooms": https://docs.google.com/spreadsheets/d/12ShVow0M6RavnzbBEabm5j5dv12zBaf0y-niwEPPlo4/edit#gid=1386399609 I noticed that the dataset exists already as a config of Universal Dependencies (universal_dependencies.py). I checked and the data exactly matches so that the new data link does not give any new data.

This PR improves the config's description a bit by linking to the paper.

Comments:
It's so cool seeing all these datasets fly by and see how they are still of interest. I did my internship at the research group of Liesbeth Augustinus et al. They're a very kind group of people!

merging since the CI is fixed on master
Title: Adding Enriched WebNLG dataset
Author: TevenLeScao (CONTRIBUTOR)
State: closed
Created: 2020-12-06T15:36:20Z

This pull requests adds the `en` and `de` versions of the [Enriched WebNLG](https://github.com/ThiagoCF05/webnlg) dataset

Comments:
Nice :) 

could you add the tags and also remove all the dummy data files that are not zipped ? The diff currently shows 800 files changes xD

Aaaaand it's rebase time - the new one is at #1264 !

closing this one since a new PR was created
Title: add lst20 with manual download
Author: cstorm125 (CONTRIBUTOR)
State: closed
Created: 2020-12-06T14:49:10Z

passed on local:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_lst20
```
Not sure how to test:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_lst20
```

```
LST20 Corpus is a dataset for Thai language processing developed by National Electronics and Computer Technology Center (NECTEC), Thailand.
It offers five layers of linguistic annotation: word boundaries, POS tagging, named entities, clause boundaries, and sentence boundaries.
At a large scale, it consists of 3,164,002 words, 288,020 named entities, 248,181 clauses, and 74,180 sentences, while it is annotated with
16 distinct POS tags. All 3,745 documents are also annotated with one of 15 news genres. Regarding its sheer size, this dataset is
considered large enough for developing joint neural models for NLP.
Manually download at https://aiforthai.in.th/corpus.php
```

Comments:
The pytest suite doesn't allow manual downloads so we just make sure that the `datasets-cli test` command to run without errors instead

@lhoestq Changes made. Thank you for the review. I've made some same mistakes for https://github.com/huggingface/datasets/pull/1253 too. Will fix them before review.
Title: Add Neural Code Search Dataset
Author: vinaykudari (CONTRIBUTOR)
State: closed
Created: 2020-12-06T14:12:39Z


Comments:
> Really good thanks !
> 
> I left a few comments

Thanks, resolved them :) 

looks like this PR includes changes about many other files than the ones for Code Search

can you create another branch and another PR please ?

> looks like this PR includes changes about many other files than the ones for Code Search
> 
> can you create another branch and another PR please ?

Okay sure
Title: Medical question pairs
Author: tuner007 (CONTRIBUTOR)
State: closed
Created: 2020-12-06T14:09:07Z

This dataset consists of 3048 similar and dissimilar medical question pairs hand-generated and labeled by Curai's doctors.
Dataset : https://github.com/curai/medical-question-pair-dataset
Paper : https://drive.google.com/file/d/1CHPGBXkvZuZc8hpr46HeHU6U6jnVze-s/view
**No splits added**

Comments:
Title: Update ADD_NEW_DATASET.md
Author: BramVanroy (CONTRIBUTOR)
State: closed
Created: 2020-12-06T13:31:32Z

Windows needs special treatment again: unfortunately adding `torch` to the requirements does not work well (crashing the installation). Users should first install torch manually and then continue with the other commands.

This issue arises all the time when adding torch as a dependency, but because so many novice users seem to participate in adding datasets, it may be useful to add an explicit note for Windows users to ensure that they do not run into issues.

Comments:
Title: Turkish NER dataset, script works fine, couldn't generate dummy data
Author: merveenoyan (CONTRIBUTOR)
State: closed
Created: 2020-12-06T12:00:03Z

I've written the script (Turkish_NER.py) that includes dataset. The dataset is a zip inside another zip, and it's extracted as .DUMP file. However, after preprocessing I only get .arrow file. After I ran the script with no error messages, I get .arrow file of dataset, LICENSE and dataset_info.json.

Comments:
the .DUMP file looks like a txt with one example per line so adding `--match_text_files *.DUMP --n_lines 50`  to the dummy generation command might work .

We can close this PR since a new PR was open at #1268
Title: Add ALT
Author: chameleonTK (CONTRIBUTOR)
State: closed
Created: 2020-12-06T11:25:30Z

ALT dataset -- https://www2.nict.go.jp/astrec-att/member/mutiyama/ALT/

Comments:
the `RemoteDatasetTest ` erros in the CI are fixed on master so it's fine

used `Translation ` feature type and fixed few typos as you suggested.

Sorry, I made a mistake. please see new PR here. https://github.com/huggingface/datasets/pull/1436
Title: Add IWSLT'15 English-Vietnamese machine translation Data
Author: Nilanshrajput (CONTRIBUTOR)
State: closed
Created: 2020-12-06T10:36:31Z

Preprocessed Dataset from IWSLT'15 English-Vietnamese machine translation: English-Vietnamese.

from https://nlp.stanford.edu/projects/nmt/data/iwslt15.en-vi/

Comments:
Thanks ! feel free to ping me once you've added the tags in the dataset card :) 

merging since the CI is fixed on master
Title: addition of py_ast
Author: reshinthadithyan (CONTRIBUTOR)
State: closed
Created: 2020-12-06T10:00:52Z

The dataset consists of parsed Parsed ASTs that were used to train and evaluate the DeepSyn tool. 
The Python programs are collected from GitHub repositories
by removing duplicate files, removing project forks (copy of another existing repository)
,keeping only programs that parse and have at most 30'000 nodes in the AST and 
we aim to remove obfuscated files

Comments:
Hi @reshinthadithyan !

As mentioned on the Slack, it would be better in this case to parse the file lines into the following feature structure:
```python
"ast": datasets.Sequence(
    {
        "type": datasets.Value("string"),
        "value": datasets.Value("string"),
        "children": datasets.Sequence(datasets.Value("int32")),
    },
)
```

Here are a few more things to fix before we can move forward:
- the class name needs to be the CamelCase equivalent of the script name, so here it will have to be `PyAst`
- the `README.md` needs to have the tags at the top
- The homepage/info list at the top should be in the same format as the template (added a suggestion)
- You should add the dataset tags and field description to the README as described here: https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#tag-the-dataset-and-write-the-dataset-card

Good luck, let us know if you need any help!

Hello @yjernite, changes have been made as we talked. Hope this would suffice. Thanks. Feel free to point out any room to improvement.

Good progress! Here's what still needs to be done:
- first, you need to rebase to master for the tests to pass :)
- the information in your `Data Fields` paragraph should go into `Data Instances`. Data fields should describe the fields one by one, as in e.g. https://github.com/huggingface/datasets/tree/master/datasets/eli5#data-fields
- you still need to add the YAML tags obtained with the tagging app

Should be good to go after that!

Hello @yjernite, changes as talked are being done.

Looks like this PR includes changes about many other files than the ones for py_ast

Could you create another branch and another PR please ?
Title: Add msr_text_compression
Author: jeromeku (CONTRIBUTOR)
State: closed
Created: 2020-12-06T09:06:11Z

Add [MSR Abstractive Text Compression Dataset](https://msropendata.com/datasets/f8ce2ec9-7fbd-48f7-a8bb-2d2279373563)

Comments:
the `RemoteDatasetTest ` error in the CI is fixed on master so it's fine
Title: Add NewsPH_NLI dataset
Author: anaerobeth (CONTRIBUTOR)
State: closed
Created: 2020-12-06T04:00:31Z

This PR adds the NewsPH-NLI Dataset, the first benchmark dataset for sentence entailment in the low-resource Filipino language. Constructed through exploting the structure of news articles. Contains 600,000 premise-hypothesis pairs, in 70-15-15 split for training, validation, and testing.

Link to the paper: https://arxiv.org/pdf/2010.11574.pdf

Link to the dataset/repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

Comments:
Title: Added Translator Human Parity Data For a Chinese-English news transla…
Author: leoxzhao (CONTRIBUTOR)
State: closed
Created: 2020-12-06T03:34:13Z

…tion system from Open dataset list for Dataset sprint, Microsoft Datasets tab.

Comments:
Can you run `make style` to format the code and fix the CI please ?

> Can you run `make style` to format the code and fix the CI please ?

I ran `make style` before this PR and just a few minutes ago. No changes to the code. Not sure why the CI is failing.

Also, I attempted to see if I can get the source Chinese sentences from `wmt17` dataset. But this call `data = load_dataset('wmt17', "zh-en")` failed with this error: `FileNotFoundError: Couldn't find file at https://storage.googleapis.com/tfdataset-data/downloadataset/uncorpus/UNv1.0.en-zh.tar.gz`. I think it should be possible and fairly straightforward to get the pairing source sentences from it. I just can not test it right now.

The `RemoteDatasetTest ` errors in the CI are fixed on master so it's fine

merging since the CI is fixed on master
Title: Add Fake News Detection in Filipino dataset
Author: anaerobeth (CONTRIBUTOR)
State: closed
Created: 2020-12-06T03:12:15Z

This PR adds the Fake News Filipino Dataset, a low-resource fake news detection corpora in Filipino. Contains 3,206 expertly-labeled news samples, half of which are real and half of which are fake.

Link to the paper: http://www.lrec-conf.org/proceedings/lrec2020/index.html

Link to the dataset/repo: https://github.com/jcblaisecruz02/Tagalog-fake-news

Comments:
Hi! I'm the author of this paper (surprised to see our datasets have been added already).

That paper link only leads to the conference index, here's a link to the actual paper: https://www.aclweb.org/anthology/2020.lrec-1.316/

Would it be fine if I also edited your gsheet entry to reflect this change?

Hi Jan, please go ahead and update. I see you are also in the sprint slack channel. Let me know if what else needs updating. Thanks.
Title: Add Dengue dataset in Filipino
Author: anaerobeth (CONTRIBUTOR)
State: closed
Created: 2020-12-06T02:50:47Z

This PR adds the Dengue Dataset, a benchmark dataset for low-resource multiclass classification, with 4,015 training, 500 testing, and 500 validation examples, each labeled as part of five classes. Each sample can be a part of multiple classes. Collected as tweets.

Link to the paper: https://ieeexplore.ieee.org/document/8459963

Link to the dataset/repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

Comments:
Title: adding hind_encorp dataset
Author: rahul-art (CONTRIBUTOR)
State: closed
Created: 2020-12-06T02:18:45Z

adding Hindi_Encorp05 dataset

Comments:
help needed in dummy data

extension of the file is .plaintext so dummy data generation is failing


you can add the `--match_text_file "*.plaintext"` flag when generating the dummy data

also it looks like the PR is empty, is this expected ?

yes it is expected because I  made all my changes in  PR  #1186  then I again run code and open PR #1188 to see if this time test passes or not only so there is no code change from #1186 to #1188  
i tried  --match_text_file "*.plaintext" this time it is also not generating dummy data don't know why

well this PR includes no code change at all, can you make sure you added your changes in this one ?

feel free to ping me when you have added the files so I can take a look and help you with the dummy data

how to do that i dont know did i have to open new PR


 actually all my changes are visible in  #1186 but  don't know how to show same changes here

these are a the which i did in #1186 and same in #1188 
![1](https://user-images.githubusercontent.com/56379013/101646577-b4864500-3a5d-11eb-8a5a-91b1b441040a.png)
![2](https://user-images.githubusercontent.com/56379013/101646965-32e2e700-3a5e-11eb-94d9-276e602c6ded.png)
![4](https://user-images.githubusercontent.com/56379013/101646989-38d8c800-3a5e-11eb-92bb-d9c4cb2c3595.png)
![5](https://user-images.githubusercontent.com/56379013/101647017-41c99980-3a5e-11eb-87cf-5268e79df19d.png)
![6](https://user-images.githubusercontent.com/56379013/101647038-48581100-3a5e-11eb-8d05-f67834fcaa7b.png)

![8](https://user-images.githubusercontent.com/56379013/101647080-55750000-3a5e-11eb-8455-8936a35b35c2.png)
![9](https://user-images.githubusercontent.com/56379013/101647084-55750000-3a5e-11eb-988e-ae87f0b252a0.png)
![10](https://user-images.githubusercontent.com/56379013/101647182-6f164780-3a5e-11eb-8af3-f0b0186483c9.png)
![11](https://user-images.githubusercontent.com/56379013/101647230-7c333680-3a5e-11eb-9aeb-2b4ce65965e0.png)
![13](https://user-images.githubusercontent.com/56379013/101647257-848b7180-3a5e-11eb-871c-2fd77b047320.png)
![14](https://user-images.githubusercontent.com/56379013/101647268-89502580-3a5e-11eb-9e2a-b9f7ff1fc95e.png)
these same codes are in both #1186 and #1188 so because it is already present from PR #1186 because of that it is showing zeor code change in #1188 because it is already present from #1186 how i can show or highlight those changes











well for me https://github.com/huggingface/datasets/pull/1188/files is blank

This PR tries to merge the master branch of you fork into this repo, however I can't find changes with your files inside your master branch.

Maybe you can fork again the repo and try to create another PR ?

@lhoestq  i  opened a new pr #1438 but this time it fails many circl ci tests

Closing this one since a new PR was created
Title: Add Hate Speech Dataset in Filipino
Author: anaerobeth (CONTRIBUTOR)
State: closed
Created: 2020-12-06T02:01:56Z

This PR adds the Hate Speech Dataset, a text classification dataset in Filipino, consisting 10k tweets (training set) that are labeled as hate speech or non-hate speech. Released with 4,232 validation and 4,232 testing samples. Collected during the 2016 Philippine Presidential Elections.

Link to the paper: https://pcj.csp.org.ph/index.php/pcj/issue/download/29/PCJ%20V14%20N1%20pp1-14%202019

Link to the dataset/repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

Comments:
Title: Add Adversarial SQuAD dataset
Author: cceyda (CONTRIBUTOR)
State: closed
Created: 2020-12-05T23:51:57Z

# Adversarial SQuAD

Adding the Adversarial [SQuAD](https://github.com/robinjia/adversarial-squad) dataset as part of the sprint 🎉 
This dataset adds adversarial sentences to a subset of the SQuAD dataset's dev examples. How to get the original squad example id is explained in readme->Data Instances. The whole data is intended for use in evaluation. (Which could of course be also used for training if one wants). So there is no classical train/val/test split, but a split based on the number of adversaries added.

There are 2 splits of this dataset:

- AddSent: Has up to five candidate adversarial sentences that don't answer the question, but have a lot of words in common with the question. This adversary is does not query the model in any way.
- AddOneSent: Similar to AddSent, but just one candidate sentences was picked at random. This adversary is does not query the model in any way.

(The AddAny and AddCommon datasets mentioned in the paper are dynamically generated based on model's output distribution thus are not included here)

The failing test look like some unrelated timeout thing, will probably clear if rerun.
- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)

Comments:
the CI error was just a connection error due to all the activity on the repo this week ^^'
I re-ran it so it should be good now

I hadn't realized the problem with the dummies since it had passed without errors.
Suggestion: maybe we can show the user a warning based on the generated dummy size.

Thanks for changing to configs ! Looks all good now :) 

Before we merge, can you re-lighten the dummy data please if you don't mind ? The idea is to have them weigh only a few KB (currently it's 50KB each). Feel free to remove any unnecessary files or chunk of text

(also you can ignore the `RemoteDatasetTest ` CI errors, they're fixed on master )

merging since the CI is fixed on master
Title: ADD COVID-QA dataset
Author: olinguyen (CONTRIBUTOR)
State: closed
Created: 2020-12-05T23:31:56Z

This PR adds the COVID-QA dataset, a question answering dataset consisting of 2,019 question/answer pairs annotated by volunteer biomedical experts on scientific articles related to COVID-19

Link to the paper: https://openreview.net/forum?id=JENSKEEzsoU
Link to the dataset/repo: https://github.com/deepset-ai/COVID-QA

Comments:
merging since the CI is fixed on master

Wow, thanks for including this dataset from my side as well!
Title: added emotions detection in arabic dataset
Author: abdulelahsm (CONTRIBUTOR)
State: closed
Created: 2020-12-05T22:08:46Z

Dataset for Emotions detection in Arabic text

more info: https://github.com/AmrMehasseb/Emotional-Tone

Comments:
Hi @abdulelahsm did you manage to fix your issue ?
Feel free to ping me if you have questions or if you're ready for a review

@lhoestq fixed it! ready to merge. I hope haha

merging since the CI is fixed on master
Title: Add  KorQuAD v2 Dataset
Author: cceyda (CONTRIBUTOR)
State: closed
Created: 2020-12-05T21:33:34Z

# The Korean Question Answering Dataset v2
Adding the [KorQuAD](https://korquad.github.io/) v2 dataset as part of the sprint 🎉 
This dataset is very similar to SQuAD and is an extension of [squad_kor_v1](https://github.com/huggingface/datasets/pull/1178) which is why I added it as `squad_kor_v2`. 

- Crowd generated questions and answer (1-answer per question) for Wikipedia articles. Differently from V1 it includes the html structure and markup, which makes it a different enough dataset. (doesn't share ids between v1 and v2 either)

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)

Edit: 🤦 looks like squad_kor_v1 commit sneaked in here too

Comments:
looks like this PR also includes the changes for the V1
Could you only include the files of the V2 ?

hmm I have made the dummy data lighter retested on local and it passed not sure why it fails here?

merging since the CI is fixed on master
Title: Add  KorQuAD v1 Dataset
Author: cceyda (CONTRIBUTOR)
State: closed
Created: 2020-12-05T21:25:46Z

# The Korean Question Answering Dataset
Adding the [KorQuAD](https://korquad.github.io/KorQuad%201.0/) v1 dataset as part of the sprint 🎉 
This dataset is very similar to SQuAD which is why I added it as `squad_kor_v1`. There is also a v2 which I added [here](https://github.com/huggingface/datasets/pull/1180).

- Crowd generated questions and answer (1-answer per question) for Wikipedia articles.

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)

Comments:
Title: Add Korean NER dataset
Author: jaketae (CONTRIBUTOR)
State: closed
Created: 2020-12-05T20:56:00Z

This PR adds the [Korean named entity recognition dataset](https://github.com/kmounlp/NER). This dataset has been used in many downstream tasks, such as training [KoBERT](https://github.com/SKTBrain/KoBERT) for NER, as seen in this [KoBERT-CRF implementation](https://github.com/eagle705/pytorch-bert-crf-ner).

Comments:
Closed via #1219