rIsHu009/github-issues-updated huggingface dataset:
Title: Add 1.5 billion words Arabic corpus 
Author: zaidalyafeai (CONTRIBUTOR)
State: closed
Created: 2020-12-10T00:32:18Z

Needs https://github.com/huggingface/datasets/pull/1429 to work.

Comments:
Can't pass dummy data tests. For the instructions, it asks me to generate the following file `dummy_data/Youm7_XML_utf_8.rar/Youm7_utf_8.xml` which is strange, any ideas @lhoestq ?

cc: I tested the data locally and it works, maybe the dummy tests doesn't support `rar` ? 

In the dummy_data.zip files you must include the rar file as if is was already extracted.
In particular here `Youm7_XML_utf_8.rar` is a directory (not an archive).

Also I'm getting `BadRarFile: Failed the read enough data: req=16384 got=51` while trying to download and extract the `Alittihad_XML_utf_8.rar` file. Do you have this issue as well ?

I have rarfile 4.0

Sorry it was my mistake, I missed up the directories, it works now. Not sure why you got that error. I have the same version of `rarfile`. Between, there were some suggestions to change the dataset from `1bn_words_arabic` to `arabic_billion_words` like https://github.com/huggingface/datasets/tree/master/datasets/spanish_billion_words. 


I'm ok with renaming the dataset `arabic_billion_words` if you want.
Note that you will need to rename class name `ArabicBillionWords` instead of `BillionWords`
(though `BillionWords` was not matching `1bn_words_arabic` anyway)

You will need to regenerate the dataset_infos.json file after this change.
OR alternatively just replace all mentions of `billion_words` with `arabic_billion_words` in dataset_infos.json <- this trick should save you some time :)

Hmmm I'm still not able to run it on my side because of the rar error (I'm running macos)
I just tried with rarfile 3.1 and it didn't work either.
I would like to be able to run it end-to-end on my side before merging if you don't mind. Let me investigate this issue a little bit

No worries, I will investigate it as well. 

I created a minimal example in [colab ](https://colab.research.google.com/drive/11ijesuGbrQylANka0VdsZ5vXwIuxkheY?usp=sharing).

Nice thanks, maybe it's just an issue on my side then

Ok I managed to solve the BadRarFile issue on my side :) 
To fix it I had to install the `unrar` tool for macos (though it seems it's not available with `brew install` anymore, I had to install it from elsewhere).
Title: init commit for MultiReQA for third PR with all issues fixed
Author: Karthik-Bhaskar (CONTRIBUTOR)
State: closed
Created: 2020-12-09T22:57:41Z

3rd PR w.r.t. PR #1349 with all the issues fixed. As #1349 had uploaded other files along with the multi_re_qa dataset

Comments:
good dataset card as well :) 

@lhoestq Thank you :)
Title: Add german common crawl dataset
Author: Phil1108 (NONE)
State: closed
Created: 2020-12-09T22:54:12Z
Labels: dataset contribution

Adding a subpart of the Common Crawl which was extracted with this repo https://github.com/facebookresearch/cc_net and additionally filtered for duplicates

Comments:
Hi @Phil1108 !
Have you had a chance to take a look at my suggestions ?
Feel free to ping me if you have questions or if you're ready for a review

Thanks again for adding this dataset, this one is very useful !

> 
> 
> Hi @Phil1108 !
> Have you had a chance to take a look at my suggestions ?
> Feel free to ping me if you have questions or if you're ready for a review
> 
> Thanks again for adding this dataset, this one is very useful !

Hello @lhoestq , 

okay so should we go on with only the 2 parts or wait until the hosting of all 200 GB is cleared at HF? 
If you want to merge it with the 2 parts only at first und upgrade later to all, sure I'll include the minor changes in the next few days and update it here

I think we can wait a bit to find a solution for the hosting and update the PR to include all the files when it's done.
In the meantime you can update the PR to take into account the suggestions regarding the dataset card, the parallel downloads and the use of eval.

A safer alternative to eval is `ast.literal_eval`, it should do the job

Thanks for your contribution, @Phil1108. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Title: Add Turkish News Category Dataset (270K)
Author: basakbuluz (CONTRIBUTOR)
State: closed
Created: 2020-12-09T21:08:33Z

This PR adds the Turkish News Categories Dataset (270K) dataset which is a text classification dataset by me and @yavuzKomecoglu.  Turkish news dataset consisting of **273601 news** in **17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.

Comments:
@lhoestq, can you please review this PR?


@SBrandeis,
Sorry. All of the latest version came to my branch. You can find final version. 
Resubmitted as a clean final version of #1466
I have completed all the review comments.

Closing this as PR is now https://github.com/huggingface/datasets/pull/1466
Title: Add Shrinked Turkish NER from Kaggle.
Author: bhctsntrk (CONTRIBUTOR)
State: closed
Created: 2020-12-09T20:38:35Z

Add Shrinked Turkish NER from [Kaggle](https://www.kaggle.com/behcetsenturk/shrinked-twnertc-turkish-ner-data-by-kuzgunlar).

Comments:
Title: Add Hate Speech and Offensive Language Detection dataset
Author: hugoabonizio (CONTRIBUTOR)
State: closed
Created: 2020-12-09T20:22:12Z

Add [Hate Speech and Offensive Language Detection dataset](https://github.com/t-davidson/hate-speech-and-offensive-language) from [this paper](https://arxiv.org/abs/1703.04009).

Comments:
@lhoestq done! The failing testes don't seem to be related, it seems to be a connection issue, if I understand it correctly.

@lhoestq done!

merging since the CI is fixed on master
Title: Add OffComBR
Author: hugoabonizio (CONTRIBUTOR)
State: closed
Created: 2020-12-09T19:38:08Z

Add [OffComBR](https://github.com/rogersdepelle/OffComBR) from [Offensive Comments in the Brazilian Web: a dataset and baseline results](https://sol.sbc.org.br/index.php/brasnam/article/view/3260/3222) paper.

But I'm having a hard time generating dummy data since the original dataset extion is `.arff` and the [_create_dummy_data function](https://github.com/huggingface/datasets/blob/a4aeaf911240057286a01bff1b1d75a89aedd57b/src/datasets/commands/dummy_data.py#L185) doesn't allow it.

Comments:
Hello @hugoabonizio, thanks for the contribution.
Regarding the fake data, you can generate it manually.
Running the `python datasets-cli dummy_data datasets/offcombr` should give you instructions on how to manually create the dummy data.
For reference, here is a spec for `.arff` files : https://www.cs.waikato.ac.nz/ml/weka/arff.html

@lhoestq again the failing tests doesn't seem to be related

merging since the CI is fixed on master
Title: Add Tweet Eval Dataset
Author: abhishekkrthakur (CONTRIBUTOR)
State: closed
Created: 2020-12-09T18:48:57Z


Comments:
Hi @lhoestq,

Seeing that it has been almost two months to this draft, I'm willing to take this forward if you and @abhishekkrthakur don't mind. :)

Hi @gchhablani !
Sure if @abhishekkrthakur doesn't mind
Thanks for your help :)

Please feel free :) 

Hi @lhoestq, @abhishekkrthakur 

I believe this can be closed. Merged in #1829.
Title: Add Portuguese Hate Speech dataset
Author: hugoabonizio (CONTRIBUTOR)
State: closed
Created: 2020-12-09T18:48:16Z

Binary Portuguese Hate Speech dataset from [this paper](https://www.aclweb.org/anthology/W19-3510/).

Comments:
@lhoestq done! (The failing tests don't seem to be related)

merging since the CI is fixed on master
Title: Adding TaPaCo Dataset with README.md
Author: pacman100 (CONTRIBUTOR)
State: closed
Created: 2020-12-09T18:42:58Z


Comments:
We want to keep the repo as light as possible so that it doesn't take ages to clone, that's why we ask for small dummy data files (especially when there are many of them). Let me know if you have questions or if we can help you on this

Hello @lhoestq , made the changes as you suggested and pushed, please review. By default, the dummy data was generated the way it was by the dummy data auto generate command. Thank you.
Title: adding covid-tweets-japanese (again)
Author: forest1988 (CONTRIBUTOR)
State: closed
Created: 2020-12-09T17:46:46Z

I had mistaken use git rebase, I was so hurried to fix it. However, I didn't fully consider the use of git reset , so I unintendedly stopped PR (#1367) altogether. Sorry about that.
I'll make a new PR.

Comments:
README.md is not created yet. I'll add it soon.

Thank you for your detailed code review! It's so helpful.
I'll reflect them to the code in 24 hours.

You may have told me in Slack (I cannot find the conversation log though I've looked through threads), but I'm sorry it seems I'm still misunderstanding how to get YAML from the tagger.
I'm now asking on Slack if I am looking at the tagger the wrong way.

One more thing I'd like to ask.
Should I make changes by myself, or can I use the "Commit suggestion" feature?
I'm new to this feature and I don't know how the rules work in this repository, so I'd like to ask just in case.

Thank you very much for merging!
Title: Add reasoning_bg
Author: saradhix (CONTRIBUTOR)
State: closed
Created: 2020-12-09T17:30:49Z

Adding reading comprehension dataset for Bulgarian language

Comments:
Hi @saradhix have you had the chance to reduce the size of the dummy data ?

Feel free to ping me when it's done so we can merge :) 

@lhoestq I have reduced the size of the dummy data manually and pushed the changes.

The CI errors are not related to your dataset.
They're fixed on master, you can ignore them

merging since the CI is fixed on master
Title: Add HoVer Dataset
Author: abhishekkrthakur (CONTRIBUTOR)
State: closed
Created: 2020-12-09T16:55:39Z

HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification
https://arxiv.org/abs/2011.03088

Comments:
@lhoestq all comments addressed :) 

merging since the CI is fixed on master
Title: initial commit for MultiReQA for second PR
Author: Karthik-Bhaskar (CONTRIBUTOR)
State: closed
Created: 2020-12-09T16:00:35Z

Since last PR #1349 had some issues passing the tests. So, a new PR is generated.

Comments:
Subsequent [PR #1426 ](https://github.com/huggingface/datasets/pull/1426) since this PR has uploaded other files along with the MultiReQA dataset.

closing this one since a new PR has been created
Title: Add script_version suggestion when dataset/metric not found
Author: joeddav (CONTRIBUTOR)
State: closed
Created: 2020-12-09T15:37:38Z

Adds a helpful prompt to the error message when a dataset/metric is not found, suggesting the user might need to pass `script_version="master"` if the dataset was added recently. The whole error looks like:

> Couldn't find file locally at blah/blah.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1/metrics/blah/blah.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/met
rics/blah/blah.py.
If the dataset was added recently, you may need to to pass script_version="master" to find the loading script on the master branch.

Comments:
Title: add amazon polarity dataset
Author: hfawaz (CONTRIBUTOR)
State: closed
Created: 2020-12-09T14:58:21Z

This corresponds to the amazon (binary dataset) requested in https://github.com/huggingface/datasets/issues/353

Comments:
`amazon_polarity` is probably a subset of `amazon_us_reviews` but I am not entirely sure about that.
I guess `amazon_polarity` will help in reproducing results of papers using this dataset since even if it is a subset from `amazon_us_reviews`, it is not trivial how to extract `amazon_polarity` from `amazon_us_reviews`, especially since `amazon_us_reviews` was released after `amazon_polarity`. 

do you know what the problem would be ? should I pull the master before ? @lhoestq 

The error just appeared on master. I will try to fix it today.
You can ignore them since it's not related to the dataset you added

merging since the CI is fixed on master

Great thanks for the help.
Title: Add LIAR dataset
Author: hugoabonizio (CONTRIBUTOR)
State: closed
Created: 2020-12-09T14:16:55Z

Add LIAR dataset from [“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection](https://www.aclweb.org/anthology/P17-2067/).

Comments:
@lhoestq done! The failing testes don't seem to be related, it seems to be a connection issue, if I understand it correctly.

merging since the CI is fixed on master
Title: add best2009
Author: cstorm125 (CONTRIBUTOR)
State: closed
Created: 2020-12-09T13:56:09Z

`best2009` is a Thai word-tokenization dataset from encyclopedia, novels, news and articles by [NECTEC](https://www.nectec.or.th/) (148,995/2,252 lines of train/test). It was created for [BEST 2010: Word Tokenization Competition](https://thailang.nectec.or.th/archive/indexa290.html?q=node/10). The test set answers are not provided publicly.

Comments:
Title: added conv ai 2
Author: rkc007 (CONTRIBUTOR)
State: closed
Created: 2020-12-09T13:30:12Z

Dataset : https://github.com/DeepPavlov/convai/tree/master/2018

Comments:
@lhoestq Thank you for the suggestions. I added the changes to the branch and seems after rebasing it to master, all the commits previous commits got added. Should I create a new PR or should I keep this one only ? 

closing this one in favor of #1527
Title: Add twi text c3
Author: dadelani (CONTRIBUTOR)
State: closed
Created: 2020-12-09T13:16:38Z

Added Twi texts for training embeddings and language models based on the paper https://www.aclweb.org/anthology/2020.lrec-1.335/

Comments:
looks like this PR includes changes about other datasets

Can you only include the changes related to twi text c3 please ?

Hi @lhoestq , I have removed the unnecessary files. Can you please confirm?

You might need to either find a way to go back to the commit before it changes 389 files or create a new branch.

okay, I have created another branch, see the latest pull https://github.com/huggingface/datasets/pull/1518  @cstorm125 

Hii please follow me

Closing this one in favor of #1518
Title: Add yoruba text c3
Author: dadelani (CONTRIBUTOR)
State: closed
Created: 2020-12-09T13:16:03Z

Added Yoruba texts for training embeddings and language models based on the paper https://www.aclweb.org/anthology/2020.lrec-1.335/

Comments:
looks like this PR includes changes about other datasets


Thanks for the review. I'm a bit confused how to remove the files. Every time I add a new branch name using the following commands:

git fetch upstream
git rebase upstream/master
git checkout -b a-descriptive-name-for-my-changes

and push to the origin, this issue occurs

Can you try to create the branch from the master branch of your fork ?

first update your master branch:
```
git checkout master
git fetch upstream
git rebase upstream/master
git push
```

then create a new one:
```
git checkout -b my-new-branch
```

I think you were still having the files because you were creating the new branch from a branch in which you've committed the files, instead of creating the new branch from the master branch

Got it, will correct that. Thanks

@lhoestq , I have removed the unnecessary files. Looks like I still have one error. How do I resolve this?

> @lhoestq , I have removed the unnecessary files. Looks like I still have one error. How do I resolve this?

I think it's connection error on piqa dataset. Can you try triggering the test again? I usually resolve similar issues with:
```
git fetch upstream
git rebase upstream/master
git push -u -f origin your_branch_name
```

thank you  @cstorm125 

I have created another pull request for this https://github.com/huggingface/datasets/pull/1515 @cstorm125  @lhoestq 

Hii please follow me

merging since the CI is fixed on master

Great, thanks a lot
Title: Add FACTCK.BR dataset
Author: hugoabonizio (CONTRIBUTOR)
State: closed
Created: 2020-12-09T13:06:22Z

This PR adds [FACTCK.BR](https://github.com/jghm-f/FACTCK.BR) dataset from [FACTCK.BR: a new dataset to study fake news](https://dl.acm.org/doi/10.1145/3323503.3361698).

Comments:
@lhoestq done!

merging since the CI is fixed on master
Title: adding marathi-wiki dataset
Author: ekdnam (NONE)
State: closed
Created: 2020-12-09T13:01:20Z
Labels: dataset contribution

Adding marathi-wiki-articles dataset.

Comments:
Can you make it a draft PR until you've added the dataset please ? @ekdnam 

Done

Thanks for your contribution, @ekdnam. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Title: Re-adding narrativeqa dataset
Author: ghomasHudson (CONTRIBUTOR)
State: closed
Created: 2020-12-09T10:53:09Z

An update of #309.

Comments:
@lhoestq I think I've fixed the dummy data - it finally passes! I'll add the model card now.

@lhoestq - pretty happy with it now

> Awesome thank you !
> 
> Could you try to reduce the size of the dummy_data.zip file before we merge ? (it's 300KB right now)
> 
> To do so feel free to take a look inside it and remove all the unnecessary files and chunks of text, to only keep a few examples. The idea is to have a zip file that is only a few KB

Ah, it only contains 1 example for each split. I think the problem is that I include an entire story (like in the full dataset). We can probably get away with a summarised version.

> Nice thank you, can you make it even lighter if possible ? Something round 10KB would be awesone
> We try to keep the repo light so that it doesn't take ages to clone. So we have to make sure the dummy data are as small as possible for every single dataset.

Have trimmed a little more out of each example now.
Title: adding covid-tweets-japanese
Author: forest1988 (CONTRIBUTOR)
State: closed
Created: 2020-12-09T10:34:01Z

Adding COVID-19 Japanese Tweets Dataset as part of the sprint.

Testing with dummy data is not working (the file is said to not exist). Sorry for the incomplete PR.

Comments:
I think it's because the file you download uncompresses into a file and not a folder so `--autogenerate` couldn't create dummy data for you. See in your dummy_data.zip if there is a file there. If not, manually create your dummy data and compress them to dummy_data.zip.

@cstorm125 Thank you for the comment! 
As you point out, it seems my code has something wrong about downloading and uncompressing the file.
However, my manually created dummy data seems to contain a file of the required format.

On Colaboratory,
`!unzip /content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data.zip`
returns:

```
Archive:  /content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data.zip
   creating: content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data/
 extracting: content/datasets/datasets/covid_tweets_japanese/dummy/1.1.0/dummy_data/data.csv.bz2  
```

The original data is `data.csv.bz2`, and I had a very hard time dealing with uncompressing bzip2.
I think I could handle it, but there may be problems remain.
Title: Add Mkqa dataset
Author: cceyda (CONTRIBUTOR)
State: closed
Created: 2020-12-09T10:06:33Z

# MKQA: Multilingual Knowledge Questions & Answers Dataset
Adding the [MKQA](https://github.com/apple/ml-mkqa) dataset as part of the sprint 🎉

There is no official data splits so I added just a `train` split.
 
differently from the original:
- answer:type field is a ClassLabel (I thought it might be possible to train on this as a label for categorizing questions)
- answer:entity field has a default value of empty string '' (since this key is not available for all in original)
- answer:alias has default value of []

- [x] All tests passed
- [x] Added dummy data
- [x] Added data card (as much as I could)

Comments:
the `RemoteDatasetTest ` error pf the CI is fixed on master so it's fine

merging since the CI is fixed on master
Title: Narrative QA (Manual Download Stories) Dataset
Author: rsanjaykamath (CONTRIBUTOR)
State: closed
Created: 2020-12-09T09:33:59Z

Narrative QA with manual download for stories.

Comments:
Hi ! Maybe we can rename it `narrativeqa_manual` to make it explicit that this one requires manual download contrary to `narrativeqa` ?
It's important to have this one as well, in case the `narrativeqa` one suffers from download issues (checksums or dead links for example).

You can also copy the dataset card from `narrativeqa` and add the dummy data as well

Thanks @lhoestq will do all this and submit a request in the coming days. 😊 

Closing this as another pull request is already done.
Title: add wisesight1000
Author: cstorm125 (CONTRIBUTOR)
State: closed
Created: 2020-12-09T07:41:30Z

`wisesight1000` contains Thai social media texts randomly drawn from the full `wisesight-sentiment`, tokenized by human annotators. Out of the labels `neg` (negative), `neu` (neutral), `pos` (positive), `q` (question), 250 samples each. Some texts are removed because they look like spam.Because these samples are representative of real world content, we believe having these annotaed samples will allow the community to robustly evaluate tokenization algorithms.

Comments:
Title: Add spider dataset
Author: olinguyen (CONTRIBUTOR)
State: closed
Created: 2020-12-09T06:06:18Z

This PR adds the Spider dataset, a large-scale complex and cross-domain semantic parsing and text-to-SQL dataset annotated by 11 Yale students. The goal of the Spider challenge is to develop natural language interfaces to cross-domain databases.

Dataset website: https://yale-lily.github.io/spider
Paper link: https://www.aclweb.org/anthology/D18-1425/

Comments:
Title: Add TweetQA dataset
Author: anaerobeth (CONTRIBUTOR)
State: closed
Created: 2020-12-09T04:44:01Z

This PR adds the  TweetQA dataset, the first dataset for QA on social media data by leveraging news media and crowdsourcing.

Paper: https://arxiv.org/abs/1907.06292
Repository: https://tweetqa.github.io/

Comments:
Title: added craigslist_bargians
Author: ZacharySBrown (CONTRIBUTOR)
State: closed
Created: 2020-12-09T01:02:31Z

`craigslist_bargains` data set from [here](https://worksheets.codalab.org/worksheets/0x453913e76b65495d8b9730d41c7e0a0c/)

(Cleaned up version of #1278)

Comments:
Title: add LeNER-Br dataset
Author: jonatasgrosman (CONTRIBUTOR)
State: closed
Created: 2020-12-09T00:06:38Z

Adding the LeNER-Br dataset, a Portuguese language dataset for named entity recognition

Comments:
I don't know what happened, my first commit passed on all checks, but after just a README.md update one of the scripts failed, is it normal? 😕 

Looks like a flaky connection error, I've launched a re-run, it should be fine :)

The RemoteDatasetTest error in the CI is just a connection error, we can ignore it

merging since the CI is fixed on master
Title: initial commit for MultiReQA 
Author: Karthik-Bhaskar (CONTRIBUTOR)
State: closed
Created: 2020-12-08T23:44:34Z

Added MultiReQA, which is a dataset containing the sentence boundary annotation from eight publicly available QA datasets including SearchQA, TriviaQA, HotpotQA, NaturalQuestions, SQuAD, BioASQ, RelationExtraction, and TextbookQA.

Comments:
looks like this dataset includes changes about many other files than the ones for multi_re_qa

Can you create another branch and another PR please ?

> looks like this dataset includes changes about many other files than the ones for multi_re_qa
> 
> Can you create another branch and another PR please ?

Sure I will do that. Thank you.
Title: add Yoruba NER dataset
Author: dadelani (CONTRIBUTOR)
State: closed
Created: 2020-12-08T23:42:35Z

Added Yoruba GV dataset based on this paper

Comments:
Thank you. Okay, other pull requests only have one dataset

The `RemoteDatasetTest` error in the CI is just a connection error, we can ignore it

merging since the CI is fixed on master

Thank you very much
Title: Add spanish billion words corpus
Author: mariagrandury (CONTRIBUTOR)
State: closed
Created: 2020-12-08T22:51:38Z

Add an unannotated Spanish corpus of nearly 1.5 billion words, compiled from different resources from the web.

Comments:
Thank you for your feedback! I've reduced the dummy data size to 2KB.

I had to rebase to fix `RemoteDatasetTest` fails, sorry about the 80 commits. 
I could create a new clean PR if you prefer.

I have seen that in similar cases you have suggested to other contributors to create another branch and another PR, so I will do that.

Yes thank you !

The new PR is #1476 :)
Title: hate_speech_18 initial commit
Author: czabo (CONTRIBUTOR)
State: closed
Created: 2020-12-08T20:10:08Z


Comments:
> Nice thanks !
> 
> Can you rename the dataset folder and the dataset script name `hate_speech18` instead of `hate_speech_18` to follow the snake case convention we're using ?
> 
> Also it looks like the dummy_data.zip file is quite big (almost 4MB).
> Can you try to reduce its size ?
> 
> To do so feel free to take a look inside it and remove all the unnecessary files or chunks of texts. The idea is to only keep a few examples

Done, thanks! 

Re-opened in https://github.com/huggingface/datasets/pull/1486
Title: Add spanish billion words
Author: mariagrandury (CONTRIBUTOR)
State: closed
Created: 2020-12-08T19:18:02Z

Add an unannotated corpus of the Spanish language of nearly 1.5 billion words, compiled from different resources from the web.

The dataset needs 10 GB (download: 1.89 GiB, generated: 8.34 GiB, post-processed: Unknown size, total: 10.22 GiB), the test using dummy data pass but my laptop isn't able to run it on the real data (I left it running for over 8 hours and it didn't finish).

Comments:
The tests failed because of ```RemoteDatasetTest``` so I tried ```git rebase``` and messed everything up. I've made a new clean PR (#1347).
Title: Added Bianet dataset
Author: param087 (CONTRIBUTOR)
State: closed
Created: 2020-12-08T19:10:32Z

Hi :hugs:, This is a PR for [Bianet: A parallel news corpus in Turkish, Kurdish and English; Source](http://opus.nlpl.eu/Bianet.php) dataset

Comments:
merging since the Ci is fixed on master
Title: added un_ga dataset
Author: param087 (CONTRIBUTOR)
State: closed
Created: 2020-12-08T17:58:38Z

Hi :hugs:, This is a PR for [United nations general assembly resolutions: A six-language parallel corpus](http://opus.nlpl.eu/UN.php) dataset

Comments:
Looks like this PR includes changes about many other files than the ones for un_ga

Can you create another branch an another PR please ?

@lhoestq, Thank you for suggestions. I have made the changes and raised the new PR https://github.com/huggingface/datasets/pull/1569.
Title: Added the NewsPH Raw dataset and corresponding dataset card
Author: jcblaisecruz02 (NONE)
State: closed
Created: 2020-12-08T17:25:45Z

This PR adds the original NewsPH dataset which is used to autogenerate the NewsPH-NLI dataset. Reopened a new PR as the previous one had problems.

Paper: https://arxiv.org/abs/2010.11574
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

Comments:
Title: ❓ Sharing ElasticSearch indexed dataset 
Author: pietrolesci (NONE)
State: open
Created: 2020-12-08T16:25:58Z
Labels: dataset request

Hi there,

First of all, thank you very much for this amazing library. Datasets have become my preferred data structure for basically everything I am currently doing.

**Question:** I'm working with a dataset and I have an elasticsearch container running at localhost:9200. I added an elasticsearch index and I was wondering

- how can I know where it has been saved?

- how can I share the indexed dataset with others?

I tried to dig into the docs, but could not find anything about that.

Thank you very much for your help.

Best,
Pietro

Edit: apologies for the wrong label

Comments:
Hello @pietrolesci , I am not sure to understand what you are trying to do here.

If you're looking for ways to save a dataset on disk, you can you the `save_to_disk` method:
```python
>>> import datasets
>>> loaded_dataset = datasets.load("dataset_name")
>>> loaded_dataset.save_to_disk("/path/on/your/disk")
```

The saved dataset can later be retrieved using:
```python
>>> loaded_dataset = datasets.Dataset.load_from_disk("/path/on/your/disk")
```

Also, I'd recommend posting your question directly in the issue section of the [elasticsearch repo](https://github.com/elastic/elasticsearch)

Hi @SBrandeis,

Thanks a lot for picking up my request. 

Maybe I can clarify my use-case with a bit of context. Say I have the IMDb dataset. I create an ES index on it. Now I can save and reload the dataset from disk normally. Once I reload the dataset, it is easy to retrieve the ES index on my machine. I was wondering: is there a way I can share the (now) indexed version of the IMDb dataset with my colleagues without requiring them to re-index it?

Thanks a lot in advance for your consideration.

Best,

Pietro

Thanks for the clarification.

I am not familiar with ElasticSearch, but if I understand well you're trying to migrate your data along with the ES index.
My advice would be to check out ES documentation, for instance, this might help you: https://www.elastic.co/guide/en/cloud/current/ec-migrate-data.html

Let me know if it helps
Title: Add CC-News dataset of English language articles
Author: vblagoje (CONTRIBUTOR)
State: closed
Created: 2020-12-08T16:18:15Z

Adds [CC-News](https://commoncrawl.org/2016/10/news-dataset-available/) dataset. It contains 708241 English language news articles. Although each article has a language field these tags are not reliable. I've used Spacy language detection [pipeline](https://spacy.io/universe/project/spacy-langdetect) to confirm that the article language is indeed English. 

The prepared dataset is temporarily hosted on my private Google Storage [bucket](https://storage.googleapis.com/hf_datasets/cc_news.tar.gz). We can move it to HF storage and update this PR before merging.

Comments:
@vblagoje nice work, please add the README.md file and it would be ready

@lhoestq @tanmoyio @yjernite please have a look at the dataset card. Don't forget that the dataset is still hosted on my private gs bucket and should eventually be moved to the HF bucket

I will move the files soon and ping you when it's done and with the new URLs :) 

Hi !

I just moved the file to a HF bucket. It's available at https://storage.googleapis.com/huggingface-nlp/datasets/cc_news/cc_news.tar.gz

Sorry for the delay ^^'

@lhoestq no worries, updated PR with the new URL and rebased to master
Title: added dutch_social
Author: skyprince999 (CONTRIBUTOR)
State: closed
Created: 2020-12-08T16:07:54Z

The Dutch social media tweets dataset. Which has a total of more than 210k tweets in dutch language. These tweets have been machine annotated with sentiment scores (`label` feature) and `industry` and `hisco_codes`

It can be used for sentiment analysis, multi-label classification and entity tagging

Comments:
@lhoestq 
Updated the `dummy_data.zip `(<10kb)I had to reduce it to just a few samples. 
Train-Test-Dev  (20-5-5 samples) 

But the push also added changes from other PRs (probably because of a rebase!) So the files changed tab shows 466 files were changed!  


Thanks ! The dummy data are all good now :) 

Looks like this PR includes changes to many other files than the ones for dutch_social now.

Can you create another branch and another PR please ?

> 
> Can you create another branch and another PR please ?
@lhoestq 

I did a rebase. Now it doesn't include the other files. Does that help? 



Yes thanks !
Title: Added the WikiText-TL39 dataset and corresponding card
Author: jcblaisecruz02 (NONE)
State: closed
Created: 2020-12-08T16:00:26Z

This PR adds the WikiText-TL-39 Filipino Language Modeling dataset. Restarted a new pull request since there were problems with the earlier one.

Paper: https://arxiv.org/abs/1907.00409
Repo: https://github.com/jcblaisecruz02/Filipino-Text-Benchmarks

Comments:
Title: adding wili-2018 language identification dataset
Author: Shubhambindal2017 (CONTRIBUTOR)
State: closed
Created: 2020-12-08T16:00:09Z


Comments:
@lhoestq Not sure what happened, I just changed the py file but it is showing some TensorFlow error now.

You can ignore it.
It's caused by the Tensorflow update that happened 30min ago. They added breaking changes.
I'm working on a fix on the master branch right now


oh okay, btw I have made the required change for reading the CSV, I think it should be fine now, please take a look at it when you have some time.

merging since the CI is fixed on master
Title: ethos first commit
Author: iamollas (CONTRIBUTOR)
State: closed
Created: 2020-12-08T15:59:47Z

Ethos passed all the tests except from this one: 
RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>

with this error: 
E               OSError: Cannot find data file. 
E               Original error:
E               [Errno 2] No such file or directory:

Comments:
> Nice thanks !
> 
> I left a few comments
> 
> Also it looks like this PR includes changes about other files than the ones for ethos
> 
> Can you create another branch and another PR please ?

@lhoestq Should I close this PR? The new one is the: #1453

You can create another PR and close this one if you don't mind

> You can create another PR and close this one if you don't mind

Perfect! You should see the #1453 PR for the fixed version! Thanks
Title: add 10k German News Article Dataset
Author: stevhliu (MEMBER)
State: closed
Created: 2020-12-08T15:44:25Z


Comments:
You can just create another branch from master on your fork and create another PR:

first update your master branch
```
git checkout master
git fetch upstream
git rebase upstream/master
git push
```

then create a new branch
```
git checkout -b my-new-branch-name
```

Then you can add, commit and push the gnad10 files and open a new PR

closing in favor of #1572
Title: Allow GitHub releases as dataset source
Author: benjaminvdb (CONTRIBUTOR)
State: closed
Created: 2020-12-08T15:39:35Z

# Summary

Providing a GitHub release URL to `DownloadManager.download()` currently throws a `ConnectionError: Couldn't reach [DOWNLOAD_URL]`. This PR fixes this problem by adding an exception for GitHub releases in `datasets.utils.file_utils.get_from_cache()`.

# Reproduce

```
import datasets
url = 'http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz'
result = datasets.utils.file_utils.get_from_cache(url)

# Returns: ConnectionError: Couldn't reach http://github.com/benjaminvdb/DBRD/releases/download/v3.0/DBRD_v3.tgz
```

# Cause

GitHub releases returns a HTTP status 403 (FOUND), indicating that the request is being redirected (to AWS S3, in this case). `get_from_cache()` checks whether the status is 200 (OK) or if it is part of two exceptions (Google Drive or Firebase), otherwise the mentioned error is thrown.

# Solution

Just like the exceptions for Google Drive and Firebase, add a condition for GitHub releases URLs that return the HTTP status 403. If this is the case, continue normally.

Comments:
Title: Add snips built in intents 2016 12
Author: bduvenhage (CONTRIBUTOR)
State: closed
Created: 2020-12-08T15:30:19Z

This PR proposes to add the Snips.ai built in intents dataset. The first configuration added is for the intent labels only, but the dataset includes entity slots that may in future be added as alternate configurations.

Comments:
It is not clear how to automatically add the dummy data if the source data is a more complex json format. Should I manually take a fraction of the source data and include it as dummy data?


Added a fraction of the real data as dummy data.

merging since the CI is fixed on master
Title: Add HateSpeech Corpus for Polish
Author: kacperlukawski (CONTRIBUTOR)
State: closed
Created: 2020-12-08T15:23:53Z

This PR adds a HateSpeech Corpus for Polish, containing offensive language examples.

- **Homepage:** http://zil.ipipan.waw.pl/HateSpeech
- **Paper:** http://www.qualitativesociologyreview.org/PL/Volume38/PSJ_13_2_Troszynski_Wawer.pdf

Comments:
@lhoestq Do you think using the ClassLabel is correct if we don't know the meaning of them?

Once we find out the meanings we can still add them to the dataset card

Feel free to ping me when the PR is ready for the final review
Title: Add OffensEval-TR 2020 Dataset
Author: yavuzKomecoglu (CONTRIBUTOR)
State: closed
Created: 2020-12-08T14:49:51Z

This PR adds the OffensEval-TR 2020 dataset which is a Turkish offensive language corpus by me and @basakbuluz. The corpus consist of randomly sampled tweets and annotated in a similar way to [OffensEval](https://sites.google.com/site/offensevalsharedtask/) and [GermEval](https://projects.fzai.h-da.de/iggsa/).

- **Homepage:** [offensive-turkish](https://coltekin.github.io/offensive-turkish/)
- **Paper:** [A Corpus of Turkish Offensive Language on Social Media](https://coltekin.github.io/offensive-turkish/troff.pdf)
- **Point of Contact:** [Çağrı Çöltekin](ccoltekin@sfs.uni-tuebingen.de)

Comments:
@lhoestq, can you please review this PR? 

> Awesome thank you !

Thanks for the small fixes @lhoestq 

@coltekin, we have added the data set that you created an article that says "Turkish Attack Language Community in Social Media", HuggingFace dataset update sprint for you. We added Sprint quickly for a short time. I hope you welcome it too. The dataset is accessible at https://huggingface.co/datasets/offenseval2020_tr. 

Thank you for the heads up. I am not familiar with the terminology above (no idea what a sprint is), but I am happy that you found the data useful. Please feel free to distribute/use it as you see fit.

The OffensEval version you included in your data set has only binary labels. There is also a version [here](https://coltekin.github.io/offensive-turkish/troff-v1.0.tsv.gz) which also includes fine-grained labels similar to the OffensEval English data set - Just in case it would be of interest.

If you have questions about the data set, or need more information please let me know.
Title: Add SAMSum Corpus dataset
Author: changjonathanc (CONTRIBUTOR)
State: closed
Created: 2020-12-08T14:40:56Z

Did not spent much time writing README, might update later.

Copied description and some stuff from tensorflow_datasets
https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/summarization/samsum.py

Comments:
also to fix the check_code_quality CI you have to remove the imports of the unused `csv` and `os`

@lhoestq Thanks for the review! I have done what you asked, README is also updated. 🤗 
The CI fails because of the added dependency. I have never used circleCI before, so I am curious how will you solve that?

I just added `py7zr` to our test dependencies

merging since the CI is fixed on master

Thanks! 🤗
Title: Add Wiki Lingua Dataset
Author: katnoria (CONTRIBUTOR)
State: closed
Created: 2020-12-08T14:30:13Z

Hello,

This is my first PR. 

I have added Wiki Lingua Dataset along with dataset card to the best of my knowledge.
There was one hiccup though. I was unable to create dummy data because the data is in pkl format.
From the document, I see that:
```At the moment it supports data files in the following format: txt, csv, tsv, jsonl, json, xml```

Comments:
I am done adding the dataset. Requesting to review and advise.

looks like this PR has changes about many other files than the ones for WIki Lingua 

Can you create another branch and another PR please ?

Any reason to have english as the default config over the other languages ?

> looks like this PR has changes about many other files than the ones for WIki Lingua
> 
> Can you create another branch and another PR please ?

Ok, I will create another branch and submit a fresh PR.

> Any reason to have english as the default config over the other languages ?

The data for all other languages has a direct reference to English article. Thus, I kept English as default.

closing in favor of #1470
Title: add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC)
Author: aseifert (CONTRIBUTOR)
State: closed
Created: 2020-12-08T13:31:34Z

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.

Comments:
I created a clean PR where I also incorporated the suggested changes here: https://github.com/huggingface/datasets/pull/1449
Title: arxiv dataset added
Author: tanmoyio (CONTRIBUTOR)
State: closed
Created: 2020-12-08T12:50:51Z

**adding arXiv dataset**: arXiv dataset and metadata of 1.7M+ scholarly papers across STEM
dataset link: https://www.kaggle.com/Cornell-University/arxiv

Comments:
Readme added


@lhoestq is it looking alright ?
Title: can't load "german_legal_entity_recognition" dataset
Author: nataly-obr (NONE)
State: closed
Created: 2020-12-08T12:42:01Z

FileNotFoundError: Couldn't find file locally at german_legal_entity_recognition/german_legal_entity_recognition.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/german_legal_entity_recognition/german_legal_entity_recognition.py

Comments:
Please if you could tell me more about the error? 

1. Please check the directory you've been working on
2. Check for any typos

> Please if you could tell me more about the error?
> 
> 1. Please check the directory you've been working on
> 2. Check for any typos

Error happens during the execution of this line:
dataset = load_dataset("german_legal_entity_recognition")

Also, when I try to open mentioned links via Opera I have errors "404: Not Found" and "This XML file does not appear to have any style information associated with it. The document tree is shown below." respectively.

Hello @nataly-obr, the `german_legal_entity_recognition` dataset has not yet been released (it is part of the coming soon v2 release).

You can still access it now if you want, but you will need to install `datasets` via the master branch:
`pip install git+https://github.com/huggingface/datasets.git@master`

Please let me know if it solves the issue :)
Title: The Snips Built In Intents 2016 dataset.
Author: bduvenhage (CONTRIBUTOR)
State: closed
Created: 2020-12-08T11:40:10Z

This PR proposes to add the Snips.ai built in intents dataset. The first configuration added is for the intent labels only, but the dataset includes entity slots that may in future be added as alternate configurations.

Comments:
It is not clear how to automatically add the dummy data if the source data is a more complex json format. Should I manually take a fraction of the source data and include it as dummy data?

Will tag the dataset and update the dataset card.
Title: Add dutch book review dataset
Author: benjaminvdb (CONTRIBUTOR)
State: closed
Created: 2020-12-08T08:50:48Z

- Name: Dutch Book Review Dataset (DBRD)
- Description: The DBRD (pronounced dee-bird) dataset contains over 110k book reviews along with associated binary sentiment polarity labels and is intended as a benchmark for sentiment classification in Dutch.
- Paper: https://arxiv.org/abs/1910.00896
- Data: https://github.com/benjaminvdb/DBRD
- Motivation: A large (real-life) dataset of Dutch book reviews and sentiment polarity (positive/negative), based on the associated rating.

Checks
- [x] Create the dataset script /datasets/dbrd/dbrd.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _info(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.

Comments:
> Really cool thanks !
> 
> I left some (minor) comments

Thank you for your comments! 👍 I went ahead and improved the dataset card using your suggestions and some tweaks of my own. I hope you like it! 😄
Title: add thaiqa_squad
Author: cstorm125 (CONTRIBUTOR)
State: closed
Created: 2020-12-08T08:14:38Z

Example format is a little different from SQuAD since `thaiqa` always have one answer per question so I added a check to convert answers to lists if they are not already one to future-proof additional questions that might have multiple answers.

`thaiqa_squad` is an open-domain, extractive question answering dataset (4,000 questions in `train` and 74 questions in `dev`) in [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) format, originally created by [NECTEC](https://www.nectec.or.th/en/) from Wikipedia articles and adapted to [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) format by [PyThaiNLP](https://github.com/PyThaiNLP/).

Comments:
Title: added para_pat
Author: bhavitvyamalik (CONTRIBUTOR)
State: closed
Created: 2020-12-08T06:28:47Z

Dataset link : https://figshare.com/articles/ParaPat_The_Multi-Million_Sentences_Parallel_Corpus_of_Patents_Abstracts/12627632
Working on README.md currently

Comments:
Updated with Translation feature type. Working on dataset tags and README

merging since the CI is fixed on master
Title: Craigslist bargains
Author: ZacharySBrown (CONTRIBUTOR)
State: closed
Created: 2020-12-08T01:45:55Z

`craigslist_bargains` dataset from [here](https://worksheets.codalab.org/worksheets/0x453913e76b65495d8b9730d41c7e0a0c/)

Comments:
Seeing this in the CircleCI builds, this is what I was originally getting before I started messing around with the download URLS to try to fix this:

`FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpwvji917g/extracted/d6185140afb24ad8fee67392100a478269cba286b0d88915a137fdf88872de14/dummy_data/train__VARIABLE_MISUSE__SStuB.txt-00001-of-00300'`

Could this be because of the files in my `dummy_data.zip`? I had to manually create it, and it looked like the test was looking for the following files, so I created the `.zip` with this structure:

```
Archive:  dummy_data.zip
   creating: dummy_data/
  inflating: dummy_data/blobtest     
  inflating: dummy_data/parsed.jsontrain  
  inflating: dummy_data/parsed.jsonvalidation 
```

Going to close this out and link to a new (cleaner) PR