rIsHu009/github-issues-updated huggingface dataset:
Title: adding HindEncorp
Author: rahul-art (CONTRIBUTOR)
State: closed
Created: 2020-12-13T15:39:07Z

adding Hindi Wikipedia corpus

Comments:
@lhoestq I have created a new PR by reforking and creating a new branch 

@rahul-art unfortunately this didn't quite work, here's how you can try again:
- `git checkout master` to go back to the main branch
- `git pull upstream master` to make it up to date
- `git checkout -b add_hind_encorp` to create a new branch

Then add the dataset script, `README.md`, `dummy_data.zip`, and `dataset_infos.json` to the tracked files for the branch with `git add` (please add all of these files individually, NOT the whole directory as we don't want the other data files)
Then after you have passed the style checks and the local tests, do:
- `git commit . -m initial_commit`
- `git push --set-upstream origin add_hind_encorp`

Then you can go to this branch on the WebApp and open a new PR

@yjernite #1557 created new PR
Title: added TTC4900: A Benchmark Data for Turkish Text Categorization dataset
Author: yavuzKomecoglu (CONTRIBUTOR)
State: closed
Created: 2020-12-13T12:43:33Z

This PR adds the TTC4900 dataset which is a Turkish Text Categorization dataset by me and @basakbuluz. 

Homepage: [https://www.kaggle.com/savasy/ttc4900](https://www.kaggle.com/savasy/ttc4900)
Point of Contact: [Savaş Yıldırım](mailto:savasy@gmail.com) / @savasy

Comments:
@lhoestq, can you help with creating dummy_data?


Hi @yavuzKomecoglu did you manage to build the dummy data ?

> Hi @yavuzKomecoglu did you manage to build the dummy data ?

Hi, sorry for the return. I've created dummy_data.zip manually.

> Nice thank you !
> 
> Before we merge can you fill the two sections of the dataset card I suggested ?
> And also remove one remaining print statement

I updated your suggestions. Thank you very much for your support.

I think you accidentally pushed the readme of another dataset (name_to_nation).
I removed it so you have to `git pull`

Because of that I guess your changes about the ttc4900 was not included.
Feel free to ping me once they're added




> I think you accidentally pushed the readme of another dataset (name_to_nation).
> I removed it so you have to `git pull`
> 
> Because of that I guess your changes about the ttc4900 was not included.
> Feel free to ping me once they're added

I did `git pull` and updated readme **ttc4900**.

merging since the Ci is fixed on master
Title: Added Wiki Asp dataset
Author: katnoria (CONTRIBUTOR)
State: closed
Created: 2020-12-13T12:18:34Z

Hello,

I have added Wiki Asp dataset. Please review the PR.

Comments:
> Awesome thank you !
> 
> I just left one comment.
> 
> Also it looks like the dummy_data.zip files are quite big (around 500KB each)
> Can you try to reduce their sizes please ? Ideally they should be <20KB each
> 
> To do so feel free to take a look inside them and in the jsonl files only keep 1 or 2 samples instead of 5 and also remove big chunks of text to only keep a few passages.

Thanks, I have updated the dummy data to keep each domain <20/30KB.

> > Awesome thank you !
> > I just left one comment.
> > Also it looks like the dummy_data.zip files are quite big (around 500KB each)
> > Can you try to reduce their sizes please ? Ideally they should be <20KB each
> > To do so feel free to take a look inside them and in the jsonl files only keep 1 or 2 samples instead of 5 and also remove big chunks of text to only keep a few passages.
> 
> Thanks, I have updated the dummy data to keep each domain <20/30KB.

Looks like this branch has other commits. I will open a new PR with suggested changes.

opened a new PR #1612
Title: tweets_hate_speech_detection
Author: darshan-gandhi (CONTRIBUTOR)
State: closed
Created: 2020-12-13T07:37:53Z


Comments:
Hi @lhoestq  I have added this new dataset for tweet's hate speech detection. 

Please if u could review it. 

Thank you

Hi @darshan-gandhi have you add a chance to take a look at my suggestions ?

Feel free to ping me when you're ready for the final review

Closing in favor of #1607
Title: added ohsumed 
Author: skyprince999 (CONTRIBUTOR)
State: closed
Created: 2020-12-13T06:58:23Z

UPDATE2: PR passed all tests. Now waiting for review.

UPDATE: pushed  a new version. cross fingers that it should complete all the tests! :) 
               If it passes all tests then it's not a draft version. 

This is a draft version

Comments:
Title: Add Hippocorpus Dataset
Author: manandey (CONTRIBUTOR)
State: closed
Created: 2020-12-13T06:13:02Z


Comments:
> Before we merge can you try to reduce the size of the dummy_data.zip file ?
> 
> To do so feel free to only keep a few lines of the csv files ans also remove unnecessary chunks of texts (for example keep only the first sentences of a story).

Hi @lhoestq, I have reduced the size of the dummy_data.zip file by making the necessary changes you had suggested. 

merging since the CI is fixed on master
Title: Adding Igbo monolingual dataset
Author: purvimisal (CONTRIBUTOR)
State: closed
Created: 2020-12-13T05:16:37Z

This PR adds the Igbo Monolingual dataset.
Data: https://github.com/IgnatiusEzeani/IGBONLP/tree/master/ig_monoling
Paper: https://arxiv.org/abs/2004.00648

Comments:
@lhoestq Thank you for the review. I have made all the changes you mentioned. PTAL!
Title: add id_panl_bppt, a parallel corpus for en-id
Author: cahya-wirawan (CONTRIBUTOR)
State: closed
Created: 2020-12-13T03:11:27Z

Parallel Text Corpora for English - Indonesian

Comments:
Hi @lhoestq, thanks for the review. I will have a look and update it accordingly.

Strange error message :-)

```
>           tf_context = tf.python.context.context()  # eager mode context
E           AttributeError: module 'tensorflow' has no attribute 'python'
```
Title: add indonlu benchmark datasets
Author: yasirabd (CONTRIBUTOR)
State: closed
Created: 2020-12-13T01:56:09Z

The IndoNLU benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems for the Indonesian language. There are 12 datasets in IndoNLU.

This is a new clean PR from [#1322](https://github.com/huggingface/datasets/pull/1322)

Comments:
Title: Ro sent
Author: iliemihai (CONTRIBUTOR)
State: closed
Created: 2020-12-13T01:55:02Z

Movies reviews dataset for Romanian language.

Comments:
Hi @iliemihai, it looks like this PR holds changes from your previous PR #1493 .
Would you mind removing them from the branch please ?

@SBrandeis I am sorry. Yes I will remove them. Thank you :D 

Hi @lhoestq @SBrandeis @iliemihai

Is this still in progress or can I take over this one?

Thanks,
Gunjan

Hi,
While trying to add this dataset, I found some potential issues. 
The homepage mentioned is : https://github.com/katakonst/sentiment-analysis-tensorflow/tree/master/datasets/ro/, where the dataset is different from the URLs: https://raw.githubusercontent.com/dumitrescustefan/Romanian-Transformers/examples/examples/sentiment_analysis/ro/train.csv. It is unclear which dataset is "correct". I checked the total examples (train+test) in both places and they do not match.

We should use the data from dumitrescustefan and set the homepage to his repo IMO, since he's first author of the paper of the dataset.

Hi @lhoestq,

Cool, I'll get working on it.

Thanks

Hi @lhoestq, 

This PR can be closed.

Closing in favor of #2011 
Thanks again for adding it !
Title: Add : Conv AI 2 (Messed up original PR)
Author: rkc007 (CONTRIBUTOR)
State: closed
Created: 2020-12-13T00:21:14Z

@lhoestq Sorry I messed up the previous 2 PR's -> https://github.com/huggingface/datasets/pull/1462 -> https://github.com/huggingface/datasets/pull/1383. So created a new one. Also, everything is fixed in this PR. Can you please review it ?
Thanks in advance.

Comments:
Title: Add eHealth Knowledge Discovery dataset
Author: mariagrandury (CONTRIBUTOR)
State: closed
Created: 2020-12-12T20:44:18Z

This Spanish dataset can be used to mine knowledge from unstructured health texts. 

In particular, for:
- Entity recognition
- Relation extraction

Comments:
Thank you very much for your review @lewtun ! 

I've updated the script metadata, created the README and fixed the two details you commented.

Ready for another review! 🤗 

I've updated the task tag as we discussed and also added a couple of lines of code to make sure I include all the available examples.

Thank you again!
Title: Add semeval 2020 task 11
Author: ZacharySBrown (CONTRIBUTOR)
State: closed
Created: 2020-12-12T20:32:14Z

Adding in propaganda detection task (task 11) from Sem Eval 2020

Comments:
@SBrandeis : Thanks for the feedback! Just updated to use context manager for the `open`s and removed the placeholder text from the `README`!

Great, thanks @ZacharySBrown !
Failing tests seem to be unrelated to your changes, merging the current master branch into yours should fix them.
Title: Atomic
Author: huu4ontocord (CONTRIBUTOR)
State: closed
Created: 2020-12-12T20:18:08Z

This is the ATOMIC common sense dataset. More info can be found here:

* README.md still to be created.

Comments:
I had to create a new PR to fix git errors. See: https://github.com/huggingface/datasets/pull/1525

I'm closing this PR.
Title: ru_reviews dataset adding
Author: darshan-gandhi (CONTRIBUTOR)
State: closed
Created: 2020-12-12T18:13:06Z
Labels: dataset contribution

RuReviews: An Automatically Annotated Sentiment Analysis Dataset for Product Reviews in Russian

Comments:
Hi @lhoestq  

I have added the readme as well 

Please do have a look at it when suitable 

Chatted with @darshan-gandhi on Slack about parsing examples into a separate text and sentiment field

Thanks for your contribution, @darshan-gandhi. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Title: Initial commit for AQuaMuSe
Author: Karthik-Bhaskar (CONTRIBUTOR)
State: closed
Created: 2020-12-12T17:46:16Z

There is an issue in generation of dummy data. Tests on real data have passed locally.

Comments:
@yjernite Thank you for your help, generating the dummy data 🤗 Having that all the tests have passed 👍🏻

merging since the CI is fixed on master

Thank you :)
Title: how to get all the options of a property in datasets 
Author: rabeehk (CONTRIBUTOR)
State: closed
Created: 2020-12-12T16:24:08Z
Labels: question

Hi
could you tell me how I can get all unique options of a property of dataset?
for instance in case of boolq, if the user wants to know which unique labels it has, is there a way to access unique labels without getting all training data lables and then forming a set i mean? thanks

Comments:
In a dataset, labels correspond to the `ClassLabel` feature that has the `names` property that returns string represenation of the integer classes (or `num_classes` to get the number of different classes).

I think the `features` attribute of the dataset object is what you are looking for:
```
>>> dataset.features
{'sentence1': Value(dtype='string', id=None),
 'sentence2': Value(dtype='string', id=None),
 'label': ClassLabel(num_classes=2, names=['not_equivalent', 'equivalent'], names_file=None, id=None),
 'idx': Value(dtype='int32', id=None)
}
>>> dataset.features["label"].names
['not_equivalent', 'equivalent']
```

For reference: https://huggingface.co/docs/datasets/exploring.html
Title: Added dataset Makhzan
Author: arkhalid (CONTRIBUTOR)
State: closed
Created: 2020-12-12T15:34:07Z

Need help with the dummy data.

Comments:
The only CI error comes from 
```
FAILED tests/test_file_utils.py::TempSeedTest::test_tensorflow
```

which is not related to your PR and is fixed on master.

You can ignore it

@lhoestq I've made the changes. Please review and merge. 

I have a similar PR https://github.com/huggingface/datasets/pull/1562 for another dataset. I'll incorporate your comment about sorting and reducing dummy dataset size there.

The CI raises an error `FAILED tests/test_file_utils.py::TempSeedTest::test_tensorflow` but it's not related to this dataset.
This issue is fixed on master

You did all the work ;) thanks

merging since the CI is fixed on master
Title: Add SelQA Dataset
Author: bharatr21 (CONTRIBUTOR)
State: closed
Created: 2020-12-12T13:58:07Z

Add the SelQA Dataset, a new benchmark for selection-based question answering tasks
Repo: https://github.com/emorynlp/selqa/
Paper: https://arxiv.org/pdf/1606.08513.pdf

Comments:
Hii please follow me

The CI error `FAILED tests/test_file_utils.py::TempSeedTest::test_tensorflow` is not related with this dataset and is fixed on master. You can ignore it

merging since the Ci is fixed on master
Title: Add nq_open question answering dataset
Author: Nilanshrajput (CONTRIBUTOR)
State: closed
Created: 2020-12-12T13:46:48Z

Added nq_open Open-domain question answering dataset.

The NQ-Open task is currently being used to evaluate submissions to the EfficientQA competition, which is part of the NeurIPS 2020 competition track.

Comments:
@SBrandeis  thanks for the review, I applied your suggested changes, but CI is failing now not sure about the error.

Many thanks @Nilanshrajput  !
The failing tests on CI are not related to your changes, merging master on your branch should fix them :)
If you're interested in what causes the CI to fail, checkout [this commit](https://github.com/huggingface/datasets/commit/9a0f1e20ca1e783cb14c1ab1cc2f54b0b5b201e8)

@SBrandeis done!


Hello @Nilanshrajput, your PR includes changes from other branches too now (485 files changed)
Would you mind creating another branch from master with your changes and opening a new PR?

@SBrandeis sorry i messed up the git history, #1587 I opened this new pr! 

closing in favor of #1587
Title: Adding COVID QA dataset in Chinese and English from UC SanDiego
Author: vrindaprabhu (CONTRIBUTOR)
State: closed
Created: 2020-12-12T12:02:48Z


Comments:
Changed the pre-processing based on the comments raised in [PR-1482](https://github.com/huggingface/datasets/pull/1482).The below command is passing in my local environment:

`python datasets-cli test datasets/covid_qa_ucsd/ --save_infos --all_configs --data_dir ~/Downloads/Medical-Dialogue-Dataset/CovidDailogue/`
Title: Add Senti_Lex Dataset
Author: KMFODA (CONTRIBUTOR)
State: closed
Created: 2020-12-12T11:55:29Z

TODO:
Fix feature format issue
Create dataset_info.json file
Run pytests
Make Style

Comments:
Better will be if you close this PR and make a fresh PR

Feel free to ping me if you also have questions about the dummy data

also it looks like this PR includes changes about dummy_data.zip files in the ./datasets//un_pc folder. Can you remove them ?

Thanks for all the advice @lhoestq. I've implemented the changes you kindly highlighted and have made sure the scripts pass all the test. I've also marked this as ready for review as I believe it's in a good place to be merged now.

Great suggestion I fixed the dummy data and the file paths.
Title: Add Multi-Dimensional Gender Bias classification data
Author: yjernite (MEMBER)
State: closed
Created: 2020-12-12T00:17:37Z

https://parl.ai/projects/md_gender/

Mostly has the ABOUT dimension since the others are inferred from other datasets in most cases.

I tried to keep the dummy data small but one of the configs has 140 splits ( > 56KB data)

Comments:
Title: Added RONEC dataset.
Author: iliemihai (CONTRIBUTOR)
State: closed
Created: 2020-12-11T22:14:50Z


Comments:
Thanks for the PR @iliemihai . 

Few comments - 

Can you run - 
`python datasets-cli dummy_data ./datasets/ronec --auto_generate` to generate dummy data.

Also, before committing files run : 
`make style`
`flake8 datasets`
then you can add and commit files.

> Thanks for the PR @iliemihai .
> 
> Few comments -
> 
> Can you run -
> `python datasets-cli dummy_data ./datasets/ronec --auto_generate` to generate dummy data.
> 
> Also, before committing files run :
> `make style`
> `flake8 datasets`
> then you can add and commit files.

Sorry, forgot to generate dummy data. I will do it now :D

Awesome, good job @iliemihai !
I think the PR is ready to merge.
@lhoestq would you mind double-checking this ?

Had to regenerate the dummy data since I just found out they were empty files
Title: Fake news english 4
Author: MisbahKhan789 (CONTRIBUTOR)
State: closed
Created: 2020-12-11T21:10:35Z


Comments:
Thanks for the PR @MisbahKhan789 !

Few comments to help you along (I'm NOT a maintainer, just offering help to unblock the process) :-
 - Could you re-run `make style` and fix the errors related to code quality specific to your dataset in the `datasets/fake_news_english` folder?
(These seem to show errors that need manual fixes on running `flake8 datasets/fake_news_english`)
- Please run the local tests and check if they pass
`RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_<your-dataset-name>`
 `RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>`
(If not, you may have to regenerate the dummy data and the `dataset_infos.json` files)



Hii please follow me

> Thanks for the PR @MisbahKhan789 !
> 
> Few comments to help you along (I'm NOT a maintainer, just offering help to unblock the process) :-
> 
> * Could you re-run `make style` and fix the errors related to code quality specific to your dataset in the `datasets/fake_news_english` folder?
>   (These seem to show errors that need manual fixes on running `flake8 datasets/fake_news_english`)
> * Please run the local tests and check if they pass
>   `RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_real_dataset_<your-dataset-name>`
>   `RUN_SLOW=1 pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_all_configs_<your-dataset-name>`
>   (If not, you may have to regenerate the dummy data and the `dataset_infos.json` files)

Hey Bharat, thanks for the reply. I actually submitted a new PR with the changes that are required :)
Title: Adding NELL
Author: huu4ontocord (CONTRIBUTOR)
State: closed
Created: 2020-12-11T20:25:25Z

NELL is a knowledge base and knowledge graph along with sentences used to create the KB. See http://rtw.ml.cmu.edu/rtw/ for more details.

Comments:
hi @lhoestq, I wanted to push another change to this branch b/c I found a bug in the parsing. I need to swap arg1 and arg2. I tried to git push -u origin nell but it didn't work. So I tried to do git push --force -u origin nell which seems to work, but nothing is happening to this branch. I think this is because it's closed. Do I need to open another PR?

The change should be below in _generate_examples:
                    best_arg1 = row[9]
                    best_arg2 = row[8]


Hi @ontocord !

Yup the easiest thing to do is to open a new PR with a title like "Fix NELL dataset argument order". If it's a simple fix we can look at it pretty fast :)
Title:  added conv_ai_3 dataset
Author: rkc007 (CONTRIBUTOR)
State: closed
Created: 2020-12-11T19:26:26Z

Dataset : https://github.com/aliannejadi/ClariQ/

Comments:
@lhoestq Thank you for suggesting changes. I fixed all the changes you suggested. Can you please review it again? 

@lhoestq Thank you for reviewing and suggesting changes. I made the requested changes. Can you please review it again?

Thanks @lhoestq for reviewing it again. I made the required changes. Can you please have a look ?

merging since the CI is fixed on master
Title: hate speech 18 dataset
Author: czabo (CONTRIBUTOR)
State: closed
Created: 2020-12-11T19:22:14Z

This is again a PR instead of #1339, because something went wrong there.

Comments:
The error `tests/test_file_utils.py::TempSeedTest::test_tensorflow` just appeared because of tensorflow's update.
Once it's fixed on master we'll be free to merge this one

It's fixed on master now :) 

merging this once
Title: Add peer-read dataset
Author: vinaykudari (CONTRIBUTOR)
State: closed
Created: 2020-12-11T18:43:44Z


Comments:
> Cool thank you !
> 
> I left a few comments

Thank you @lhoestq  addressed your comments. Haven't changed the code but I see that tests are failing now. Do I need to rebase or something? 

The CI error is not related to your dataset and is fixed on master.
You can ignore it
Title: Added Times of India News Headlines Dataset
Author: tanmoyio (CONTRIBUTOR)
State: closed
Created: 2020-12-11T18:12:38Z

Dataset name: Times of India News Headlines
link: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/DPQMQH

Comments:
@lhoestq @abhishekkrthakur what happened here ?


@lhoestq everything alright here ?

@tanmoyio please have patience. @lhoestq has to look at 150+ PRs and it may take time. The PR looks good to me but we wait for his confirmation :) 🤗
Title: Adding medical database chinese and english
Author: vrindaprabhu (CONTRIBUTOR)
State: closed
Created: 2020-12-11T17:50:39Z

Error in creating dummy dataset

Comments:
Let me know it that helps !
Also feel free to ping me if you have other questions or if I can help you.

Now I am getting an Assertion Error!
![image](https://user-images.githubusercontent.com/16264631/101943915-f5bf5600-3c11-11eb-84e5-045bbc472162.png)


All tests have passed. However, PyTest is still failing with the `AssertionError` as before. Also _datasets_info.json_ actually does not seem to provide much info. Please review and let me know what has to be improved.

Thanks!

[PR-1503](https://github.com/huggingface/datasets/pull/1503) on the COVID dialog dataset from the same University has similar features. I kept it separate because it is only on COVID qa. Also consists of only single files per language. Kindly let me know if it has to be added as two more configurations to this existing dataset itself. If it has to be added, then can I still freeze the file names like in PR-1503?

It's ok to have them separate
Title: Add narrativeQA
Author: ghomasHudson (CONTRIBUTOR)
State: closed
Created: 2020-12-11T12:58:31Z

Redo of #1368 #309 #499

In redoing the dummy data a few times, I ended up adding a load of files to git. Hopefully this should work.

Comments:
@lhoestq this is now only failing some random windows test (it appears to be somewhere in wnut_17)

This is a connection error, you can ignore it :) 
The level of activity on the lib is quite overwhelming, it stresses a bit the CI ^^
Title: Inconsistent argument names.
Author: Fraser-Greenlee (CONTRIBUTOR)
State: closed
Created: 2020-12-11T12:19:38Z

Just find it a wee bit odd that in the transformers library `predictions` are those made by the model:
https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_utils.py#L51-L61

While in many datasets metrics they are the ground truth labels:
https://github.com/huggingface/datasets/blob/c3f53792a744ede18d748a1133b6597fdd2d8d18/metrics/accuracy/accuracy.py#L31-L40

Do you think predictions & references should be swapped? I'd be willing to do some refactoring here if you agree.

Comments:
Also for the `Accuracy` metric the `accuracy_score` method should have its args in the opposite order so `accuracy_score(predictions, references,,,)`.

Thanks for pointing this out ! 🕵🏻 
Predictions and references should indeed be swapped in the docstring.
However, the call to `accuracy_score` should not be changed, it [signature](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html#sklearn.metrics.accuracy_score) being:
```
sklearn.metrics.accuracy_score(y_true, y_pred, *, normalize=True, sample_weight=None)
```

Feel free to open a PR if you want to fix this :)
Title: Create JSON dummy data without loading all dataset in memory
Author: albertvillanova (MEMBER)
State: open
Created: 2020-12-11T08:44:23Z

See #1442.

The statement `json.load()` loads **all the file content in memory**.

In order to avoid this, file content should be parsed **iteratively**, by using the library `ijson` e.g.

I have refactorized the code into a function `_create_json_dummy_data` and I have added some tests.

Comments:
Title: Adding the HAREM dataset
Author: jonatasgrosman (CONTRIBUTOR)
State: closed
Created: 2020-12-11T03:21:10Z

Adding the HAREM dataset, a Portuguese language dataset for NER tasks

Comments:
Thanks for the changes !

Sorry if I wasn't clear about the suggestion of adding the `raw` dataset as well.
By `raw` I meant the dataset with its original features, i.e. not tokenized to follow the conll format for NER.
The `raw` dataset has data fields `doc_text`, `doc_id` and `entities`.

Alright, @lhoestq, now I understand your suggestion, but the JSON files the script downloads aren't actually the "raw" HAREM dataset, the [real raw version](https://www.linguateca.pt/primeiroHAREM/harem/ColeccaoDouradaHAREM.zip) of it is on XML format that needs a lot of preprocessing. Those JSON are just pre-processed versions of the dataset.

I can make the config of the raw version of the pre-processed dataset, but I'll leave a comment on the dataset summary about those details.

Oh I see ! Sorry I thought there really were the raw (i.e. original/unprocessed) files

The original xml format doesn't seem very practical to use :/ 
We can ignore that and only keep the default and selective configs then.
Can you revert to the old configs ?

Thanks again for adding this dataset !

Hi @lhoestq, I've reverted the changes and put more details on README about the dataset.
Title: Add wiki lingua dataset
Author: katnoria (CONTRIBUTOR)
State: closed
Created: 2020-12-11T02:04:18Z

Hello @lhoestq ,

I am opening a fresh pull request as advised in my original PR https://github.com/huggingface/datasets/pull/1308

Thanks

Comments:
it’s failing because of `RemoteDatasetTest.test_load_dataset_orange_sum`
which i think is not the dataset you are doing a PR for. Try rebasing with:
```
git fetch upstream
git rebase upstream/master
git push -u -f origin your_branch
```

> it’s failing because of `RemoteDatasetTest.test_load_dataset_orange_sum`
> which i think is not the dataset you are doing a PR for. Try rebasing with:
> 
> ```
> git fetch upstream
> git rebase upstream/master
> git push -u -f origin your_branch
> ```

Thanks, my branch seems to be up to date. 
```Current branch add-wiki-lingua-dataset is up to date.```

Also where do the google drive urls come from ?

looks like this PR includes changes about many other files than the ones for wiki_lingua.

Can you create another branch and another PR ?
(or you can try to fix this branch with rebase and push force if you're familiar with it)

Thanks for fixing the dummy data and removing the glob call :) 

> looks like this PR includes changes about many other files than the ones for wiki_lingua.
> 
> Can you create another branch and another PR ?
> (or you can try to fix this branch with rebase and push force if you're familiar with it)

Easier to create a new branch and submit, I have submitted a new PR #1582 

Closing this one in favor of #1582
Title: ADD: Wino_bias dataset
Author: akshayb7 (CONTRIBUTOR)
State: closed
Created: 2020-12-10T20:59:45Z

Updated PR to counter messed up history of previous one (https://github.com/huggingface/datasets/pull/1235) due to rebase.
Removed manual downloading of dataset.

Comments:
merging since the CI is fixed on master
Title: add Indonesian newspapers (id_newspapers_2018)
Author: cahya-wirawan (CONTRIBUTOR)
State: closed
Created: 2020-12-10T20:54:12Z

The dataset contains around 500K articles (136M of words) from 7 Indonesian newspapers. The size of uncompressed 500K json files (newspapers-json.tgz) is around 2.2GB.

Comments:
Looks like there's a `Path` issue on windows. Could you try switching to
`glob.glob(os.path.join(article_dir, "*.json"))`

> Looks like there's a `Path` issue on windows. Could you try switching to
> `glob.glob(os.path.join(article_dir, "*.json"))`

Thanks, I replaced it with glob. Let's see if it solves the issue. Anyway, the main directory has a space, could it make the issue on windows? the test on linux don't have this problem.

It seems glob doesn't help also. Btw, one of the failing test tried to connect aws which failed:
```
C:\tools\miniconda3\lib\site-packages\urllib3\connection.py:160: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

address = ('s3.amazonaws.com', 443), timeout = 10, source_address = None
socket_options = [(6, 1, 1)]

```
Why did it try to connect to aws? I don't use it.

It seems that the circleci make a test for whole datasets repository, that means if only one of the dataset in the official repository has a download issue, this will also affect the test of a new dataset like mine, isn't it?
I changed the url to my newspaper dataset which contains only few simple json files and simple directory structure. But it still failed. And it failed not only on windows test. This is one of the error message:
```
-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
===== 4 failed, 2667 passed, 2052 skipped, 4 warnings in 432.05s (0:07:12) =====

Exited with code exit status 1
CircleCI received exit code 1
```
The test failed on twitter dataset even my dataset has nothing to do with twitter? 

merging since the CI is fixed on master

Hi, thanks for merging the dataset. I create a new PR (#1499) since I need to update the link to the dataset.
Title: adding snow_simplified_japanese_corpus
Author: forest1988 (CONTRIBUTOR)
State: closed
Created: 2020-12-10T19:45:03Z

Adding simplified Japanese corpus "SNOW T15" and "SNOW T23".
They contain original Japanese, simplified Japanese, and original English (the original text is gotten from en-ja translation corpus). Hence, it can be used not only for Japanese simplification but also for en-ja translation.

- http://www.jnlp.org/SNOW/T15
- http://www.jnlp.org/SNOW/T23

Comments:
merging since the CI is fixed on master

Thank you for the updates and merging!
Title: Add Turkish News Category Dataset (270K).Updates were made for review…
Author: basakbuluz (CONTRIBUTOR)
State: closed
Created: 2020-12-10T19:41:12Z

This PR adds the **Turkish News Categories Dataset (270K)** dataset which is a text classification dataset by me and @yavuzKomecoglu. Turkish news dataset consisting of **273601 news in 17 categories**, compiled from printed media and news websites between 2010 and 2017 by the [Interpress](https://www.interpress.com/) media monitoring company.

**Note**: Resubmitted as a clean version of the previous Pull Request(#1419).  @SBrandeis @lhoestq

Comments:
@SBrandeis, What exactly is it that makes the tests fail? Can you help me please?

These errors
```
=========================== short test summary info ============================
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
```
appeared on master 3 hours ago and are now fixed.
(it was due to today's update of `xlrd` that broke two datasets)

You can ignore them, they're not related to your dataset

> These errors
> 
> ```
> =========================== short test summary info ============================
> FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_ajgt_twitter_ar
> FAILED tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_chr_en
> FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_ajgt_twitter_ar
> FAILED tests/test_dataset_common.py::RemoteDatasetTest::test_load_dataset_chr_en
> ```
> 
> appeared on master 3 hours ago and are now fixed.
> (it was due to today's update of `xlrd` that broke two datasets)
> 
> You can ignore them, they're not related to your dataset

**If it is not a problem caused by us, we have already completed the review notes. You can then check it out and confirm?**

merging since the CI is fixed on master
Title: Reddit jokes
Author: tanmoyio (CONTRIBUTOR)
State: closed
Created: 2020-12-10T19:15:19Z

196k Reddit Jokes dataset
Dataset link- https://raw.githubusercontent.com/taivop/joke-dataset/master/reddit_jokes.json

Comments:
@lhoestq would you please rerun the test, 

I re-started the test.

@lhoestq let's hold off on merging for now though, having a conversation on Slack about some of the offensive content in the dataset and how/whether we want to present it.
Title: Adding enriched_web_nlg features + handling xml bugs
Author: TevenLeScao (CONTRIBUTOR)
State: closed
Created: 2020-12-10T18:48:19Z

This PR adds features of the enriched_web_nlg dataset that were not present yet (most notably sorted rdf triplet sets), and deals with some xml issues that led to returning no data in cases where surgery could be performed to salvage it.

Comments:
Title: Added conv ai 2 (Again)
Author: rkc007 (CONTRIBUTOR)
State: closed
Created: 2020-12-10T18:21:55Z

The original PR -> https://github.com/huggingface/datasets/pull/1383

Reason for creating again - 

The reason I had to create the PR again was due to the master rebasing issue. After rebasing the changes, all the previous commits got added to the branch.

Comments:
Looking perfect to me, need to rerun the tests


Thanks, @tanmoyio. 
How do I rerun the tests? Should I change something or push a new commit?

@rkc007 you don't need to rerun it, @lhoestq @yjernite will rerun it, as there are huge number of PRs in the queue it might take lil bit of time. 

ive just re-run the tests

Thank you @abhishekkrthakur. Can you please rerun it again? It seems something was broken in CI during the previous test.

@lhoestq Sorry for the mess. I don't know why this keeps on happening. I tried step by step process of updating the PR but seems something is wrong. This happened for 2nd time with the same PR. Apologies for that. 

New PR -> https://github.com/huggingface/datasets/pull/1527
Also, I fixed everything in the new PR.
Title: add Bengali Hate Speech dataset
Author: stevhliu (MEMBER)
State: closed
Created: 2020-12-10T15:40:55Z


Comments:
@lhoestq I think you might want to look at the dataset, and the first data instances mentioned in the README.md is very much offensive. Though this dataset is based on hate speech but I found the dataset heavily disturbing as Bengali is my native language.

Hi @tanmoyio indeed you're right.
We should *at least* add very explicit mentions in the dataset card that the content of this dataset contains very offensive language. We should also put it in perspective with the tasks it tries to solve, the annotation process and the limitations.

We have to make sure that nothing is unclear/misleading nor could lead to bad usage of the dataset.

What do you think @tanmoyio ?
Also feel free to suggest modifications in the dataset cards if you feel like some sections require corrections or more details

> Hi @tanmoyio indeed you're right.
> We should _at least_ add very explicit mentions in the dataset card that the content of this dataset contains very offensive language. We should also put it in perspective with the tasks it tries to solve, the annotation process and the limitations.
> 
> We have to make sure that nothing is unclear/misleading nor could lead to bad usage of the dataset.
> 
> What do you think @tanmoyio ?
> Also feel free to suggest modifications in the dataset cards if you feel like some sections require corrections or more details

yeah I agree with you. It would be good if "Personal and Sensitive Information" and "Considerations for Using the Data" is being explained properly in the README.md. @stevhliu 

please let me know if there is anything else you'd like to see!

This looks ok to merge for me. Let me know @stevhliu and @tanmoyio if you want to add something or if it looks good to you

looks good to me @lhoestq 👍 

merging since the CI is fixed on master
Title: Add id_nergrit_corpus
Author: cahya-wirawan (CONTRIBUTOR)
State: closed
Created: 2020-12-10T13:20:34Z

Nergrit Corpus is a dataset collection of Indonesian Named Entity Recognition, Statement Extraction, and Sentiment Analysis. 
Recently my PR for id_nergrit_ner has been accepted and merged to the main branch. The id_nergrit_ner has only one dataset (NER), and this new PR renamed the dataset from id_nergrit_ner to id_nergrit_corpus and added 2 other remaining datasets (Statement Extraction, and Sentiment Analysis.)

Comments:
merging since the CI is fixed on master
Title: Add HEAD-QA: A Healthcare Dataset for Complex Reasoning
Author: mariagrandury (CONTRIBUTOR)
State: closed
Created: 2020-12-10T12:36:56Z

HEAD-QA is a multi-choice HEAlthcare Dataset, the questions come from exams to access a specialized position in the
Spanish healthcare system.

Comments:
Thank you for your review @lhoestq, I've changed the types of `qid` and `ra` and now they are integers as `aid`.

Ready for another review!
Title: SNLI dataset contains labels with value -1
Author: aarnetalman (NONE)
State: closed
Created: 2020-12-10T10:16:55Z

```
import datasets
nli_data = datasets.load_dataset("snli")
train_data = nli_data['train']
train_labels = train_data['label']
label_set = set(train_labels)
print(label_set)
```

**Output:**
`{0, 1, 2, -1}`

Comments:
I believe the `-1` label is used for missing/NULL data as per HuggingFace Dataset conventions. If I recall correctly SNLI has some entries with no (gold) labels in the dataset.

Ah, you're right. The dataset has some pairs with missing labels. Thanks for reminding me.
Title: add W&I + LOCNESS dataset (BEA-2019 workshop shared task on GEC) [PROPER]
Author: aseifert (CONTRIBUTOR)
State: closed
Created: 2020-12-10T09:51:08Z

- **Name:** W&I + LOCNESS dataset (from the BEA-2019 workshop shared task on GEC)
- **Description:** https://www.cl.cam.ac.uk/research/nl/bea2019st/#data
- **Paper:** https://www.aclweb.org/anthology/W19-4406/
- **Motivation:** This is a recent dataset (actually two in one) for grammatical error correction and is used for benchmarking in this field of NLP.

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.

Comments:
linter your code with flake8 and also run the commands present in Makefile for proper formatting 


merging since the CI is fixed on master
Title: add thai_toxicity_tweet
Author: cstorm125 (CONTRIBUTOR)
State: closed
Created: 2020-12-10T09:48:02Z

Thai Toxicity Tweet Corpus contains 3,300 tweets (506 tweets with texts missing) annotated by humans with guidelines including a 44-word dictionary. The author obtained 2,027 and 1,273 toxic and non-toxic tweets, respectively; these were labeled by three annotators. The result of corpus analysis indicates that tweets that include toxic words are not always toxic. Further, it is more likely that a tweet is toxic, if it contains toxic words indicating their original meaning. Moreover, disagreements in annotation are primarily because of sarcasm, unclear existing target, and word sense ambiguity.

Notes from data cleaner: The data is included into [huggingface/datasets](https://www.github.com/huggingface/datasets) in Dec 2020. By this time, 506 of the tweets are not available publicly anymore. We denote these by `TWEET_NOT_FOUND` in `tweet_text`.
Processing can be found at [this PR](https://github.com/tmu-nlp/ThaiToxicityTweetCorpus/pull/1).

Comments:
Title: Update step-by-step guide for windows
Author: thomwolf (MEMBER)
State: closed
Created: 2020-12-10T09:30:59Z

Update step-by-step guide for windows to give an alternative to `make style`.

Comments:
Hi @thomwolf, for simplification purposes, I think you could remove the "`pip install ...`" steps from this commit, 'cause these deps (black, isort, flake8) are already installed on `pip install -e ".[dev]"` on the [Start by preparing your environment](https://github.com/huggingface/datasets/blob/704107f924e74445f6f0fbde69a218b72178b588/ADD_NEW_DATASET.md#start-by-preparing-your-environment)
Title: Added dataset clickbait_news_bg
Author: tsvm (CONTRIBUTOR)
State: closed
Created: 2020-12-10T09:17:28Z


Comments:
Looks like this PR includes changes about many other files than the ones for clickbait_news_bg

Can you create another branch and another PR please ?

I created a new branch with the dataset code and submitted a new PR for it: https://github.com/huggingface/datasets/pull/1568
Title: FileNotFound remotly, can't load a dataset
Author: sadakmed (NONE)
State: closed
Created: 2020-12-10T09:14:47Z

```py
!pip install datasets
import datasets as ds

corpus = ds.load_dataset('large_spanish_corpus')
```
gives the error

> FileNotFoundError: Couldn't find file locally at large_spanish_corpus/large_spanish_corpus.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.3/datasets/large_spanish_corpus/large_spanish_corpus.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/large_spanish_corpus/large_spanish_corpus.py

not just `large_spanish_corpus`,  `zest` too, but `squad` is available. 

this was using colab and localy

Comments:
This dataset will be available in version-2 of the library. If you want to use this dataset now, install datasets from `master` branch rather.

Command to install datasets from `master` branch:
`!pip install git+https://github.com/huggingface/datasets.git@master`

Closing this, thanks @VasudevGupta7
Title: Add OPUS Wikimedia Translations Dataset
Author: abhishekkrthakur (CONTRIBUTOR)
State: closed
Created: 2020-12-10T08:43:02Z
Labels: dataset contribution


Comments:
Thanks for your contribution, @abhishekkrthakur. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Title: Create XML dummy data without loading all dataset in memory
Author: albertvillanova (MEMBER)
State: closed
Created: 2020-12-10T08:32:07Z

While I was adding one XML dataset, I noticed that all the dataset was loaded in memory during the dummy data generation process (using nearly all my laptop RAM).

Looking at the code, I have found that the origin is the use of `ET.parse()`. This method loads **all the file content in memory**.

In order to fix this, I have refactorized the code and use `ET.iterparse()` instead, which **parses the file content incrementally**.

I have also implemented a test.

Comments:
Title: Adding english plaintext jokes dataset
Author: purvimisal (CONTRIBUTOR)
State: closed
Created: 2020-12-10T07:04:17Z

This PR adds a dataset of 200k English plaintext Jokes from three sources: Reddit, Stupidstuff, and Wocka.
Link: https://github.com/taivop/joke-dataset 

This is my second PR. 
First was: [#1269 ](https://github.com/huggingface/datasets/pull/1269)

Comments:
Hi @purvimisal, thanks for your contributions!

This jokes dataset has come up before, and after a conversation with the initial submitter, we decided not to add it then. Humor is important, but looking at the actual data points in this set raises several concerns :) 

The main issue is the Reddit part of the dataset which has most of the examples. A cursory look at the data shows a large number of highly offensive jokes that reproduce some pretty harmful biases (the second one from the top is a Holocaust joke). 

The other two sources have similar issues (especially the "Blond Jokes") to a slightly lesser extent.

While such datasets can be useful in the right context, there is a real concern that people using the library might miss some of this context (however much we outline it), and end up unwittingly training models that rely on some pretty racist and sexist associations.

We would recommend skipping this dataset altogether.

If you feel really strongly about having a joke dataset, then we would ask that you:
- remove the Reddit part of the dataset altogether
- write an in-depth description of the social biases present in the remaining data

Let us know which of the two you decide! And if you want recommendations on other datasets to add, hit us up on Slack 🤗 

Hi @yjernite, thanks so much. I should've totally thought about this earlier. The harmful biases make so much sense. I should've consulted before making a PR. 
I will be closing this one and skipping this dataset altogether. 
Thanks again
Title: A descriptive name for my changes
Author: rahul-art (CONTRIBUTOR)
State: closed
Created: 2020-12-10T06:47:24Z

hind encorp resubmited

Comments:
I have noticed that the master branch of your fork has diverged from the one of the repo. This is probably what causes the mess in the github diff "Files changed".

I would suggest to re-fork the `datasets` repo and recreate a new branch and a new PR. 

You're pretty close to having all things ready to merge !
Feel free to ping me when you have a new PR

Closing this one in favor of #1575
Title: Add Indosum dataset
Author: prasastoadi (NONE)
State: closed
Created: 2020-12-10T05:02:00Z
Labels: dataset contribution


Comments:
Hi @prasastoadi have you had a chance to take a look at my suggestions ?

Feel free to ping ;e if you have questions or when you're ready for a review

Thanks for your contribution, @prasastoadi. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Title: add_sofc_materials_articles
Author: ZacharySBrown (CONTRIBUTOR)
State: closed
Created: 2020-12-10T02:15:02Z

adding [SOFC-Exp Corpus](https://arxiv.org/abs/2006.03039)

Comments:
Hey @lhoestq , thanks for the feedback on this! I updated the `_generate_examples` with some comments on the process, and reduced the `dummy_data.zip` down quite a bit as well. 

For the dummy data, I reduced the text to only three sentences, and aligned the corresponding entity/token/sentence annotations to that (reduced accordingly). The frames file is a strange combined format for the annotations and I found if I reduced that that would break the parser no matter what I did, so I left that as is. The difference between a reduced frames and non-reduced frames file in the compressed dummy data was only about ~4kb, so hopefully leaving this as is will be ok!
Title: Ar cov19
Author: Fatima-Haouari (CONTRIBUTOR)
State: closed
Created: 2020-12-10T00:59:34Z

Adding ArCOV-19 dataset. ArCOV-19 is an Arabic COVID-19 Twitter dataset that covers the period from 27th of January till 30th of April 2020. ArCOV-19 is the first publicly-available Arabic Twitter dataset covering COVID-19 pandemic that includes over 1M tweets alongside the propagation networks of the most-popular subset of them (i.e., most-retweeted and-liked). The propagation networks include both retweets and conversational threads (i.e., threads of replies). ArCOV-19 is designed to enable research under several domains including natural language processing, information retrieval, and social computing, among others.

Comments:
merging since the CI is fixed on master