rIsHu009/github-issues-updated huggingface dataset:
Title: Empty output/answer in TriviaQA test set (both in 'kilt_tasks' and 'trivia_qa')
Author: PaulLerner (CONTRIBUTOR)
State: closed
Created: 2020-11-05T11:38:01Z

# The issue

It's all in the title, it appears to be fine on the train and validation sets.

Is there some kind of mapping to do like for the questions (see https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md) ? 

# How to reproduce
```py
from datasets import load_dataset
kilt_tasks = load_dataset("kilt_tasks")
trivia_qa = load_dataset('trivia_qa', 'unfiltered.nocontext')
# both in "kilt_tasks"
In [18]: any([output['answer'] for output in kilt_tasks['test_triviaqa']['output']])                                                                                                                        
Out[18]: False
# and "trivia_qa"
In [13]: all([answer['value'] == '<unk>' for answer in trivia_qa['test']['answer']])                                                                                                                        
Out[13]: True
# appears to be fine on the train and validation sets.
In [14]: all([answer['value'] == '<unk>' for answer in trivia_qa['train']['answer']])                                                                                                                       
Out[14]: False

In [15]: all([answer['value'] == '<unk>' for answer in trivia_qa['validation']['answer']])                                                                                                                  
Out[15]: False

In [16]: any([output['answer'] for output in kilt_tasks['train_triviaqa']['output']])                                                                                                                       
Out[16]: True

In [17]: any([output['answer'] for output in kilt_tasks['validation_triviaqa']['output']])                                                                                                                  
Out[17]: True

```

Comments:
cc @yjernite is this expected ?

Yes: TriviaQA has a private test set for the leaderboard [here](https://competitions.codalab.org/competitions/17208)

For the KILT training and validation portions, you need to link the examples from the TriviaQA dataset as detailed here:
https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md

Oh ok, I guess I read the paper too fast 😅, thank you for your answer!
Title: Add XGlue
Author: patrickvonplaten (CONTRIBUTOR)
State: closed
Created: 2020-11-04T17:29:54Z

Dataset is ready to merge. An important feature of this dataset is that for each config the train data is in English, while dev and test data are in multiple languages. Therefore, @lhoestq and I decided offline that we will give the dataset the following API, *e.g.* for 

```python
load_dataset("xglue", "ner")  # would give the splits 'train', 'validation.en', 'test.en', 'validation.es', 'test.es', ... 
```
=> therefore one can load a single language test via

```python
load_dataset("xglue", "ner", split="test.es")
```

Close #749.

Comments:
Really cool to add XGlue, this will be a nice addition !

Splits shouldn't depend on the language. There must be configurations for each language, as we're doing for xnli, xtreme, etc.
For example for XGlue we'll have these configurations: NER.de, NER.en etc.
Title: How to join two datasets?
Author: shangw-nvidia (NONE)
State: closed
Created: 2020-11-04T03:53:11Z

Hi,

I'm wondering if it's possible to join two (preprocessed) datasets with the same number of rows but different labels? 

I'm currently trying to create paired sentences for BERT from `wikipedia/'20200501.en`, and I couldn't figure out a way to create a paired sentence using `.map()` where the second sentence is **not** the next sentence (i.e., from a different article) of the first sentence.

Thanks!

Comments:
Hi this is also my question. thanks 

Hi ! Currently the only way to add new fields to a dataset is by using `.map` and picking items from the other dataset


Closing this one. Feel free to re-open if you have other questions about this issue.

Also linking another discussion about joining datasets: #853
Title: Token classification labels are strings and we don't have the list of labels
Author: sgugger (CONTRIBUTOR)
State: closed
Created: 2020-11-03T15:33:30Z
Labels: enhancement

Dataset discussion

Not sure if this is an issue we want to fix or not, putting it here so it's not forgotten. Right now, in token classification datasets, the labels for NER, POS and the likes are typed as `Sequence` of `strings`, which is wrong in my opinion. These should be `Sequence` of `ClassLabel` or some types that gives easy access to the underlying labels.

The main problem for preprocessing those datasets is that the list of possible labels is not stored inside the `Dataset` object which makes converting the labels to IDs quite difficult (you either have to know the list of labels in advance or run a full pass through the dataset to get the list of labels, the `unique` method being useless with the type `Sequence[str]`).

Comments:
Indeed. Pinging @stefan-it here if he want to give an expert opinion :)

Related is https://github.com/huggingface/datasets/pull/636

Should definitely be a ClassLabel 👍 

Already done.
Title: Descriptions of raw and processed versions of wikitext are inverted
Author: fraboniface (NONE)
State: closed
Created: 2020-11-03T10:24:51Z
Labels: dataset bug

Nothing of importance, but it looks like the descriptions of wikitext-n-v1 and wikitext-n-raw-v1 are inverted for both n=2 and n=103. I just verified by loading them and the `<unk>` tokens are present in the non-raw versions, which confirms that it's a mere inversion of the descriptions and not of the datasets themselves.

Also it would be nice if those descriptions appeared in the dataset explorer.

https://github.com/huggingface/datasets/blob/87bd0864845ea0a1dd7167918dc5f341bf807bd3/datasets/wikitext/wikitext.py#L52

Comments:
Yes indeed ! Thanks for reporting

Fixed by:
- #3241
Title: [Datasets] fix discofuse links
Author: patrickvonplaten (CONTRIBUTOR)
State: closed
Created: 2020-11-03T08:03:45Z

The discofuse links were changed: https://github.com/google-research-datasets/discofuse/commit/d27641016eb5b3eb2af03c7415cfbb2cbebe8558. 
The old links are broken

I changed the links and created the new dataset_infos.json.

Pinging @thomwolf @lhoestq for notification.

Comments:
Title: add amazon reviews
Author: joeddav (CONTRIBUTOR)
State: closed
Created: 2020-11-02T16:42:57Z

Adds the Amazon US Reviews dataset as requested in #353. Converted from [TensorFlow Datasets](https://www.tensorflow.org/datasets/catalog/amazon_us_reviews). cc @clmnt @sshleifer

Comments:
@patrickvonplaten Yeah this is adapted from tfds so a lot is just how they wrote the code. Addressed your comments and also simplified the weird `AmazonUSReviewsConfig` definition. Will merge once tests pass.

Thanks for checking this one :) 
Looks good to me 

Just one question : is there a particular reason to use `names=["Y", "N"]` in this order ? Usually the positive label is at index 1 and the negative one at index 0 for binary classification

> is there a particular reason to use `names=["Y", "N"]` in this order ? Usually the positive label is at index 1 and the negative one at index 0 for binary classification

Hmm that's a good point. I'll submit a quick fix.
Title: Error running pip install -e ".[dev]" on MacOS 10.13.6: faiss/python does not exist
Author: shawwn (NONE)
State: closed
Created: 2020-11-02T12:36:35Z

I was following along with https://huggingface.co/docs/datasets/share_dataset.html#adding-tests-and-metadata-to-the-dataset when I ran into this error.

```sh
git clone https://github.com/huggingface/datasets
cd datasets
virtualenv venv -p python3 --system-site-packages
source venv/bin/activate
pip install -e ".[dev]"
```


![image](https://user-images.githubusercontent.com/59632/97868518-72871800-1cd5-11eb-9cd2-37d4e9d20b39.png)

![image](https://user-images.githubusercontent.com/59632/97868592-977b8b00-1cd5-11eb-8f3c-0c409616149c.png)

Python 3.7.7

Comments:
I saw that `faiss-cpu` 1.6.4.post2 was released recently to fix the installation on macos. It should work now

Closing this one.
Feel free to re-open if you still have issues
Title: dataset(ncslgr): add initial loading script
Author: AmitMY (CONTRIBUTOR)
State: closed
Created: 2020-11-02T06:50:10Z

Its a small dataset, but its heavily annotated
https://www.bu.edu/asllrp/ncslgr.html

![image](https://user-images.githubusercontent.com/5757359/97838609-3c539380-1ce9-11eb-885b-a15d4c91ea49.png)

Comments:
Hi @AmitMY, sorry for leaving you hanging for a minute :) 

We've developed a new pipeline for adding datasets with a few extra steps, including adding a dataset card. You can find the full process [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md)

Would you be up for adding the tags and description in the README.md so we can merge this cool dataset?

@lhoestq should be ready for another review :) 

Awesome thank you !

It looks like the PR now includes changes from other PR that were previously merged. 
Feel free to create another branch and another PR so that we can have a clean diff.


Closing for #958
Title: failed to reuse cache
Author: WangHexie (NONE)
State: closed
Created: 2020-11-02T02:42:36Z

I packed the `load_dataset ` in a function of class, and cached data in a directory. But when I import the class and use the function, the data still have to be downloaded again. The information (Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.28 GiB, post-processed: Unknown size, total: 1.82 GiB) to ******) which logged to terminal shows the path is right to the cache directory, but the files still have to be downloaded again.

Comments:
Title: Adding nli_tr dataset
Author: e-budur (CONTRIBUTOR)
State: closed
Created: 2020-11-01T21:49:44Z

Hello,

In this pull request, we have implemented the necessary interface to add our recent dataset [NLI-TR](https://github.com/boun-tabi/NLI-TR).  The datasets will be presented on a full paper at EMNLP 2020 this month. [[arXiv link] ](https://arxiv.org/pdf/2004.14963.pdf)

The dataset is the neural machine translation of SNLI and MultiNLI datasets into Turkish.  So, we followed a similar format with the original datasets hosted in the HuggingFace datasets hub.  

Our dataset is designed to be accessed as follows by following the interface of the GLUE dataset that provides multiple datasets in a single interface over the HuggingFace datasets hub.  

```
from datasets import load_dataset
multinli_tr = load_dataset("nli_tr", "multinli_tr")
snli_tr = load_dataset("nli_tr", "snli_tr")
```

Thanks for your help in reviewing our pull request.

Comments:
Thank you @lhoestq for the time you take to review our pull request. We appreciate your help.

We've made the changes you described. Hope that it is ready for being merged. Please let me know if you have any additional requests for revisions.
Title: feat(dataset): multiprocessing _generate_examples
Author: AmitMY (CONTRIBUTOR)
State: closed
Created: 2020-10-31T16:52:16Z

forking this out of #741, this issue is only regarding multiprocessing

I'd love if there was a dataset configuration parameter `workers`, where when it is `1` it behaves as it does right now, and when its `>1` maybe `_generate_examples` can also get the `pool` and return an iterable using the pool.

In my use case, I would instead of:
```python
for datum in data:
     yield self.load_datum(datum)
```
do:
```python
return pool.map(self.load_datum, data)
```

As the dataset in question, as an example, has **only** 7000 rows, and takes 10 seconds to load each row on average, it takes almost 20 hours to load the entire dataset.
If this was a larger dataset (and many such datasets exist), it would take multiple days to complete.

Using multiprocessing, for example, 40 cores, could speed it up dramatically. For this dataset, hopefully to fully load in under an hour.

Comments:
I agree that would be cool :)
Right now the only distributed dataset builder is based on Apache Beam so you can use distributed processing frameworks like Dataflow, Spark, Flink etc. to build your dataset but it's not really well suited for single-worker parallel processing afaik

`_generate_examples` can now be run in parallel thanks to https://github.com/huggingface/datasets/pull/5107. You can find more info [here](https://huggingface.co/docs/datasets/dataset_script#sharding).
Title: feat(aslg_pc12): add dev and test data splits
Author: AmitMY (CONTRIBUTOR)
State: closed
Created: 2020-10-31T13:25:38Z

For reproducibility sake, it's best if there are defined dev and test splits.

The original paper author did not define splits for the entire dataset, not for the sample loaded via this library, so I decided to define:
- 5/7th for train
- 1/7th for dev
- 1/7th for test

Comments:
Hi ! I'm not sure we should make this split decision arbitrarily on our side. Users can split it afterwards to whatever they want using `dataset.train_test_split` for example.
Moreover it looks like there's already papers that use this dataset and propose their own splits ([here](http://xanthippi.ceid.upatras.gr/HealthSign/resources/Publications/sitis_paper_25_10.pdf) 80-20) 
What do you think ?

I was not aware of the `train_test_split` method, thanks!
Soe ven though it contributes to reproducibility, no need to do this split then.
Title: Add XNLI train set
Author: lhoestq (MEMBER)
State: closed
Created: 2020-10-30T13:21:53Z

I added the train set that was built using the translated MNLI.
Now you can load the dataset specifying one language:
```python
from datasets import load_dataset

xnli_en = load_dataset("xnli", "en")
print(xnli_en["train"][0])
# {'hypothesis': 'Product and geography are what make cream skimming work .', 'label': 1, 'premise': 'Conceptually cream skimming has two basic dimensions - product and geography .'}
print(xnli_en["test"][0])                                                                                                                   
# {'hypothesis': 'I havent spoken to him again.', 'label': 2, 'premise': "Well, I wasn't even thinking about that, but I was so frustrated, and, I ended up talking to him again."}
```

Cc @sgugger

Comments:
Hi! Thanks for adding the translated MNLI! Do you know what translations system / model you used when you created the datasets in the other languages?

According to the [paper](https://arxiv.org/pdf/1809.05053.pdf) it's the result of the work of professional translators ;)

Thanks for getting back to me.

The training data is not from translators. And it appears to be machine
translation for all languages. If we can know what system was used to
create the training data that would be great!

Yifan.


On Thu, Jun 9, 2022, 05:51 Quentin Lhoest ***@***.***> wrote:

> According to the paper <https://arxiv.org/pdf/1809.05053.pdf> it's the
> result of the work of professional translators ;)
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/781#issuecomment-1150914429>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKLKWDAPTMGB6BE5GJ4GULVOG5BLANCNFSM4TE67NMQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>


> The training data is not from translators.

What makes you think that ? The paper litteraly says

> we hire translators to translate the resulting sentences into 15 languages using the One Hour Translation platform.

However the annotators only did test and validation sets, as this was what
in the paper: “we construct an evaluation set for XLU by extending the
development and test sets of the Multi-Genre Natural Language Inference
Corpus (MultiNLI) to 15 languages".

On Thu, Jun 9, 2022 at 10:35 AM Quentin Lhoest ***@***.***>
wrote:

> The training data is not from translators.
>
> What makes you think that ? The paper litteraly says
>
> we hire translators to translate the resulting sentences into 15 languages
> using the One Hour Translation platform.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/781#issuecomment-1151202195>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAKLKWFZOQPLK4WSKFRLW6DVOH6LLANCNFSM4TE67NMQ>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>
Title: Feature/fidelity metrics from emnlp2020 evaluating and characterizing human rationales
Author: rathoreanirudh (NONE)
State: closed
Created: 2020-10-29T17:31:14Z
Labels: transfer-to-evaluate

This metric computes fidelity (Yu et al. 2019, DeYoung et al. 2019) and normalized fidelity (Carton et al. 2020).

Comments:
Hi ! This looks interesting, thanks for adding it :) 

For metrics there should only be two features fields: references and predictions.
Both of them can be defined as you want using nested structures if you need to.
Also I'm not sure what goes into references and what goes into predictions, could you give more details please ?
All the other computations parameters (model etc.) are fine though. Maybe explain a bit more what they're used for

> Hi ! This looks interesting, thanks for adding it :)
> 
> For metrics there should only be two features fields: references and predictions.
> Both of them can be defined as you want using nested structures if you need to.
> Also I'm not sure what goes into references and what goes into predictions, could you give more details please ?
> All the other computations parameters (model etc.) are fine though. Maybe explain a bit more what they're used for

The `predictions` are the predicted labels by a model for a particular input. Do you mean making `prob_y_hat` - the probability of the prediction being the predicted label, `prob_y_hat_alpha` - the probability of the prediction being the predicted label when the input is reduced subject to alpha and the `null_difference` is the difference between the probability of the prediction being the predicted label in full information minus the probability in zero information a part of references? Also, I have added the description for other parameters in kwargs_description. I can expand it if that makes sense?

I think every value that is generated by the model (so label, prob_y_hat, prob_y_hat_alpha etc.) should be in `predictions`.
Feel free to add more details in the kwargs_description, this is very useful for the end user.

Hi @lhoestq , I have updated the code according to your feedback. Please, let me know if it looks good and can be merged now.

Metrics are deprecated in `datasets` and `evaluate` should be used instead: https://github.com/huggingface/evaluate
Title: Unexpected behavior when loading cached csv file?
Author: dcfidalgo (CONTRIBUTOR)
State: closed
Created: 2020-10-29T16:06:10Z

I read a csv file from disk and forgot so specify the right delimiter. When i read the csv file again specifying the right delimiter it had no effect since it was using the cached dataset. I am not sure if this is unwanted behavior since i can always specify `download_mode="force_redownload"`. But i think it would be nice if the information what `delimiter` or what `column_names` were used would influence the identifier of the cached dataset.

Small snippet to reproduce the behavior:
```python
import datasets

with open("dummy_data.csv", "w") as file:
    file.write("test,this;text\n")

print(datasets.load_dataset("csv", data_files="dummy_data.csv", split="train").column_names)
# ["test", "this;text"]

print(datasets.load_dataset("csv", data_files="dummy_data.csv", split="train", delimiter=";").column_names)
# still ["test", "this;text"]
```

By the way, thanks a lot for this amazing library! :)

Comments:
Hi ! Thanks for reporting.
The same issue was reported in #730 (but with the encodings instead of the delimiter). It was fixed by #770 .
The fix will be available in the next release :)

Thanks for the prompt reply and terribly sorry for the spam! 
Looking forward to the new release!
Title: Allow custom split names in text dataset
Author: lhoestq (MEMBER)
State: closed
Created: 2020-10-29T14:04:06Z

The `text` dataset used to return only splits like train, test and validation. Other splits were ignored.
Now any split name is allowed.

I did the same for `json`, `pandas` and `csv`

Fix #735

Comments:
Awesome! This will make the behaviour much more intuitive for some non-standard code.

Thanks!
Title: Properly delete metrics when a process is killed
Author: lhoestq (MEMBER)
State: closed
Created: 2020-10-29T12:52:07Z

Tests are flaky when using metrics in distributed setup.
There is because of one test that make sure that using two possibly incompatible metric computation (same exp id) either works or raises the right error.
However if the error is raised, all the processes of the metric are killed, and the open files (arrow + lock files) are not closed correctly. This causes PermissionError on Windows when deleting the temporary directory.

To fix that I added a `finally` clause in the function passed to multiprocess to properly close the files when the process exits.

Comments:
Title: Adding CC-100: Monolingual Datasets from Web Crawl Data
Author: yjernite (MEMBER)
State: closed
Created: 2020-10-28T18:20:41Z
Labels: dataset request

## Adding a Dataset
- **Name:** CC-100: Monolingual Datasets from Web Crawl Data
- **Description:** https://twitter.com/alex_conneau/status/1321507120848625665
- **Paper:** https://arxiv.org/abs/1911.02116
- **Data:** http://data.statmt.org/cc-100/
- **Motivation:** A large scale multi-lingual language modeling dataset. Text is de-duplicated and filtered by how "Wikipedia-like" it is, hopefully helping avoid some of the worst parts of the common crawl.

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).

Comments:
cc @aconneau ;) 

These dataset files are no longer available.  https://data.statmt.org/cc-100/ files provided in this link are no longer available. Can anybody fix that issue?
@abhishekkrthakur @yjernite 

Hi ! Can you open an issue to report this problem ? This will help keep track of the fix :)

Ok
Title: Fix metric with cache dir
Author: lhoestq (MEMBER)
State: closed
Created: 2020-10-28T16:43:13Z

The cache_dir provided by the user was concatenated twice and therefore causing FileNotFound errors.
The tests didn't cover the case of providing `cache_dir=` for metrics because of a stupid issue (it was not using the right parameter).

I remove the double concatenation and I fixed the tests

Fix #728

Comments:
Title: Using `Dataset.map` with `n_proc>1` print multiple progress bars
Author: sgugger (CONTRIBUTOR)
State: closed
Created: 2020-10-28T14:13:27Z

When using `Dataset.map` with `n_proc > 1`, only one of the processes should print a progress bar (to make the output readable). Right now, `n_proc` progress bars are printed.

Comments:
Yes it allows to monitor the speed of each process. Currently each process takes care of one shard of the dataset.

At one point we can consider using streaming batches to a pool of processes instead of sharding the dataset in `num_proc` parts. At that point it will be easy to use only one progress bar

Hi @lhoestq, I am facing a similar issue, it is annoying when lots of progress bars are printed. Is there a way to turn off this behavior? 

You can disable the progress bars with
```python
import datasets

datasets.disable_progress_bar()
```
Title: Fix custom builder caching
Author: lhoestq (MEMBER)
State: closed
Created: 2020-10-28T13:32:24Z

The cache directory of a dataset didn't take into account additional parameters that the user could specify such as `features` or any parameter of the builder configuration kwargs (ex: `encoding` for the `text` dataset).

To fix that, the cache directory name now has a suffix that depends on all of them.

Fix #730
Fix #750

Comments:
Title: How to choose proper download_mode in function load_dataset?
Author: jzq2000 (NONE)
State: closed
Created: 2020-10-28T09:16:19Z

Hi, I am a beginner to datasets and I try to use datasets to load my csv file.
my csv file looks like this

``` 
text,label
"Effective but too-tepid biopic",3
"If you sometimes like to go to the movies to have fun , Wasabi is a good place to start .",4
"Emerges as something rare , an issue movie that 's so honest and keenly observed that it does n't feel like one .",5
```

First I try to use this command to load my csv file .  

``` python
dataset=load_dataset('csv', data_files=['sst_test.csv'])
```

It seems good, but when i try to overwrite the convert_options to convert  'label' columns from int64 to float32 like this.

``` python
import pyarrow as pa
from pyarrow import csv
read_options = csv.ReadOptions(block_size=1024*1024)
parse_options = csv.ParseOptions()
convert_options = csv.ConvertOptions(column_types={'text': pa.string(), 'label': pa.float32()})
dataset = load_dataset('csv', data_files=['sst_test.csv'], read_options=read_options,
                       parse_options=parse_options, convert_options=convert_options)
```

It keeps the same:

```shell
Dataset(features: {'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None)}, num_rows: 2210)
```

I think this issue is caused by the parameter "download_mode" Default to REUSE_DATASET_IF_EXISTS because after I delete the cache_dir, it seems right.

Is it a bug? How to choose proper download_mode to avoid this issue?

Comments:
`download_mode=datasets.GenerateMode.FORCE_REDOWNLOAD` should work.
This makes me think we we should rename this to DownloadMode.FORCE_REDOWNLOAD. Currently that's confusing

Can we just use `features=...` in `load_dataset` for this @lhoestq?

Indeed you should use `features` in this case. 
```python
features = Features({'text': Value('string'), 'label': Value('float32')})
dataset = load_dataset('csv', data_files=['sst_test.csv'], features=features)
```
Note that because of an issue with the caching when you change the features (see #750 ) you still need to specify the `FORCE_REDOWNLOAD ` flag. I'm working on a fix for this one

https://github.com/huggingface/datasets/issues/769#issuecomment-717837832
> This makes me think we we should rename this to DownloadMode.FORCE_REDOWNLOAD. Currently that's confusing

@lhoestq do you still think we should rename it?


It's no big deal, but since it can be confusing to users I think it's worth renaming it, and deprecate `GenerateMode` until `datasets` 2.0 at least. IMO it's confusing to have `download_mode=GenerateMode.something`
Title: Add a `lazy_map` method to `Dataset` and `DatasetDict`
Author: sgugger (CONTRIBUTOR)
State: open
Created: 2020-10-27T22:33:03Z
Labels: enhancement

The library is great, but it would be even more awesome with a `lazy_map` method implemented on `Dataset` and `DatasetDict`. This would apply a function on a give item but when the item is requested. Two use cases:

1. load image on the fly
2. apply a random function and get different outputs at each epoch (like data augmentation or randomly masking a part of a sentence for BERT-like objectives).

Comments:
This is cool! I think some aspects to think about and decide in terms of API are:
- do we allow several methods (chained i guess)
- how do we inspect the currently set method(s)
- how do we control/reset them
Title: Add option for named splits when using ds.train_test_split
Author: nateraw (CONTRIBUTOR)
State: open
Created: 2020-10-27T19:59:44Z
Labels: enhancement

### Feature Request 🚀 

Can we add a way to name your splits when using the `.train_test_split` function?

In almost every use case I've come across, I have a `train` and a `test` split in my `DatasetDict`, and I want to create a `validation` split. Therefore, its kinda useless to get a `test` split back from `train_test_split`, as it'll just overwrite my real `test` split that I intended to keep.

### Workaround

this is my hack for dealin with this, for now :slightly_smiling_face:

```python
from datasets import load_dataset
​
​
ds = load_dataset('imdb')
ds['train'], ds['validation'] = ds['train'].train_test_split(.1).values()
```

Comments:
Yes definitely we should give more flexibility to control the name of the splits outputted by `train_test_split`.

Related is the very interesting feedback from @bramvanroy on how we should improve this method: https://discuss.huggingface.co/t/how-to-split-main-dataset-into-train-dev-test-as-datasetdict/1090/5

And in particular that it should advantageously be able to split in 3 splits as well instead of just 2 like we copied from sklearn.
Title: [GEM] add DART data-to-text generation dataset
Author: yjernite (MEMBER)
State: closed
Created: 2020-10-27T17:34:04Z
Labels: dataset request

## Adding a Dataset
- **Name:** DART
- **Description:** DART consists of 82,191 examples across different domains with each input being a semantic RDF triple set derived from data records in tables and the tree ontology of the schema, annotated with sentence descriptions that cover all facts in the triple set.
- **Paper:** https://arxiv.org/abs/2007.02871v1
- **Data:** https://github.com/Yale-LILY/dart
- **Motivation:** the dataset will likely be included in the GEM benchmark

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).

Comments:
Is this a duplicate of #924 ?

Yup, closing! Haven't been keeping track of the solved issues during the sprint.
Title: [GEM] Add DART data-to-text generation dataset
Author: yjernite (MEMBER)
State: closed
Created: 2020-10-27T17:32:23Z
Labels: dataset request

## Adding a Dataset
- **Name:** DART
- **Description:** DART consists of 82,191 examples across different domains with each input being a semantic RDF triple set derived from data records in tables and the tree ontology of the schema, annotated with sentence descriptions that cover all facts in the triple set.
- **Paper:** https://arxiv.org/abs/2007.02871v1
- **Data:** https://github.com/Yale-LILY/dart
- **Motivation:** It will likely be included in the GEM generation evaluation benchmark

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).

Comments:
Title: Fixed errors in bertscore related to custom baseline
Author: juanjucm (CONTRIBUTOR)
State: closed
Created: 2020-10-27T16:08:35Z

[bertscore version 0.3.6 ](https://github.com/Tiiiger/bert_score) added support for custom baseline files. This update added extra argument `baseline_path` to BERTScorer class as well as some extra boolean parameters `use_custom_baseline` in functions like `get_hash(model, num_layers, idf, rescale_with_baseline, use_custom_baseline)`.

This PR fix those matching errors in bertscore metric implementation.

Comments:
Title: [GEM] Add Czech Restaurant data-to-text generation dataset
Author: yjernite (MEMBER)
State: closed
Created: 2020-10-27T16:00:47Z
Labels: dataset request

- Paper: https://www.aclweb.org/anthology/W19-8670.pdf
- Data: https://github.com/UFAL-DSG/cs_restaurant_dataset
- The dataset will likely be part of the GEM benchmark

Comments:
Title: Downloaded datasets are not usable offline
Author: ghazi-f (CONTRIBUTOR)
State: closed
Created: 2020-10-26T20:54:46Z

I've been trying to use the IMDB dataset offline, but after downloading it and turning off the internet it still raises  an error from the ```requests``` library trying to reach for the online dataset.
Is this the intended behavior ?
(Sorry, I wrote the the first version of this issue while still on nlp 0.3.0).

Comments:
Yes currently you need an internet connection because the lib tries to check for the etag of the dataset script online to see if you don't have it locally already.

If we add a way to store the etag/hash locally after the first download, it would allow users to first download the dataset with an internet connection, and still have it working without an internet connection.

I'll let you know when we add this feature.

Already fixed by:
- #1726
Title: Add meta-data to the HANS dataset
Author: yjernite (MEMBER)
State: closed
Created: 2020-10-26T14:56:53Z
Labels: good first issue

dataset bug

The current version of the [HANS dataset](https://github.com/huggingface/datasets/blob/master/datasets/hans/hans.py) is missing the additional information provided for each example, including the sentence parses, heuristic and subcase.

Comments:
Title: Process 0 very slow when using num_procs with map to tokenizer
Author: ksjae (NONE)
State: closed
Created: 2020-10-24T02:40:20Z

<img width="721" alt="image" src="https://user-images.githubusercontent.com/17930170/97066109-776d0d00-15ed-11eb-8bba-bb4d2e0fcc33.png">
The code I am using is
```

        dataset = load_dataset("text", data_files=[file_path], split='train')
        dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                                truncation=True, max_length=args.block_size), num_proc=8)
        dataset.set_format(type='torch', columns=['input_ids'])
        dataset.save_to_disk(file_path+'.arrow')
```

Comments:
Hi ! Thanks for reporting.
Is the distribution of text length of your data evenly distributed across your dataset ? I mean, could it be because the examples in the first part of your dataset are slower to process ?
Also could how many CPUs can you use for multiprocessing ?
```python
import multiprocessing
print(multiprocessing.cpu_count())
```
Which tokenizer are you using ?

Using pre trained HF tokenizer. The result is the same with tokenizer multiprocessing off and on.
I have (absolutely) no idea about the distribution, but since this issue occurs on all of my datasets(regardless of files), I don't think distribution is the problems.

I can use up to 16 cores.

Ok weird, I don't manage to reproduce this issue on my side.
Does it happen even with `num_proc=2` for example ?
Also could you provide more details about your OS and the versions of tokenizers/datasets/multiprocess that you're using ?

Yes, I can confirm it also happens with ```num_proc=2```.
```
tokenizers               0.9.2
datasets                 1.1.2
multiprocess             0.70.10
```
```
Linux nipa2020-0629 4.4.0-178-generic #208-Ubuntu SMP Sun Apr 5 23:45:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
```

I can't reproduce on my side unfortunately with the same versions.

Do you have issues when doing multiprocessing with python ?
```python
from tqdm.auto import tqdm
from multiprocess import Pool, RLock

def process_data(shard):
    # implement

num_proc = 8
shards = [] # implement, this must be a list of size num_proc

with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:
    results = [pool.apply_async(process_data, shard=shard) for shard in shards]
    transformed_shards = [r.get() for r in results]
```

Nah, I'll just wait a few hours. Thank you for helping, though.
Title: CUDA out of memory
Author: li1117heex (NONE)
State: closed
Created: 2020-10-23T13:57:00Z

In your dataset ,cuda run out of memory as long as the trainer begins:
however, without changing any other element/parameter,just switch dataset to `LineByLineTextDataset`,everything becames OK.

Comments:
Could you provide more details ? What's the code you ran ?

```python
tokenizer = FunnelTokenizer.from_pretrained('funnel-transformer/small')

def tokenize(batch):
    return tokenizer(batch['text'], padding='max_length', truncation=True,max_length=512)

dataset = load_dataset("bookcorpus",split='train[:1000]').shuffle()
dataset = dataset.map(tokenize, batched=True, batch_size=512)

# dataset = LineByLineTextDataset(
#     tokenizer=tokenizer,
#     file_path="./wiki1000.txt",
#     block_size=128
# )

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

config=FunnelConfig(
    return_dict=True
)

model= FunnelForMaskedLM(config=config)

training_args = TrainingArguments(
    output_dir="./checkpoints",
    overwrite_output_dir=True,
    do_train=True,
    num_train_epochs=1,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    save_steps=10000,
    logging_dir='./ptlogs'
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)
trainer.train()
```

`RuntimeError: CUDA out of memory. Tried to allocate 954.00 MiB (GPU 0; 15.90 GiB total capacity; 14.35 GiB already allocated; 753.75 MiB free; 14.39 GiB reserved in total by PyTorch)
Exception raised from malloc at /pytorch/c10/cuda/CUDACachingAllocator.cpp:272 (most recent call first):`

part of error output

from funnel model to bert model : error still happened

from your dataset to LineByLineTextDataset : error disapeared

notice i just loaded 1000 rows of data

the error happens when executing loss.backward()

Since you're using a data collator you don't need to tokenizer the dataset using `map`. Could you try not to use `map` and only the data collator instead ? The data collator is supposed to pad to the longest sequence in each batch afaik, instead of padding to 512.

Also cc @sgugger 

Closing this one.
Feel free to re-open if you have other questions about this issue
Title: Start community-provided dataset docs 
Author: sshleifer (CONTRIBUTOR)
State: closed
Created: 2020-10-23T13:17:41Z

Continuation of #736 with clean fork.

#### Old description
This is what I did to get the pseudo-labels updated. Not sure if it generalizes, but I figured I would write it down. It was pretty easy because all I had to do was make properly formatted directories and change URLs.

In slack @thomwolf called it a user-namespace dataset, but the docs call it community dataset.
I think the first naming is clearer, but I didn't address that here.

I didn't add metadata, will try that.

Comments:
Oh, really cool @sshleifer!
Title: Use full released xsum dataset
Author: jbragg (CONTRIBUTOR)
State: closed
Created: 2020-10-23T03:29:49Z

#672 Fix xsum to expand coverage and include IDs
Code based on parser from older version of `datasets/xsum/xsum.py`
@lhoestq

Comments:
@lhoestq I took a shot at addressing your comments but the build scripts seem to be complaining about not being able to open dummy files. How do I resolve those errors without copying the full dataset into the dummy dir?

Could you check that the names of the dummy data files are right ?
You can use 
```
datasets-cli dummy_data ./datasets/xum
```
to print the expected file names

Ok @lhoestq looks like I got the tests to pass :)
Title: Clicking on a metric in the search page points to datasets page giving "Missing dataset" warning
Author: ogabrielluiz (NONE)
State: closed
Created: 2020-10-21T22:56:23Z

Hi! Sorry if this isn't the right place to talk about the website, I just didn't exactly where to write this.

Searching a metric in https://huggingface.co/metrics gives the right results but clicking on a metric (E.g ROUGE) points to https://huggingface.co/datasets/rouge. Clicking on a metric without searching points to the right page.

Thanks for all the great work!

Comments:
Thanks for the report, can reproduce. Will fix

Fixed now @ogabrielluiz
Title: Error loading ms_marco v2.1 using load_dataset()
Author: JainSahit (NONE)
State: closed
Created: 2020-10-21T19:54:43Z

Code:
`dataset = load_dataset('ms_marco', 'v2.1')`

Error:
```
`---------------------------------------------------------------------------
JSONDecodeError                           Traceback (most recent call last)
<ipython-input-16-34378c057212> in <module>()
      9 
     10 # Downloading and loading a dataset
---> 11 dataset = load_dataset('ms_marco', 'v2.1')

10 frames
/usr/lib/python3.6/json/decoder.py in raw_decode(self, s, idx)
    353         """
    354         try:
--> 355             obj, end = self.scan_once(s, idx)
    356         except StopIteration as err:
    357             raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Unterminated string starting at: line 1 column 388988661 (char 388988660)
`
```

Comments:
There was a similar issue in #294 
Clearing the cache and download again the dataset did the job. Could you try to clear your cache and download the dataset again ?

I was able to load the dataset successfully, I'm pretty sure it's just a cache issue that you have.
Let me know if clearing your cache fixes the problem

Yes, it indeed was a cache issue!
Thanks for reaching out!!
Title: load_dataset doesn't include `features` in its hash
Author: sgugger (CONTRIBUTOR)
State: closed
Created: 2020-10-21T15:16:41Z

It looks like the function `load_dataset` does not include what's passed in the `features` argument when creating a hash for a given  dataset. As a result, if a user includes new features from an already downloaded dataset, those are ignored.

Example: some models on the hub have a different ordering for the labels than what `datasets` uses for MNLI so I'd like to do something along the lines of:
```
dataset = load_dataset("glue", "mnli")
features = dataset["train"].features
features["label"] = ClassLabel(names = ['entailment', 'contradiction', 'neutral'])  # new label order
dataset = load_dataset("glue", "mnli", features=features)
```

Comments:
Title: Add Quail question answering dataset
Author: sai-prasanna (CONTRIBUTOR)
State: closed
Created: 2020-10-20T19:33:14Z

QuAIL is a multi-domain RC dataset featuring news, blogs, fiction and user stories. Each domain is represented by 200 texts,    which gives us a 4-way data split. The texts are 300-350 word excerpts from CC-licensed texts that were hand-picked so as to make sense to human readers without larger context. Domain diversity mitigates the issue of possible overlap between training and test data of large pre-trained models, which the current SOTA systems are based on. For instance, BERT is trained on Wikipedia + BookCorpus, and was tested on Wikipedia-based SQuAD (Devlin, Chang, Lee, & Toutanova, 2019).

https://text-machine-lab.github.io/blog/2020/quail/ @annargrs

Comments:
Title: dataset(ngt): add ngt dataset initial loading script
Author: AmitMY (CONTRIBUTOR)
State: closed
Created: 2020-10-20T14:04:58Z

Currently only making the paths to the annotation ELAN (eaf) file and videos available.
This is the first accessible way to download this dataset, which is not manual file-by-file.

Only downloading the necessary files, the annotation files are very small, 20MB for all of them, but the video files are large, 100GB in total, saved in `mpg` format. 
I do not intend to actually store these as an uncompressed array of frames, because it will be huge.

Future updates may add pose estimation files for all videos, making it easier to work with this data

Comments:
Title: Fix emotion description
Author: lewtun (MEMBER)
State: closed
Created: 2020-10-20T13:28:39Z

Fixes the description of the emotion dataset to reflect the class names observed in the data, not the ones described in the paper.

I also took the liberty to make use of `ClassLabel` for the emotion labels.

Comments:
Hello, I probably have a silly question but the labels of the emotion dataset are in the form of numbers and not string, so I can not use the function classification_report because it mixes numbers and string (prediction). How can I access the label in the form of a string and not a number? 
Thank you in advance.
Title: Dataset Explorer Doesn't Work for squad_es and squad_it
Author: gaotongxiao (NONE)
State: closed
Created: 2020-10-19T19:34:12Z
Labels: nlp-viewer

https://huggingface.co/nlp/viewer/?dataset=squad_es
https://huggingface.co/nlp/viewer/?dataset=squad_it

Both pages show "OSError: [Errno 28] No space left on device".

Comments:
Oups wrong click.
This one is for you @srush
Title: Add OCNLI, a new CLUE dataset
Author: JetRunner (CONTRIBUTOR)
State: closed
Created: 2020-10-19T11:06:33Z

OCNLI stands for Original Chinese Natural Language Inference. It is a corpus for
            Chinese Natural Language Inference, collected following closely the procedures of MNLI,
            but with enhanced strategies aiming for more challenging inference pairs. We want to
            emphasize we did not use human/machine translation in creating the dataset, and thus
            our Chinese texts are original and not translated.

Comments:
Thanks :) merging it
Title: Add wiki dpr multiset embeddings
Author: lhoestq (MEMBER)
State: closed
Created: 2020-10-16T09:05:49Z

There are two DPR encoders, one trained on Natural Questions and one trained on a multiset/hybrid dataset.
Previously only the embeddings from the encoder trained on NQ were available. I'm adding the ones from the encoder trained on the multiset/hybrid dataset.
In the configuration you can now specify `embeddings_name="nq"` or `embeddings_name="multiset"`

Comments:
I still have to compute the dataset_infos, and build + host the indexes

update: I'm computing the metadata, will update the PR soon

Finally all green and ready to merge :)
Title: Replace seqeval code with original classification_report for simplicity
Author: Hironsan (CONTRIBUTOR)
State: closed
Created: 2020-10-16T08:51:45Z

Recently, the original seqeval has enabled us to get per type scores and overall scores as a dictionary.

This PR replaces the current code with the original function(`classification_report`) to simplify it.

Also, the original code has been updated to fix #352.
- Related issue: https://github.com/chakki-works/seqeval/pull/38


```python
from datasets import load_metric
metric = load_metric("seqeval")
y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
metric.compute(predictions=y_pred, references=y_true)
# Output: {'MISC': {'precision': 0.0, 'recall': 0.0, 'f1': 0, 'number': 1}, 'PER': {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'number': 1}, 'overall_precision': 0.5, 'overall_recall': 0.5, 'overall_f1': 0.5, 'overall_accuracy': 0.8}
```

Comments:
Hello,

I ran https://github.com/huggingface/transformers/blob/master/examples/token-classification/run.sh

And received this error:
```
100%|██████████| 407/407 [21:37<00:00,  3.44s/it]Traceback (most recent call last):
  File "run_ner.py", line 445, in <module>
    main()
  File "run_ner.py", line 398, in main
    results = trainer.evaluate()
  File "/data/2021/transformers/src/transformers/trainer.py", line 1470, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/data/2021/transformers/src/transformers/trainer.py", line 1622, in prediction_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=preds, label_ids=label_ids))
  File "run_ner.py", line 345, in compute_metrics
    results = metric.compute(predictions=true_predictions, references=true_labels)
  File "/usr/local/lib/python3.6/dist-packages/datasets/metric.py", line 398, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/root/.cache/huggingface/modules/datasets_modules/metrics/seqeval/81eda1ff004361d4fa48754a446ec69bb7aa9cf4d14c7215f407d1475941c5ff/seqeval.py", line 97, in _compute
    report = classification_report(y_true=references, y_pred=predictions, suffix=suffix, output_dict=True)
TypeError: classification_report() got an unexpected keyword argument 'output_dict'
```

I'm still trying multiple things to see if I can work around this, but I thought it might be useful to mention it here.

```
Name: transformers
Version: 4.3.0.dev0

Name: datasets
Version: 1.2.1
```

Hi, can you try to update your local installation of `seqeval` ?

```
pip install --upgrade seqeval
```

@lhoestq thanks for the reply. Indeed it was some issue with my setup. I removed the "transformers" and "datasets" (that I had previously installed from the source code), cleared the cache and installed everything again. It works great now!