Updated OPUS Open Subtitles Dataset with metadata information

Hi !
About the problems you mentioned:
- Saving the infos is only done for the configurations inside the BUILDER_CONFIGS. Otherwise you would need to run the scripts on ALL language pairs, which is not what we want.
- Moreover when you're on your branch, please specify the path to your local version of the dataset script, like "./datasets/open_subtitles". Otherwise the dataset is loaded from the master branch on github.
Hope that clarifies things a bit

And of course feel free to add methods or classmethods to your builder.

Great! Thank you :)
I'll close the issue as well.

Close #1844

Problems:
- I ran `python datasets-cli test datasets/open_subtitles --save_infos --all_configs`, hence the change in `dataset_infos.json`, but it appears that the metadata features have not been added for all pairs. Any idea why that might be?
- Possibly related to the above, I tried doing `pip uninstall datasets && pip install -e ".[dev]"` after the changes, and loading the dataset via `load_dataset("open_subtitles", lang1='hi', lang2='it')` to check if the update worked, but the loaded dataset did not contain the metadata fields (neither in the features nor doing `next(iter(dataset['train']))`). What step(s) did I miss?

Questions:
- Is it ok to have a `classmethod` in there? I have not seen any in the few other datasets I have checked. I could make it a local method of the `_generate_examples` method, but I'd rather not duplicate the logic...


Add Winogender Schemas

dataset request

Requesting to add a new dataset

Nevermind, this one is already available on the hub under the name `'wino_bias'`: https://huggingface.co/datasets/wino_bias

## Adding a Dataset
- **Name:** Winogender Schemas
- **Description:** Winogender Schemas (inspired by Winograd Schemas) are minimal pairs of sentences that differ only by the gender of one pronoun in the sentence, designed to test for the presence of gender bias in automated coreference resolution systems.
- **Paper:** https://arxiv.org/abs/1804.09301
- **Data:** https://github.com/rudinger/winogender-schemas (see data directory)
- **Motivation:** Testing gender bias in automated coreference resolution systems, improve coreference resolution in general.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add WikiCREM

Requesting to add a new dataset

Hi @NielsRogge I would like to work on this dataset.

Thanks!

## Adding a Dataset
- **Name:** WikiCREM
- **Description:** A large unsupervised corpus for coreference resolution.
- **Paper:** https://arxiv.org/abs/1905.06290
- **Github repo:**: https://github.com/vid-koci/bert-commonsense
- **Data:** https://ora.ox.ac.uk/objects/uuid:c83e94bb-7584-41a1-aef9-85b0e764d9e3
- **Motivation:** Coreference resolution, common sense reasoning

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Fix writing GPU Faiss index

As reported in by @corticalstack there is currently an error when we try to save a faiss index on GPU.

I fixed that by checking the index `getDevice()` method before calling `index_gpu_to_cpu`

Close #1859


Fix Limit url

The test.json file of the Literal-Motion-in-Text (LiMiT) dataset was removed recently on the master branch of the repo at https://github.com/ilmgut/limit_dataset

This PR uses the previous commit sha to download the file instead, as suggested by @Paethon

Close #1836


Add loading from the Datasets Hub + add relative paths in download manager

I just added the steps to share a dataset on the datasets hub. It's highly inspired by the steps to share a model in the `transformers` doc.

Moreover once the new huggingface_hub is released we can update the version in the setup.py. We also need to update the command to create a dataset repo in the documentation

I added a few more tests with the "lhoestq/test" dataset I added on the hub and it works fine :)

Here is the PR adding support for datasets repos in `huggingface_hub`: https://github.com/huggingface/huggingface_hub/pull/14

With the new Datasets Hub on huggingface.co it's now possible to have a dataset repo with your own script and data.
For example: https://huggingface.co/datasets/lhoestq/custom_squad/tree/main contains one script and two json files.

You can load it using
```python
from datasets import load_dataset

d = load_dataset("lhoestq/custom_squad")
```

To be able to use the data files that live right next to the dataset script on the repo in the hub, I added relative paths support for the DownloadManager. For example in the repo mentioned above, there are two json files that can be downloaded via
```python
_URLS = {
    "train": "train-v1.1.json",
    "dev": "dev-v1.1.json",
}
downloaded_files = dl_manager.download_and_extract(_URLS)
```

To make it work, I set the `base_path` of the DownloadManager to be the parent path of the dataset script (which comes from either a local path or a remote url).

I also had to add the auth header of the requests to huggingface.co for private datasets repos. The token is fetched from [huggingface_hub](https://github.com/huggingface/huggingface_hub).


Error "in void don't know how to serialize this type of index" when saving index to disk when device=0 (GPU)

Hi @corticalstack ! Thanks for reporting. Indeed in the recent versions of Faiss we must use `getDevice` to check if the index in on GPU.

I'm opening a PR

I fixed this issue. It should work fine now.
Feel free to try it out by installing `datasets` from source.
Otherwise you can wait for the next release of `datasets` (in a few days)

Thanks for such a quick fix and merge to master, pip installed git master, tested all OK

Error serializing faiss index.  Error as follows:

`Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) at /home/conda/feedstock_root/build_artifacts/faiss-split_1612472484670/work/faiss/impl/index_write.cpp:453: don't know how to serialize this type of index`

Note:

`torch.cuda.is_available()` reports:

```
Cuda is available
cuda:0

```

Adding index, device=0 for GPU.

`dataset.add_faiss_index(column='embeddings', index_name='idx_embeddings', device=0)`

However, during a quick debug, self.faiss_index has no attr "device" when checked in` search.py, method save`, so fails to transform gpu index to cpu index.  If I add index without device, index is saved OK.

```
def save(self, file: str):
        """Serialize the FaissIndex on disk"""
        import faiss  # noqa: F811

        if (
            hasattr(self.faiss_index, "device")
            and self.faiss_index.device is not None
            and self.faiss_index.device > -1
        ):
            index = faiss.index_gpu_to_cpu(self.faiss_index)
        else:
            index = self.faiss_index
        faiss.write_index(index, file)
```


Clean config getenvs

Following #1848
Remove double getenv calls and fix one issue with rarfile

cc @albertvillanova


Unable to upload "community provided" dataset - 400 Client Error

Hi ! We're in the process of switching the community datasets to git repos, exactly like what we're doing for models.
You can find an example here:
https://huggingface.co/datasets/lhoestq/custom_squad/tree/main

We'll update the CLI in the coming days and do a new release :)

Also cc @julien-c maybe we can make improve the error message ?

Hi,
i'm trying to a upload a dataset as described [here](https://huggingface.co/docs/datasets/v1.2.0/share_dataset.html#sharing-a-community-provided-dataset). This is what happens:

```
$ datasets-cli login
$ datasets-cli upload_dataset my_dataset
About to upload file /path/to/my_dataset/dataset_infos.json to S3 under filename my_dataset/dataset_infos.json and namespace username
About to upload file /path/to/my_dataset/my_dataset.py to S3 under filename my_dataset/my_dataset.py and namespace username
Proceed? [Y/n] Y
Uploading... This might take a while if files are large
400 Client Error: Bad Request for url: https://huggingface.co/api/datasets/presign
huggingface.co migrated to a new model hosting system.
You need to upgrade to transformers v3.5+ to upload new models.
More info at https://discuss.hugginface.co or https://twitter.com/julien_c. Thank you!
```
I'm using the latest releases of datasets and transformers.


load_dataset("amazon_polarity") NonMatchingChecksumError

Hi ! This issue may be related to #996
This comes probably from the Quota Exceeded error from Google Drive.
Can you try again tomorrow and see if you still have the error ?

On my side I didn't get any error today with `load_dataset("amazon_polarity")`

@lhoestq Hi! I encounter the same error when loading `yelp_review_full`.

```
from datasets import load_dataset
dataset_yp = load_dataset("yelp_review_full")
```

When you say the "Quota Exceeded from Google drive". Is this a quota from the dataset owner? or the quota from our (the runner) Google Drive?

> When you say the "Quota Exceeded from Google drive". Is this a quota from the dataset owner? or the quota from our (the runner) Google Drive?

Each file on Google Drive can be downloaded only a certain amount of times per day because of a quota. The quota is reset every day. So if too many people download the dataset the same day, then the quota is likely to exceed.
That's a really bad limitations of Google Drive and we should definitely find another host for these dataset than Google Drive.
For now I would suggest to wait and try again later..

So far the issue happened with CNN DailyMail, Amazon Polarity and Yelp Reviews.
Are you experiencing the issue with other datasets ? @calebchiam @dtch1997

@lhoestq Gotcha, that is quite problematic...for what it's worth, I've had no issues with the other datasets I tried, such as `yelp_reviews_full` and `amazon_reviews_multi`.

Same issue today with "big_patent", though the symptoms are slightly different.

When running

```py
from datasets import load_dataset
load_dataset("big_patent", split="validation")
```

I get the following
`FileNotFoundError: Local file \huggingface\datasets\downloads\6159313604f4f2c01e7d1cac52139343b6c07f73f6de348d09be6213478455c5\bigPatentData\train.tar.gz doesn't exist`

I had to look into `6159313604f4f2c01e7d1cac52139343b6c07f73f6de348d09be6213478455c5`  (which is a file instead of a folder) and got the following:

`<!DOCTYPE html><title>Google Drive - Quota exceeded</title><meta http-equiv="content-type" content="text/html; charset=utf-8"/><link href=/static/doclist/client/css/4033072956-untrustedcontent.css rel="stylesheet" nonce="JV0t61Smks2TEKdFCGAUFA"><link rel="icon" href="//ssl.gstatic.com/images/branding/product/1x/drive_2020q4_32dp.png"/>
<nobr>Search Images Maps Play YouTube News Gmail Drive More »</nobr>

<nobr>Settings | Help | Sign in</nobr>

Sorry, you can't view or download this file at this time.

Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator.

© 2021 Google - Help - Privacy & Terms
`

A similar issue arises when trying to stream the dataset

```python
>>> from datasets import load_dataset
>>> iter_dset = load_dataset("amazon_polarity", split="test", streaming=True)
>>> iter(iter_dset).__next__()

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
~\lib\tarfile.py in nti(s)
    186             s = nts(s, "ascii", "strict")
--> 187             n = int(s.strip() or "0", 8)
    188         except ValueError:

ValueError: invalid literal for int() with base 8: 'e nonce='

During handling of the above exception, another exception occurred:

InvalidHeaderError                        Traceback (most recent call last)
~\lib\tarfile.py in next(self)
   2288             try:
-> 2289                 tarinfo = self.tarinfo.fromtarfile(self)
   2290             except EOFHeaderError as e:

~\lib\tarfile.py in fromtarfile(cls, tarfile)
   1094         buf = tarfile.fileobj.read(BLOCKSIZE)
-> 1095         obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)
   1096         obj.offset = tarfile.fileobj.tell() - BLOCKSIZE

~\lib\tarfile.py in frombuf(cls, buf, encoding, errors)
   1036
-> 1037         chksum = nti(buf[148:156])
   1038         if chksum not in calc_chksums(buf):

~\lib\tarfile.py in nti(s)
    188         except ValueError:
--> 189             raise InvalidHeaderError("invalid header")
    190     return n

InvalidHeaderError: invalid header

During handling of the above exception, another exception occurred:

ReadError                                 Traceback (most recent call last)
<ipython-input-5-6b9058341b2b> in <module>
----> 1 iter(iter_dset).__next__()

~\lib\site-packages\datasets\iterable_dataset.py in __iter__(self)
    363
    364     def __iter__(self):
--> 365         for key, example in self._iter():
    366             if self.features:
    367                 # we encode the example for ClassLabel feature types for example

~\lib\site-packages\datasets\iterable_dataset.py in _iter(self)
    360         else:
    361             ex_iterable = self._ex_iterable
--> 362         yield from ex_iterable
    363
    364     def __iter__(self):

~\lib\site-packages\datasets\iterable_dataset.py in __iter__(self)
     77
     78     def __iter__(self):
---> 79         yield from self.generate_examples_fn(**self.kwargs)
     80
     81     def shuffle_data_sources(self, seed: Optional[int]) -> "ExamplesIterable":

~\.cache\huggingface\modules\datasets_modules\datasets\amazon_polarity\56923eeb72030cb6c4ea30c8a4e1162c26b25973475ac1f44340f0ec0f2936f4\amazon_polarity.py in _generate_examples(self, filepath, files)
    114     def _generate_examples(self, filepath, files):
    115         """Yields examples."""
--> 116         for path, f in files:
    117             if path == filepath:
    118                 lines = (line.decode("utf-8") for line in f)

~\lib\site-packages\datasets\utils\streaming_download_manager.py in __iter__(self)
    616
    617     def __iter__(self):
--> 618         yield from self.generator(*self.args, **self.kwargs)
    619
    620

~\lib\site-packages\datasets\utils\streaming_download_manager.py in _iter_from_urlpath(cls, urlpath, use_auth_token)
    644     ) -> Generator[Tuple, None, None]:
    645         with xopen(urlpath, "rb", use_auth_token=use_auth_token) as f:
--> 646             yield from cls._iter_from_fileobj(f)
    647
    648     @classmethod

~\lib\site-packages\datasets\utils\streaming_download_manager.py in _iter_from_fileobj(cls, f)
    624     @classmethod
    625     def _iter_from_fileobj(cls, f) -> Generator[Tuple, None, None]:
--> 626         stream = tarfile.open(fileobj=f, mode="r|*")
    627         for tarinfo in stream:
    628             file_path = tarinfo.name

~\lib\tarfile.py in open(cls, name, mode, fileobj, bufsize, **kwargs)
   1603             stream = _Stream(name, filemode, comptype, fileobj, bufsize)
   1604             try:
-> 1605                 t = cls(name, filemode, stream, **kwargs)
   1606             except:
   1607                 stream.close()

~\lib\tarfile.py in __init__(self, name, mode, fileobj, format, tarinfo, dereference, ignore_zeros, encoding, errors, pax_headers, debug, errorlevel, copybufsize)
   1484             if self.mode == "r":
   1485                 self.firstmember = None
-> 1486                 self.firstmember = self.next()
   1487
   1488             if self.mode == "a":

~\lib\tarfile.py in next(self)
   2299                     continue
   2300                 elif self.offset == 0:
-> 2301                     raise ReadError(str(e))
   2302             except EmptyHeaderError:
   2303                 if self.offset == 0:

ReadError: invalid header

```

This error still happens, but for a different reason now: Google Drive returns a warning instead of the dataset.

Hi ! Thanks for reporting. Google Drive changed the way to bypass the warning message recently.

The latest release `1.18.4` fixes this for datasets loaded in a regular way.

We opened a PR to fix this recently for streaming mode at #3843 - we'll do a new release once the fix is merged :)

Hi, it seems that loading the amazon_polarity dataset gives a NonMatchingChecksumError.

To reproduce:
```
load_dataset("amazon_polarity")
```
This will give the following error:
```
---------------------------------------------------------------------------
NonMatchingChecksumError                  Traceback (most recent call last)
<ipython-input-3-8559a03fe0f8> in <module>()
----> 1 dataset = load_dataset("amazon_polarity")

3 frames
/usr/local/lib/python3.6/dist-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     37     if len(bad_urls) > 0:
     38         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 39         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     40     logger.info("All the checksums matched successfully" + for_verification_name)
     41

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://drive.google.com/u/0/uc?id=0Bz8a_Dbh9QhbaW12WVVZS2drcnM&export=download']
```


Minor fix in the docs


Feature Request: Dataset.add_item

enhancement

New feature or request

Hi @sshleifer.

I am not sure of understanding the need of the `add_item` approach...

By just reading your "Desired API" section, I would say you could (nearly) get it with a 1-column Dataset:
```python
data = {"input_ids": [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])]}
ds = Dataset.from_dict(data)
assert (ds["input_ids"][0] == np.array([4,4,2])).all()
```

Hi @sshleifer :)

We don't have methods like `Dataset.add_batch` or `Dataset.add_entry/add_item` yet.
But that's something we'll add pretty soon. Would an API that looks roughly like this help ? Do you have suggestions ?
```python
import numpy as np
from datasets import Dataset

tokenized = [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])

# API suggestion (not available yet)
d = Dataset()
for input_ids in tokenized:
    d.add_item({"input_ids": input_ids})

print(d[0]["input_ids"])
# [4, 4, 2]
```

Currently you can define a dataset with what @albertvillanova suggest, or via a generator using dataset builders. It's also possible to [concatenate datasets](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate#datasets.concatenate_datasets).

Your API looks perfect @lhoestq, thanks!

I'm trying to integrate `huggingface/datasets` functionality into `fairseq`, which requires (afaict) being able to build a dataset through an `add_item` method, such as https://github.com/pytorch/fairseq/blob/master/fairseq/data/indexed_dataset.py#L318, as opposed to loading all the text into arrow, and then `dataset.map(binarizer)`.
Is this possible at the moment? Is there an example? I'm happy to use raw `pa.Table` but not sure whether it will support uneven length entries.

### Desired API

```python
import numpy as np
tokenized: List[np.NDArray[np.int64]] = [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])

def build_dataset_from_tokenized(tokenized: List[np.NDArray[int]]) -> Dataset:
   """FIXME"""
   dataset = EmptyDataset()
   for t in tokenized: dataset.append(t)
   return dataset
ds = build_dataset_from_tokenized(tokenized)
assert (ds[0] == np.array([4,4,2])).all()
```

### What I tried
grep, google for "add one entry at a time", "datasets.append"

### Current Code
This code achieves the same result but doesn't fit into the `add_item` abstraction.

```python
    dataset = load_dataset('text', data_files={'train': 'train.txt'})
    tokenizer = RobertaTokenizerFast.from_pretrained('roberta-base', max_length=4096)
    def tokenize_function(examples):
        ids = tokenizer(examples['text'], return_attention_mask=False)['input_ids']
        return {'input_ids': [x[1:] for x in ids]}
    ds = dataset.map(tokenize_function, batched=True, num_proc=4, remove_columns=['text'], load_from_cache_file=not overwrite_cache)
	print(ds['train'][0]) => np array
```

Thanks in advance!


Configure library root logger at the module level

Configure library root logger at the datasets.logging module level (singleton-like).

By doing it this way:
- we are sure configuration is done only once: module level code is only runned once
- no need of global variable
- no need of threading lock


Add Arabic Speech Corpus


set bert_score version dependency

Set the bert_score version in requirements since previous versions of bert_score will fail with datasets (closes #843)


Add cord 19 dataset

Cleaned-up version of previous PR: https://github.com/huggingface/datasets/pull/1129

Before merging I might tweak a little bit the dummy data to avoid having to check if the `document_parses` and `embeddings` directories exist or not. I'll do that later today

Looks all good now ! Thanks a lot @ggdupont :)
Merging

Initial version only reading the metadata in CSV.

### Checklist:
- [x] Create the dataset script /datasets/my_dataset/my_dataset.py using the template
- [x] Fill the _DESCRIPTION and _CITATION variables
- [x] Implement _infos(), _split_generators() and _generate_examples()
- [x] Make sure that the BUILDER_CONFIGS class attribute is filled with the different configurations of the dataset and that the BUILDER_CONFIG_CLASS is specified if there is a custom config class.
- [x] Generate the metadata file dataset_infos.json for all configurations
- [x] Generate the dummy data dummy_data.zip files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card README.md using the template and at least fill the tags
- [x] Both tests for the real data and the dummy data pass.

### Extras:
- [x] add more metadata
- [x] add full text
- [x] add pre-computed document embedding


Add TIMIT

dataset request

Requesting to add a new dataset

@patrickvonplaten Could you please help me with how the output text has to be represented in the data? TIMIT has Words, Phonemes and texts. Also has lot on info on the speaker and the dialect. Could you please help me? An example of how to arrange it would be super helpful!

Hey @vrindaprabhu - sure I'll help you :-) Could you open a first PR for TIMIT where you copy-paste more or less the `librispeech_asr` script: https://github.com/huggingface/datasets/blob/28be129db862ec89a87ac9349c64df6b6118aff4/datasets/librispeech_asr/librispeech_asr.py#L93 (obviously replacing all the naming and links correctly...) and then you can list all possible outputs in the features dict: https://github.com/huggingface/datasets/blob/28be129db862ec89a87ac9349c64df6b6118aff4/datasets/librispeech_asr/librispeech_asr.py#L104 (words, phonemes should probably be of kind `datasets.Sequence(datasets.Value("string"))` and texts I think should be of type `"text": datasets.Value("string")`.

When you've opened a first PR, I think it'll be much easier for us to take a look together :-)

I am sorry! I created the PR [#1903](https://github.com/huggingface/datasets/pull/1903#). Requesting your comments! CircleCI tests are failing, will address them along with your comments!

## Adding a Dataset
- **Name:** *TIMIT*
- **Description:** *The TIMIT corpus of read speech has been designed to provide speech data for the acquisition of acoustic-phonetic knowledge and for the development and evaluation of automatic speech recognition systems*

- **Paper:** *Homepage*: http://groups.inf.ed.ac.uk/ami/corpus/ / *Wikipedia*: https://en.wikipedia.org/wiki/TIMIT
- **Data:** *https://deepai.org/dataset/timit*
- **Motivation:** Important speech dataset

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Refactoring: Create config module

Refactorize configuration settings into their own module.

This could be seen as a Pythonic singleton-like approach. Eventually a config instance class might be created.


[Metrics] Add word error metric metric

Feel free to merge once the CI is all green ;)

This PR adds the word error rate metric to datasets.
WER: https://en.wikipedia.org/wiki/Word_error_rate
for speech recognition. WER is the main metric used in ASR.

`jiwer` seems to be a solid library (see https://github.com/asteroid-team/asteroid/pull/329#discussion_r525158939)


Make DownloadManager downloaded/extracted paths accessible

First I was thinking of the dict, which makes sense for .download, mapping URL to downloaded path. However does this make sense for .extract, mapping the downloaded path to the extracted path? I ask this because the user did not chose the downloaded path, so this is completely unknown for them...

There could be several situations:
- download a file with no extraction
- download a file and extract it
- download a file, extract it and then inside the output folder extract some more files
- extract a local file (for datasets with data that are manually downloaded for example)
- extract a local file, and then inside the output folder extract some more files

So I think it's ok to have `downloaded_paths` as a dict url -> downloaded_path and `extracted_paths` as a dict local_path -> extracted_path.

OK. I am refactoring this. I have opened #1879, as an intermediate step...

Make accessible the file paths downloaded/extracted by DownloadManager.

Close #1831.

The approach:
- I set these paths as DownloadManager attributes: these are DownloadManager's concerns
- To access to these from DatasetBuilder, I set the DownloadManager instance as DatasetBuilder attribute: object composition


Enable logging propagation and remove logging handler

Thank you @lhoestq. This logging configuration makes more sense to me.

Once propagation is allowed, the end-user can customize logging behavior and add custom handlers to the proper top logger in the hierarchy.

And I also agree with following the best practices and removing any custom handlers:
- it is the end user who has to implement any custom handlers
- indeed, the previous logging problem with TensorFlow was due to the fact that absl did not follow best practices and had implemented a custom handler

Our errors/warnings will be displayed anyway, even if we do not implement any custom handler. Since Python 3.2, logging has a built-in "default" handler (logging.lastResort) with the expected default behavior (sending error/warning messages to sys.stderr), which is used only if the end user has not configured any custom handler.

We used to have logging propagation disabled because of this issue: https://github.com/tensorflow/tensorflow/issues/26691
But since it's now fixed we should re-enable it. This is important to keep the default logging behavior for users, and propagation is also needed for pytest fixtures as asked in #1826

I also removed the handler that was added since, according to the logging [documentation](https://docs.python.org/3/howto/logging.html#configuring-logging-for-a-library):
> It is strongly advised that you do not add any handlers other than NullHandler to your library’s loggers. This is because the configuration of handlers is the prerogative of the application developer who uses your library. The application developer knows their target audience and what handlers are most appropriate for their application: if you add handlers ‘under the hood’, you might well interfere with their ability to carry out unit tests and deliver logs which suit their requirements.

It could have been useful if we wanted to have a custom formatter for the logging but I think it's more important to keep the logging as default to not interfere with the users' logging management.

Therefore I also removed the two methods `datasets.logging.enable_default_handler` and `datasets.logging.disable_default_handler`.

cc @albertvillanova this should let you use capsys/caplog in pytest
cc @LysandreJik @sgugger if you want to do the same in `transformers`


Update Open Subtitles corpus with original sentence IDs

good first issue

Good for newcomers

Hi ! You're right this can can useful.
This should be easy to add, so feel free to give it a try if you want to contribute :)
I think we just need to add it to the _generate_examples method of the OpenSubtitles dataset builder [here](https://github.com/huggingface/datasets/blob/master/datasets/open_subtitles/open_subtitles.py#L103)

Hey @lhoestq , absolutely yes! Just one question before I start implementing. The ids found in the zip file have this format:
(the following is line `22497315` of the `ids` file of the `de-en` dump)

`de/2017/7006210/7063319.xml.gz  en/2017/7006210/7050201.xml.gz  335     339 340` (every space is actually a tab, aside from the space between `339` and `340`)

Where filenames encode the information like this: `lang/year/imdb_id/opensubtitles_id.xml.gz` whereas the numbers correspond to the sentence ids which are linked together (i.e. sentence `335` of the German subtitle corresponds to lines `339` and `340` of the English file)

That being said, do you think I should stick to the raw sentence id (and replace the current sequential id) or should I include more detailed metadata (or both things maybe)?

Going with raw ID is surely simpler, but including `year`, `imdbId` and `subtitleId` should save space as they're just integers; besides, any operation (like filtering or grouping) will be much easier if users don't have to manually parse the ids every time.
As for the language-specific sentenceIds, what could be the best option? A list of integers or a comma-separated string?

**Note:** I did not find any official information about this encoding, but it appears to check out:
https://www.imdb.com/title/tt7006210/, https://www.opensubtitles.org/en/subtitles/7063319 and https://www.opensubtitles.org/en/subtitles/7050201 all link to the same episode, so I guess (I hope!) it's correct.

I like the idea of having `year`, `imdbId` and `subtitleId` as columns for filtering for example.
And for the `sentenceIds` a list of integers is fine.

Something like this? (adapted from [here](https://github.com/huggingface/datasets/blob/master/datasets/open_subtitles/open_subtitles.py#L114))

```python
result = (
  sentence_counter,
    {
      "id": str(sentence_counter),
      "meta": {
        "year": year,
        "imdbId": imdb_id,
        "subtitleId": {l1: l1_sub_id, l2: l2_sub_id},
        "sentenceIds": {l1: [... source_sids ...], l2: [... target_sids ...]},
        # or maybe src/tgt? I'd go with the first one for consistency with 'translation'
        "subtitleId": {"src": l1_sub_id, "tgt": l2_sub_id},
        "sentenceIds": {"src": [... source_sids ...], "tgt": [... target_sids ...]},
      },
      "translation": {l1: x, l2: y},
    },
  )
```
Or at top level, avoiding nesting into 'meta'?

Hi! It would be great if you could add the original sentence ids to [Open Subtitles](https://huggingface.co/datasets/open_subtitles).

I can think of two reasons: first, it's possible to gather sentences for an entire document (the original ids contain media id, subtitle file id and sentence id), therefore somewhat allowing for document-level machine translation (and other document-level stuff which could be cool to have); second, it's possible to have parallel sentences in multiple languages, as they share the same ids across bitexts.

I think I should tag @abhishekkrthakur as he's the one who added it in the first place.

Thanks!


MustC Speech Translation

dataset request

Requesting to add a new dataset

Hi @patrickvonplaten  I would like to work on this dataset.

Thanks!

That's awesome! Actually, I just noticed that this dataset might become a bit too big!

MuST-C is the main dataset used for IWSLT19 and should probably be added as a standalone dataset. Would you be interested also in adding `datasets/MuST-C` instead?

Description:
_MuST-C is a multilingual speech translation corpus whose size and quality facilitates the training of end-to-end systems for speech translation from English into several languages. For each target language, MuST-C comprises several hundred hours of audio recordings from English TED Talks, which are automatically aligned at the sentence level with their manual transcriptions and translations._

Paper: https://www.aclweb.org/anthology/N19-1202.pdf

Dataset: https://ict.fbk.eu/must-c/ (One needs to fill out a short from to download the data, but it's very easy).

It would be awesome if you're interested in adding this datates. I'm very happy to guide you through the PR! I think the easiest way to start would probably be to read [this README on how to add a dataset](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md) and open a PR. Think you can copy & paste some code from:

- Librispeech_asr: https://github.com/huggingface/datasets/blob/master/datasets/librispeech_asr/librispeech_asr.py
- Flores Translation: https://github.com/huggingface/datasets/blob/master/datasets/flores/flores.py

Think all the rest can be handled on the PR :-)

Hi @patrickvonplaten
I have tried downloading this dataset, but the connection seems to reset all the time. I have tried it via the browser, wget, and using gdown . But it gives me an error message. _"The server is busy or down, pls try again"_ (rephrasing the message here)

I have completed adding 4 datasets in the previous data sprint (including the IWSLT dataset #1676 ) ...so just checking if you are able to download it at your end. Otherwise will write to the dataset authors to update the links.

Let me check tomorrow! Thanks for leaving this message!

@skyprince999, I think I'm getting the same error you're getting :-/

```
Sorry, you can't view or download this file at this time.

Too many users have viewed or downloaded this file recently. Please try accessing the file again later. If the file you are trying to access is particularly large or is shared with many people, it may take up to 24 hours to be able to view or download the file. If you still can't access a file after 24 hours, contact your domain administrator.
```

It would be great if you could write the authors to see whether they can fix it.
Also cc @lhoestq - do you think we could mirror the dataset?

Also there are huge those datasets. Think downloading MuST-C v1.2 amounts to ~ 1000GB... because there are 14 possible configs each around 60-70GB. I think users mostly will only use one of the 14 configs so that they would only need, in theory, will have to download ~60GB which is ok. But I think this functionality doesn't exist yet in `datasets` no? cc @lhoestq

> Also cc @lhoestq - do you think we could mirror the dataset?

Yes we can mirror it if the authors are fine with it. You can create a dataset repo on huggingface.co (possibly under the relevant org) and add the mirrored data files.

> I think users mostly will only use one of the 14 configs so that they would only need, in theory, will have to download ~60GB which is ok. But I think this functionality doesn't exist yet in datasets no? cc @lhoestq

If there are different download links for each configuration we can make the dataset builder download only the files related to the requested configuration.

I have written to the dataset authors, highlighting this issue. Waiting for their response.

Update on 25th Feb:
The authors have replied back, they are updating the download link and will revert back shortly!

```
first of all thanks a lot for being interested in MuST-C and for building the data-loader.

Before answering your request, I'd like to clarify that the creation, maintenance, and expansion of MuST-c are not supported by any funded project, so this means that we need to find economic support for all these activities. This also includes permanently moving all the data to AWS or GCP.  We are working at this with the goal of facilitating the use of MuST-C, but this is not something that can happen today. We hope to have some news ASAP and you will be among the first to be informed.

I hope you understand our situation.
```

Awesome, actually @lhoestq let's just ask the authors if we should host the dataset no? They could just use our links then as well for their website - what do you think? Is it fine to use our AWS dataset storage also as external links?

Yes definitely. Shall we suggest them to create a dataset repository under their org on huggingface.co ? @julien-c
The dataset is around 1TB

Sounds good!

Order of magnitude is storage costs ~$20 per TB per month (not including bandwidth).

Happy to provide this to the community as I feel this is an important dataset. Let us know what the authors want to do!

Great! @skyprince999, do you think you could ping the authors here or link to this thread? I think it could be a cool idea to host the dataset on our side then

Done. They replied back, and they want to have a call over a meet/ skype. Is that possible ?
Btw @patrickvonplaten you are looped in that email (_pls check you gmail account_)

@gegallego  there were some concerns regarding dataset usage & attribution by a for-profit company, so couldn't take it forward. Also the download links were unstable.
But I guess if you want to test the fairseq benchmarks, you can connect with them directly for downloading the dataset.

Yes, that dataset is not easy to download... I had to copy it to my Google Drive and use `rsync` to be able to download it.
However, we could add the dataset with a manual download, right?

yes that is possible. I couldn't unfortunately complete this PR, If you would like to add it, please feel free to do it.

## Adding a Dataset
- **Name:** *IWSLT19*
- **Description:** *The Speech Translation Task addresses the translation of English audio into German and Portuguese text.*
- **Hompage:** *https://sites.google.com/view/iwslt-evaluation-2019/speech-translation*
- **Data:** *https://sites.google.com/view/iwslt-evaluation-2019/speech-translation* - all data under "Allowed Training Data" and "Development and Evalutaion Data for TED/How2"
- **Motivation:** Important speech dataset

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add AMI Corpus

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *AMI*
- **Description:** *The AMI Meeting Corpus is a multi-modal data set consisting of 100 hours of meeting recordings. For a gentle introduction to the corpus, see the corpus overview. To access the data, follow the directions given there. Around two-thirds of the data has been elicited using a scenario in which the participants play different roles in a design team, taking a design project from kick-off to completion over the course of a day. The rest consists of naturally occurring meetings in a range of domains. Detailed information can be found in the documentation section.*

- **Paper:** *Homepage*: http://groups.inf.ed.ac.uk/ami/corpus/
- **Data:** *http://groups.inf.ed.ac.uk/ami/download/* - Select all cases in 1) and select "Individual Headsets" & "Microphone array" for 2)
- **Motivation:** Important speech dataset

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add ljspeech

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *ljspeech*
- **Description:** *This is a public domain speech dataset consisting of 13,100 short audio clips of a single speaker reading passages from 7 non-fiction books. A transcription is provided for each clip. Clips vary in length from 1 to 10 seconds and have a total length of approximately 24 hours.

The texts were published between 1884 and 1964, and are in the public domain. The audio was recorded in 2016-17 by the LibriVox project and is also in the public domain.)*
- **Paper:** *Homepage*: https://keithito.com/LJ-Speech-Dataset/
- **Data:** *https://keithito.com/LJ-Speech-Dataset/*
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/ljspeech
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add common voice

Requesting to add a new dataset

I have started working on adding this dataset.

Hey @BirgerMoell - awesome that you started working on Common Voice. Common Voice is a bit special since, there is no direct download link to download the data. In these cases we usually consider two options:

1) Find a hacky solution to extract the download link somehow from the XLM tree of the website
2) If this doesn't work we force the user to download the data himself and add a `"data_dir"` as an input parameter. E.g. you can take a look at how it is done for [this](https://github.com/huggingface/datasets/blob/66f2a7eece98d2778bd22bb5034cb7c2376032d4/datasets/arxiv_dataset/arxiv_dataset.py#L66)

Also the documentation here: https://huggingface.co/docs/datasets/add_dataset.html?highlight=data_dir#downloading-data-files-and-organizing-splits (especially the "note") might be helpful.

Let me know if you have any other questions

I added a Work in Progress pull request (hope that is ok). I've made a card for the dataset and filled out the common_voice.py file with information about the datset (not completely).

I didn't manage to get the tagging tool working locally on my machine but will look into that later.

Left to do.

- Tag the dataset
- Add missing information and update common_voice.py

https://github.com/huggingface/datasets/pull/1886

Awesome! I left a longer comment on the PR :-)

I saw that this current datasets package holds common voice version 6.1, how to add the new version 7.0 that is already available?

Will me merged next week - we're working on it :-)

Common voice still appears to be a 6.1. Is the plan still to upgrade to 7.0?

We actually already have the code and everything ready to add Common Voice 7.0 to `datasets` but are still waiting for the common voice authors to give us the green light :-)

Also gently pinging @phirework and @milupo here

Common Voice 7.0 is available here now: https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0

For anyone else stumbling upon this thread, the 8.0 version is also available now: https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0

## Adding a Dataset
- **Name:** *common voice*
- **Description:** *Mozilla Common Voice Dataset*
- **Paper:** Homepage: https://voice.mozilla.org/en/datasets
- **Data:** https://voice.mozilla.org/en/datasets
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/common_voice
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add Voxforge

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *voxforge*
- **Description:** *VoxForge is a language classification dataset. It consists of user submitted audio clips submitted to the website. In this release, data from 6 languages is collected - English, Spanish, French, German, Russian, and Italian. Since the website is constantly updated, and for the sake of reproducibility, this release contains only recordings submitted prior to 2020-01-01. The samples are splitted between train, validation and testing so that samples from each speaker belongs to exactly one split.*
- **Paper:** *Homepage*: http://www.voxforge.org/
- **Data:** *http://www.voxforge.org/home/downloads*
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/voxforge
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add tedlium

Requesting to add a new dataset

Hi @patrickvonplaten
I can have a look to this dataset later since I am trying to add the OpenSLR dataset https://github.com/huggingface/datasets/pull/2173
Hopefully I have enough space since the compressed file is 21GB. The release 3 is even bigger: 54GB :-0

## Adding a Dataset
- **Name:** *tedlium*
- **Description:** *The TED-LIUM 1-3 corpus is English-language TED talks, with transcriptions, sampled at 16kHz. It contains about 118 hours of speech.*
- **Paper:** Homepage: http://www.openslr.org/7/, https://lium.univ-lemans.fr/en/ted-lium2/ &, https://www.openslr.org/51/
- **Data:** http://www.openslr.org/7/
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/tedlium
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add VCTK

Requesting to add a new dataset

@patrickvonplaten I'd like to take this, if nobody has already done it. I have added datasets before through the datasets sprint, but I feel rusty on the details, so I'll look at the guide as well as similar audio PRs (#1878 in particular comes to mind). If there is any detail I should be aware of please, let me know! Otherwise, I'll try to write up a PR in the coming days.

That sounds great @jaketae - let me know if you need any help i.e. feel free to ping me on a first PR :-)

## Adding a Dataset
- **Name:** *VCTK*
- **Description:** *This CSTR VCTK Corpus includes speech data uttered by 110 English speakers with various accents. Each speaker reads out about 400 sentences, which were selected from a newspaper, the rainbow passage and an elicitation paragraph used for the speech accent archive.*
- **Paper:** Homepage: https://datashare.ed.ac.uk/handle/10283/3443
- **Data:** https://datashare.ed.ac.uk/handle/10283/3443
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/vctk

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


test.json has been removed from the limit dataset repo (breaks dataset)

dataset bug

A bug in a dataset script provided in the library

Thanks for the heads up ! I'm opening a PR to fix that

https://github.com/huggingface/datasets/blob/16042b233dbff2a7585110134e969204c69322c3/datasets/limit/limit.py#L51

The URL is not valid anymore since test.json has been removed in master for some reason. Directly referencing the last commit works:

`https://raw.githubusercontent.com/ilmgut/limit_dataset/0707d3989cd8848f0f11527c77dcf168fefd2b23/data`


Add CHiME4 dataset

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** Chime4
- **Description:** Chime4 is a dataset for automatic speech recognition. It is especially useful for evaluating models in a noisy environment and for multi-channel ASR
- **Paper:** Dataset comes from a channel: http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/ . Results paper:
- **Data:** http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/download.html
- **Motivation:** So far there are very little datasets for speech in `datasets`. Only `lbirispeech_asr` so far.

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Fixes base_url of limit dataset

OK, apparently it is a lot more complicated than simply changing the URL? Going to make an issue.

`test.json` is not available in the master branch of the repository anymore. Linking to a specific commit.


Add OSCAR dataset card

@lhoestq Thanks for the suggestions! I agree with all of them. Should I accept them one by one or can I accept them all at once? When I try to load the whole diff GitHub is complaining and it does no render them well (probably my browser?) 😅

I just merged the tables as suggested 😄 . However I noticed something weird, the train sizes are identical for both the original and deduplicated files ... This is not normal, in general the original files are almost twice as big as the deduplicated ones 🤔

Good catch @pjox ! I just checked and this is because the scripts doesn't handle having several blank lines in a row.
Blank lines introduced by deduplication are currently not ignored so we end up with the same number of examples in the dataset as the original version (but with empty examples...)
I fixed that in this [commit](https://github.com/huggingface/datasets/commit/837a152e4724adc5308e2c4481908c00a8d93383). I'm re-running the metadata generation for deduplicated configs.

> I got the new sizes today, will update the dataset_infos.json and the dataset card tomorrow

great, I just wanted to report that I got error message "NonMatchingSplitsSizesError" when I tried to load one of the oscar dataset.

Hi @cahya-wirawan, which configuration of oscar do you have this issue with ?

Ok I see you're having this issue because I haven't updated the sizes yet ! I'm opening a PR

I just checked and indeed there's an issue with the `deduplicated` configurations since the commit I mentioned above.
I'm fixing this by using the new sizes I got yesterday :)

I just updated the size in the table @pjox it should be good now :)
I also updated the sizes in the dataset_infos.json in https://github.com/huggingface/datasets/pull/1868 (merged)

Thanks @lhoestq for fixing the issue, it works now

I added more information and completed the dataset card for OSCAR which was started by @lhoestq in his previous [PR](https://github.com/huggingface/datasets/pull/1824).


Looks like nokogumbo is up-to-date now, so this is no longer needed.

__Originally posted by @dependabot in https://github.com/discourse/discourse/pull/11373#issuecomment-738993432__


Some question about raw dataset download info in the project .

Hi ! The `dl_manager` is a `DownloadManager` object and is responsible for downloading the raw data files.
It is used by dataset builders in their `_split_generators` method to download the raw data files that are necessary to build the datasets splits.

The `Conll2003` class is a dataset builder, and so you can download all the raw data files by calling `_split_generators` with a download manager:
```python
from datasets import DownloadManager
from datasets.load import import_main_class

conll2003_builder = import_main_class(...)

dl_manager = DownloadManager()
splis_generators = conll2003_builder._split_generators(dl_manager)
```

Then you can see what files have been downloaded with
```python
dl_manager.get_recorded_sizes_checksums()
```
It returns a dictionary with the format {url: {num_bytes: int, checksum: str}}

Then you can get the actual location of the downloaded files with
```python
from datasets import cached_path

local_path_to_downloaded_file = cached_path(url)
```

------------------

Note that you can also get the urls from the Dataset object:
```python
from datasets import load_dataset

conll2003 = load_dataset("conll2003")
print(conll2003["train"].download_checksums)
```
It returns the same dictionary with the format {url: {num_bytes: int, checksum: str}}

I am afraid that there is not a very straightforward way to get that location.

Another option, from _split_generators would be to use:
- `dl_manager._download_config.cache_dir` to get the directory where all the raw downloaded files are:
  ```python
  download_dir = dl_manager._download_config.cache_dir
  ```
- the function `datasets.utils.file_utils.hash_url_to_filename` to get the filenames of the raw downloaded files:
  ```python
  filenames = [hash_url_to_filename(url) for url in urls_to_download.values()]
  ```
Therefore the complete path to the raw downloaded files would be the join of both:
```python
downloaded_paths = [os.path.join(download_dir, filename) for filename in filenames]
```

Maybe it would be interesting to make these paths accessible more easily. I could work on this. What do you think, @lhoestq ?

Sure it would be nice to have an easier access to these paths !
The dataset builder could have a method to return those, what do you think ?
Feel free to work on this @albertvillanova , it would be a nice addition :)

Your suggestion does work as well @albertvillanova if you complete it by specifying `etag=` to `hash_url_to_filename`.

The ETag is obtained by a HEAD request and is used to know if the file on the remote host has changed. Therefore if a file is updated on the remote host, then the hash returned by `hash_url_to_filename` is different.

Once #1846 will be merged, the paths to the raw downloaded files will be accessible as:
```python
builder_instance.dl_manager.downloaded_paths
```

Hi , i review the code in
https://github.com/huggingface/datasets/blob/master/datasets/conll2003/conll2003.py
in the _split_generators function is the truly logic of download raw datasets with dl_manager
and use Conll2003 cls by use import_main_class in load_dataset function
My question is that , with this logic it seems that i can not have the raw dataset download location
in variable in downloaded_files in _split_generators.
If someone also want use huggingface datasets as raw dataset downloader,
how can he retrieve the raw dataset download path from attributes in
datasets.dataset_dict.DatasetDict ?


using map on loaded Tokenizer 10x - 100x slower than default Tokenizer?

Hi @wumpusman
`datasets` has a caching mechanism that allows to cache the results of `.map` so that when you want to re-run it later it doesn't recompute it again.
So when you do `.map`, what actually happens is:
1. compute the hash used to identify your `map` for the cache
2. apply your function on every batch

This can explain the time difference between your different experiments.

The hash computation time depends of how complex your function is. For a tokenizer, the hash computation scans the lists of the words in the tokenizer to identify this tokenizer. Usually it takes 2-3 seconds.

Also note that you can disable caching though using
```python
import datasets

datasets.set_caching_enabled(False)
```

Hi @lhoestq ,

Thanks for the reply. It's entirely possible that is the issue. Since it's a side project I won't be looking at it till later this week, but, I'll verify it by disabling caching and hopefully I'll see the same runtime.

Appreciate the reference,

Michael

I believe this is an actual issue, tokenizing a ~4GB txt file went from an hour and a half to ~10 minutes when I switched from my pre-trained tokenizer(on the same dataset) to the default gpt2 tokenizer.
Both were loaded using:
```
AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
```
I trained the tokenizer using ByteLevelBPETokenizer from the Tokenizers library and save it to a tokenizer.json file.

I have tested the caching ideas above, changing the number of process, the TOKENIZERS_PARALLELISM env variable, keep_in_memory=True and batching with different sizes.

Apologies I can't really upload much code, but wanted to back up the finding and hopefully a fix/the problem can be found.
I will comment back if I find a fix as well.

Hi @johncookds do you think this can come from one tokenizer being faster than the other one ? Can you try to compare their speed without using `datasets` just to make sure ?

Hi yes, I'm closing the loop here with some timings below. The issue seems to be at least somewhat/mainly with the tokenizer's themselves. Moreover legacy saves of the trainer tokenizer perform faster but differently than the new tokenizer.json saves(note nothing about the training process/adding of special tokens changed between the top two trained tokenizer tests, only the way it was saved). This is only a 3x slowdown vs like a 10x but I think the slowdown is most likely due to this.

```
trained tokenizer - tokenizer.json save (same results for AutoTokenizer legacy_format=False):
Tokenizer time(seconds): 0.32767510414123535
Tokenized avg. length: 323.01

trained tokenizer - AutoTokenizer legacy_format=True:
Tokenizer time(seconds): 0.09258866310119629
Tokenized avg. length: 301.01

GPT2 Tokenizer from huggingface
Tokenizer time(seconds): 0.1010282039642334
Tokenized avg. length: 461.21
```

@lhoestq ,

Hi, which version of datasets has datasets.set_caching_enabled(False)? I get
module 'datasets' has no attribute 'set_caching_enabled'. To hopefully get around this, I reran my code on a new set of data, and did so only once.

@johncookds , thanks for chiming in, it looks this might be an issue of Tokenizer.

**Tokenizer**: The runtime of GPT2TokenizerFast.from_pretrained("gpt2") on 1000 chars is: **143 ms**
**SlowTokenizer**: The runtime of a locally saved and loaded Tokenizer using the same vocab  on 1000 chars is:  **4.43 s**

That being said, I compared performance on the map function:

Running Tokenizer versus using it in the map function for 1000 chars goes from **141 ms** to **356 ms**
Running SlowTokenizer versus using it in the map function for 1000 chars with a single element goes from **4.43 s** to **9.76 s**

I'm trying to figure out why the overhead of map would increase the time by double (figured it would be a fixed increase in time)? Though maybe this is expected behavior.

@lhoestq, do you by chance know how I can redirect this issue to Tokenizer?

Regards,

Michael

Thanks for the experiments @johncookds and @wumpusman !

> Hi, which version of datasets has datasets.set_caching_enabled(False)?

Currently you have to install `datasets` from source to have this feature, but this will be available in the next release in a few days.

> I'm trying to figure out why the overhead of map would increase the time by double (figured it would be a fixed increase in time)? Though maybe this is expected behavior.

Could you also try with double the number of characters ? This should let us have an idea of the fixed cost (hashing) and the dynamic cost (actual tokenization, grows with the size of the input)

> @lhoestq, do you by chance know how I can redirect this issue to Tokenizer?

Feel free to post an issue on the `transformers` repo. Also I'm sure there should be related issues so you can also look for someone with the same concerns on the `transformers` repo.

@lhoestq,

I just checked that previous run time was actually  3000 chars. I increased it to 6k chars, again, roughly double.

SlowTokenizer **7.4 s** to **15.7 s**
Tokenizer: **276 ms** to **616 ms**

I'll post this issue on Tokenizer, seems it hasn't quite been raised (albeit I noticed a similar issue that might relate).

Regards,

Michael

Hi,
I'm following up here as I found my exact issue. It was with saving and re-loading the tokenizer. When I trained then processed the data without saving and reloading it, it was 10x-100x faster than when I saved and re-loaded it.
Both resulted in the exact same tokenized datasets as well.
There is additionally a bug where the older legacy tokenizer save does not preserve a learned tokenizing behavior if trained from scratch.
Understand its not exactly Datasets related but hope it can help someone if they have the same issue.
Thanks!

This could total relate to me misunderstanding particular call functions, but I added words to a GPT2Tokenizer, and saved it to disk (note I'm only showing snippets but I can share more) and the map function ran much slower:

````
def save_tokenizer(original_tokenizer,text,path="simpledata/tokenizer"):
    words_unique = set(text.split(" "))
    for i in words_unique:
        original_tokenizer.add_tokens(i)
    original_tokenizer.save_pretrained(path)

tokenizer2 = GPT2Tokenizer.from_pretrained(os.path.join(experiment_path,experiment_name,"tokenizer_squad"))

train_set_baby=Dataset.from_dict({"text":[train_set["text"][0][0:50]]})
````

I then applied the dataset map function on a fairly small set of text:

```
%%time
train_set_baby = train_set_baby.map(lambda d:tokenizer2(d["text"]),batched=True)

```

The run time for train_set_baby.map was 6 seconds, and the batch itself was 2.6 seconds

**100% 1/1 [00:02<00:00, 2.60s/ba] CPU times: user 5.96 s, sys: 36 ms, total: 5.99 s Wall time: 5.99 s**

In comparison using (even after adding additional tokens):
`
tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")`

```
%%time
train_set_baby = train_set_baby.map(lambda d:tokenizer2(d["text"]),batched=True)

```
The time is
**100% 1/1 [00:00<00:00, 34.09ba/s] CPU times: user 68.1 ms, sys: 16 µs, total: 68.1 ms Wall time: 62.9 ms**

It seems this might relate to the tokenizer save or load function, however, the issue appears to come up when I apply the loaded tokenizer  to the map function.

I should also add that playing around with the amount of words I add to the tokenizer before I save it to disk and load it into memory  appears to impact the time it takes to run the map function.


Add Tweet Eval Dataset

Closes Draft PR #1407.

Notes:
1. I have excluded `mapping.txt` from the dataset at it only contained the name mappings, which are already present in the ClassLabels.
2. I have also exluded the textual names for the emojis mentioned in the [mapping](https://github.com/cardiffnlp/tweeteval/blob/main/datasets/emoji/mapping.txt).
3. I do not understand @abhishekkrthakur's example generator on #1407. Maybe he was trying to build up on code from some other dataset.

Requesting @lhoestq  to review.


Add CelebA Dataset

Hi @gchhablani! Thanks for all the contributions! We definitely want more image datasets, but Face datasets are tricky in general, in this one includes predicting attributes such as Attractiveness, Gender, or Race, which can be pretty problematic.

Would you be up for starting with only object classification or object detection datasets instead? (Your CIFAR-100 contribution will be super useful for example!)

Hi @yjernite, You're welcome. I am enjoying adding new datasets :)
By "pretty problematic", are you referring to the ethical issues? I used TFDS's [CelebA](https://github.com/tensorflow/datasets/blob/5ef7861470896acb6f74dacba85036001e4f1b8c/tensorflow_datasets/image/celeba.py#L91) as a reference. Here they mention in a "Note" that CelebA  "may contain potential bias". Can we not do the same? I skipped the note for now, and we can add it. However, if you feel this isn't the right time, then I won't pursue this further.

But, can this issue be handled at a later stage? Does this also apply for my Hateful Memes Issue #1810?

Also, how can I
1. load a part of the dataset? since `load_dataset(<>,split='train[10:20]')` still loads all the examples.
2. make `datasets_infos.json` for huge datasets which have a single configuration?

I will ofcourse be looking for other datasets to add regardless.

It's definitely a thorny question. The short answer is: Hateful Memes and hate speech detection datasets are different since their use case is specifically to train systems to identify and hopefully remove hateful content, whereas the purpose of a dataset that has an Attractiveness score as output is implicitly to train more models to rate "Attractiveness".

As far as warning about the "potential biases", I do not think it is quite enough, especially because it is hard to guarantee that every potential user will read the documentation (it is also an insufficient warning.)

Note that we do have higher standards for the dataset cards of hate speech and hateful memes datasets, so if you do choose to add that one yourself we will ask that you summarize the relevant literature in the Social Impact section.

If you really need to add this dataset for your own research for the explicit purpose of studying these biases, you can add it as a community provided dataset following https://huggingface.co/docs/datasets/master/share_dataset.html#sharing-a-community-provided-dataset but I'd recommend just skipping it for now.

So currently you do need to download the whole dataset when using it, we are working on making it easier to stream parts of it from a remote host. You can also use the filesystem integration if local storage is an issue:
https://huggingface.co/docs/datasets/master/filesystems.html

I don't think we have a great solution for `dataset_infos.json` with a single very large config when storage space is an issue, but it should be solved by the same upcoming feature mentioned above

Okay, then I won't pursue this one further. I'll keep this branch on my repository just in case the possibility of adding this dataset comes up in the future.

> So currently you do need to download the whole dataset when using it, we are working on making it easier to stream parts of it from a remote host. You can also use the filesystem integration if local storage is an issue:
> https://huggingface.co/docs/datasets/master/filesystems.html

After downloading the whole dataset (around 1.4GB), it still loads all the examples despite using `split='train[:10%]'` or `split='train[10:20]'`.

EDIT: I think this would happen only when the examples are generated for the first time and saved to the cache. Streaming parts of the data from a remote host sounds amazing! But, would that also allow for streaming examples of the data from the local cache? (without saving all the examples the first time).

What I used:
`d = load_dataset('./datasets/celeb_a',split='train[:10]')`
Output:
`570 examples [01:33,  6.25 examples/s]` and it keeps going.

EDIT 2: After a few thousand images, I get the following error:
```python
OSError: [Errno 24] Too many open files: '~/.cache/huggingface/datasets/celeb_a/default/1.1.0/01f9dca66039ab7c40b91b09af47a5fa8c3e49dc8d55df50da55b14116229207.incomplete'
```
I understand this is because of the way I load the images :
```python
Image.open(<path>)
```
What could be better alternative? I am only asking in case I face the same issues in the future.

Just some addition about loading only a subset of the data:
Currently if even you specify `split='train[:10]'`, it downloads and generate the full dataset, so that you can pick another part afterward if you want to. We may change that in the future and use streaming.

And about your open files issue, you can try to close each image file after reading its content.

Hi @lhoestq,
Thanks for your response.

I used `gc.collect()` inside the loop and that worked for me. I think since we are using a generator, and if I have something like `train[100000:100002]`, we will need to generate the first 1000001 examples and store. Ofcourse, this feature isn't a necessity right now, I suppose.

Trying to add CelebA Dataset.
Need help with testing. Loading examples takes a lot of time so I am unable to generate the `dataset_infos.json` and unable to test. Also, need help with creating `dummy_data.zip`.

Additionally, trying to load a few examples using `load_dataset('./datasets/celeb_a',split='train[10:20]')` still loads all the examples (doesn't stop at 10).


Regarding On-the-fly Data Loading

Possible duplicate

#1776 https://github.com/huggingface/datasets/issues/

really looking PR for this feature

Hi @acul3

Issue #1776 talks about  doing on-the-fly data pre-processing, which I think is solved in the next release as mentioned in the issue #1825. I also look forward to using this feature, though :)

I wanted to ask about on-the-fly data loading from the cache (before pre-processing).

Hi ! Currently when you load a dataset via `load_dataset` for example, then the dataset is memory-mapped from an Arrow file on disk. Therefore there's almost no RAM usage even if your dataset contains TB of data.
Usually at training time only one batch of data at a time is loaded in memory.

Does that answer your question or were you thinking about something else ?

Hi @lhoestq,

I apologize for the late response. This answers my question. Thanks a lot.

Hi,

I was wondering if it is possible to load images/texts as a batch during the training process, without loading the entire dataset on the RAM at any given point.

Thanks,
Gunjan


Print error message with filename when malformed CSV

Print error message specifying filename when malformed CSV file.

Close #1821


Datasets library not suitable for huge text datasets.

Hi ! Looks related to #861

You are right: tokenizing a dataset using map takes a lot of space since it can store `input_ids` but also `token_type_ids`, `attention_mask` and `special_tokens_mask`. Moreover if your tokenization function returns python integers then by default they'll be stored as int64 which can take a lot of space. Padding can also increase the size of the tokenized dataset.

To make things more convenient, we recently added a "lazy map" feature that allows to tokenize each batch at training time as you mentioned. For example you'll be able to do
```python
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

def encode(batch):
    return tokenizer(batch["text"], padding="longest", truncation=True, max_length=512, return_tensors="pt")

dataset.set_transform(encode)
print(dataset.format)
# {'type': 'custom', 'format_kwargs': {'transform': <function __main__.encode(batch)>}, 'columns': ['idx', 'label', 'sentence1', 'sentence2'], 'output_all_columns': False}
print(dataset[:2])
# {'input_ids': tensor([[  101,  2572,  3217, ... 102]]), 'token_type_ids': tensor([[0, 0, 0, ... 0]]), 'attention_mask': tensor([[1, 1, 1, ... 1]])}

```
In this example the `encode` transform is applied on-the-fly on the "text" column.

This feature will be available in the next release 2.0 which will happen in a few days.
You can already play with it by installing `datasets` from source if you want :)

Hope that helps !

How recently was `set_transform` added? I am actually trying to implement it and getting an error:

`AttributeError: 'Dataset' object has no attribute 'set_transform'
`

I'm on v.1.2.1.

EDIT: Oh, wait I see now it's in the v.2.0. Whoops! This should be really useful.

Yes indeed it was added a few days ago. The code is available on master
We'll do a release next week :)

Feel free to install `datasets` from source to try it out though, I would love to have some feedbacks

For information: it's now available in `datasets` 1.3.0.
The 2.0 is reserved for even cooler features ;)

Hi @alexvaca0 , we have optimized Datasets' disk usage in the latest release v1.5.

Feel free to update your Datasets version
```shell
pip install -U datasets
```
and see if it better suits your needs.

Hi,

I'm trying to use datasets library to load a 187GB dataset of pure text, with the intention of building a Language Model. The problem is that from the 187GB it goes to some TB when processed by Datasets. First of all, I think the pre-tokenizing step (with tokenizer.map()) is not really thought for datasets this big, but for fine-tuning datasets, as this process alone takes so much time, usually in expensive machines (due to the need of tpus - gpus) which is not being used for training. It would possibly be more efficient in such cases to tokenize each batch at training time (receive batch - tokenize batch - train with batch), so that the whole time the machine is up it's being used for training.
Moreover, the pyarrow objects created from a 187 GB datasets are huge, I mean, we always receive OOM, or No Space left on device errors when only 10-12% of the dataset has been processed, and only that part occupies 2.1TB in disk, which is so many times the disk  usage of the pure text (and this doesn't make sense, as tokenized texts should be lighter than pure texts).

Any suggestions??


Add OSCAR dataset card

Hi @lhoestq! When are you planning to release the version with this dataset?

BTW: What a huge README file :astonished:

I started adding the dataset card for OSCAR !

For now it's just basic info for all the different configurations in `Dataset Structure`.
In particular the Data Splits section tells how may samples there are for each config. The Data Instances section show an example for each config, and it also shows the size in MB. Since the Data Instances section is very long the user has to click to expand the info. I was able to generate it thanks to the tools made by  @madlag and @yjernite :D

Cc @pjox could you help me with the other sections ? (Dataset Description, Dataset Creation, Considerations for Using the Data, Additional Information)


Add FewRel Dataset

Hi @lhoestq,

Sorry for the late response. What do you mean when you say "adding names to default config"? Should I handle "pid2name" in the same config as "default"?

Yes I was thinking of having the pid2name field available in the default configuration (and therefore only have one config). What do you think ?

Hi @lhoestq,

Sorry again, the last couple of weeks were a bit busy for me. I am wondering how do you want me to achieve that. Using a custom BuilderConfig which takes in whether it is the regular data or "pid2name"? "pid2name" is only useful for "train_wiki", "val_nyt" and "val_wiki". So, based on my understanding, it would look like this:

```python
wiki_data = load_dataset('few_rel','train_wiki')
id2name = load_dataset('few_rel','pid2name')
```
and this will be handled in the multiple configs.

A better alternative could be providing name of the relationship in only "train_wiki", "val_nyt" and "val_wiki" as an extra feature in the dataset, and doing away with "pid2name" entirely. I'll only download pid2name if any of those datasets are requested, and then during generation I'll return the list with the dataset under "names" feature. How does this sound?

EDIT:
There is one issue with the second approach, the entire pid2name is saved with all three datasets - "train_wiki", "val_nyt" and "val_wiki" ([see code below](https://github.com/huggingface/datasets/pull/1823#issuecomment-786402026)). In dummy data, I can address this by manually editing the pid2name to contain only a few id-name pairs, those matching with the examples in the corresponding example file. But this seems to be inefficient for the entire dataset - storing the same file in multiple places.

Okay, I apologize, I guess I finally understand what is required.

Basically, using:

```python
few_rel = load_dataset('few_rel')
```
should give all the files. This seems difficult since "pid2name" has a different format. Any suggestions on this?

Yes that's it, sorry if that wasn't clear !

Hi @lhoestq,

Since pid2name has different features from the rest of the files, how will I add them to the same config?

Do we want to exclude pid2name totally and add "names" to every example?

If I understand correctly each sample in the "default" config has one relation, and each relation has corresponding names in pid2name.
Would it be possible to also include the names in the "default" configuration for each sample ? The names of one sample can be retrieved using the relation id no ?

Yes, that can be done. But for some files, the name is already given instead of ID. Only "train_wiki", "val_wiki", "val_nyc" have IDs. For others, I can set the names equal to a list of key.

I think that's fine as long as we mention this processing explicitly in the dataset card.

Hi @lhoestq,

I have added the changes. Please let me know in case of any remaining issues.

Thanks,
Gunjan

Hi @lhoestq,

Thanks for fixing it and approving :)

Hi,

This PR closes this [Card](https://github.com/huggingface/datasets/projects/1#card-53285184) and Issue #1757.

I wasn't sure how to add `pid2name` along with the dataset so I added it as a separate configuration. For each (head, tail, tokens) triplet, I have created one example. I have added the dictionary key as `"relation"` in the dataset. Additionally, for `pubmed_unsupervised`, I kept `"relation":""` in the dictionary.

Please recommend better alternatives, if any.

Thanks,
Gunjan


Add Hindi Discourse Analysis Natural Language Inference Dataset

Could you also run `make style` to fix the CI check on code formatting ?

@lhoestq completed and resolved all comments.

# Dataset Card for Hindi Discourse Analysis Dataset

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-fields)
  - [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)
  - [Contributions](#contributions)

## Dataset Description

- HomePage : https://github.com/midas-research/hindi-nli-data
- Paper : https://www.aclweb.org/anthology/2020.aacl-main.71
- Point of Contact : https://github.com/midas-research/hindi-nli-data

### Dataset Summary

- Dataset for Natural Language Inference in Hindi Language. Hindi Discourse Analysis (HDA) Dataset consists of textual-entailment pairs.
- Each row of the Datasets if made up of 4 columns - Premise, Hypothesis, Label and Topic.
- Premise and Hypothesis is written in Hindi while Entailment_Label is in English.
- Entailment_label is of 2 types - entailed and not-entailed.
- Entailed means that hypotheis can be inferred from premise and not-entailed means vice versa
- Dataset can be used to train models for Natural Language Inference tasks in Hindi Language.

### Supported Tasks and Leaderboards

- Natural Language Inference for Hindi

### Languages

- Dataset is in Hindi

## Dataset Structure

- Data is structured in TSV format.
- train, test and dev files are in seperate files

### Dataset Instances

An example of 'train' looks as follows.

```
{'hypothesis': 'यह एक वर्णनात्मक कथन है।', 'label': 1, 'premise': 'जैसे उस का सारा चेहरा अपना हो और आँखें किसी दूसरे की जो चेहरे पर पपोटों के पीछे महसूर कर दी गईं।', 'topic': 1}

```
### Data Fields

- Each row contatins 4 columns - premise, hypothesis, label and topic.

### Data Splits

- Train : 31892
- Valid : 9460
- Test : 9970

## Dataset Creation

- We employ a recasting technique from Poliak et al. (2018a,b) to convert publicly available Hindi Discourse Analysis classification datasets in Hindi and pose them as TE problems
- In this recasting process, we build template hypotheses for each class in the label taxonomy
- Then, we pair the original annotated sentence with each of the template hypotheses to create TE samples.
- For more information on the recasting process, refer to paper https://www.aclweb.org/anthology/2020.aacl-main.71

### Source Data

Source Dataset for the recasting process is the BBC Hindi Headlines Dataset(https://github.com/NirantK/hindi2vec/releases/tag/bbc-hindi-v0.1)

#### Initial Data Collection and Normalization

-  Initial Data was collected by members of MIDAS Lab from Hindi Websites. They crowd sourced the data annotation process and selected two random stories from our corpus and had the three annotators work on them independently and classify each sentence based on the discourse mode.
- Please refer to this paper for detailed information: https://www.aclweb.org/anthology/2020.lrec-1.149/
- The Discourse is further classified into "Argumentative" , "Descriptive" , "Dialogic" , "Informative"  and  "Narrative" - 5 Clases.

#### Who are the source language producers?

Please refer to this paper for detailed information: https://www.aclweb.org/anthology/2020.lrec-1.149/

### Annotations

#### Annotation process

Annotation process has been described in Dataset Creation Section.

#### Who are the annotators?

Annotation is done automatically by machine and corresponding recasting process.

### Personal and Sensitive Information

No Personal and Sensitive Information is mentioned in the Datasets.

## Considerations for Using the Data

Pls refer to this paper: https://www.aclweb.org/anthology/2020.aacl-main.71

### Discussion of Biases

No known bias exist in the dataset.
Pls refer to this paper: https://www.aclweb.org/anthology/2020.aacl-main.71

### Other Known Limitations

No other known limitations . Size of data may not be enough to train large models

## Additional Information

Pls refer to this link: https://github.com/midas-research/hindi-nli-data

### Dataset Curators

It is written in the repo : https://github.com/midas-research/hindi-nli-data that
- This corpus can be used freely for research purposes.
- The paper listed below provide details of the creation and use of the corpus. If you use the corpus, then please cite the paper.
- If interested in commercial use of the corpus, send email to midas@iiitd.ac.in.
- If you use the corpus in a product or application, then please credit the authors and Multimodal Digital Media Analysis Lab - Indraprastha Institute of Information Technology, New Delhi appropriately. Also, if you send us an email, we will be thrilled to know about how you have used the corpus.
- Multimodal Digital Media Analysis Lab - Indraprastha Institute of Information Technology, New Delhi, India disclaims any responsibility for the use of the corpus and does not provide technical support. However, the contact listed above will be happy to respond to queries and clarifications.
- Rather than redistributing the corpus, please direct interested parties to this page
- Please feel free to send us an email:
  - with feedback regarding the corpus.
  - with information on how you have used the corpus.
  - if interested in having us analyze your data for natural language inference.
  - if interested in a collaborative research project.

### Licensing Information

Copyright (C) 2019 Multimodal Digital Media Analysis Lab - Indraprastha Institute of Information Technology, New Delhi (MIDAS, IIIT-Delhi).
Pls contact authors for any information on the dataset.

### Citation Information

```
    @inproceedings{uppal-etal-2020-two,
    title = "Two-Step Classification using Recasted Data for Low Resource Settings",
    author = "Uppal, Shagun  and
      Gupta, Vivek  and
      Swaminathan, Avinash  and
      Zhang, Haimin  and
      Mahata, Debanjan  and
      Gosangi, Rakesh  and
      Shah, Rajiv Ratn  and
      Stent, Amanda",
    booktitle = "Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing",
    month = dec,
    year = "2020",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.aacl-main.71",
    pages = "706--719",
    abstract = "An NLP model{'}s ability to reason should be independent of language. Previous works utilize Natural Language Inference (NLI) to understand the reasoning ability of models, mostly focusing on high resource languages like English. To address scarcity of data in low-resource languages such as Hindi, we use data recasting to create NLI datasets for four existing text classification datasets. Through experiments, we show that our recasted dataset is devoid of statistical irregularities and spurious patterns. We further study the consistency in predictions of the textual entailment models and propose a consistency regulariser to remove pairwise-inconsistencies in predictions. We propose a novel two-step classification method which uses textual-entailment predictions for classification task. We further improve the performance by using a joint-objective for classification and textual entailment. We therefore highlight the benefits of data recasting and improvements on classification performance using our approach with supporting experimental results.",
}
```

### Contributions

Thanks to [@avinsit123](https://github.com/avinsit123) for adding this dataset.


Provide better exception message when one of many files results in an exception

Hi!

Thank you for reporting this issue. I agree that the information about the exception should be more clear and explicit.

I could take on this issue.

On the meantime, as you can see from the exception stack trace, HF Datasets uses pandas to read the CSV files. You can pass arguments to `pandas.read_csv` by passing additional keyword arguments to `load_dataset`. For example, you may find useful this argument:
- `error_bad_lines` : bool, default True
  Lines with too many fields (e.g. a csv line with too many commas) will by default cause an exception to be raised, and no DataFrame will be returned. If False, then these “bad lines” will be dropped from the DataFrame that is returned.

You could try:
```python
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files), error_bad_lines=False)
```

I find when I process many files, i.e.

```
train_files = glob.glob('rain*.csv')
validation_files = glob.glob(validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))
```

I sometimes encounter an error due to one of the files being misformed (i.e. no data, or a comma in a field that isn't quoted, etc).

For example, this is the tail of an exception which I suspect is due to a stray comma.

>   File "pandas/_libs/parsers.pyx", line 756, in pandas._libs.parsers.TextReader.read
>   File "pandas/_libs/parsers.pyx", line 783, in pandas._libs.parsers.TextReader._read_low_memory
>   File "pandas/_libs/parsers.pyx", line 827, in pandas._libs.parsers.TextReader._read_rows
>   File "pandas/_libs/parsers.pyx", line 814, in pandas._libs.parsers.TextReader._tokenize_rows
>   File "pandas/_libs/parsers.pyx", line 1951, in pandas._libs.parsers.raise_parser_error
> pandas.errors.ParserError: Error tokenizing data. C error: Expected 2 fields in line 559, saw 3

It would be nice if the exception trace contained the name of the file being processed (I have 250 separate files!)


Add metrics usage examples and tests

All metrics finally have usage examples and proper fast + slow tests :)

I added examples of usage for every metric, and I use doctest to make sure they all work as expected.

For "slow" metrics such as bert_score or bleurt which require to download + run a transformer model, the download + forward pass are only done in the slow test.
In the fast test on the other hand, the download + forward pass are monkey patched.

Metrics that need to be installed from github are not added to setup.py because it prevents uploading the `datasets` package to pypi.
An additional-test-requirements.txt file is used instead. This file also include `comet` in order to not have to resolve its *impossible* dependencies.

Also `comet` is not tested on windows because one of its dependencies (fairseq) can't be installed in the CI for some reason.


Fixed spelling `S3Fileystem` to `S3FileSystem`

Fixed documentation spelling errors.
Wrong `S3Fileystem`
Right `S3FileSystem`


Loading local dataset raise requests.exceptions.ConnectTimeout

Hi ! Thanks for reporting. This was indeed a bug introduced when we moved the `json` dataset loader inside the `datasets` package (before that, the `json` loader was fetched online, as all the other dataset scripts).

This should be fixed on master now. Feel free to install `datasets` from source to try it out.
The fix will be available in the next release of `datasets` in a few days

Load local dataset:
```
dataset = load_dataset('json', data_files=["../../data/json.json"])
train = dataset["train"]
print(train.features)
train1 = train.map(lambda x: {"labels": 1})
print(train1[:2])
```

but it raised requests.exceptions.ConnectTimeout:

```
/Users/littlely/myvirtual/tf2/bin/python3.7 /Users/littlely/projects/python_projects/pytorch_learning/nlp/dataset/transformers_datasets.py
Traceback (most recent call last):
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
    conn.connect()
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/urllib3/connection.py", line 309, in connect
    conn = self._new_conn()
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/urllib3/connection.py", line 167, in _new_conn
    % (self.host, self.timeout),
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPSConnection object at 0x1181e9940>, 'Connection to s3.amazonaws.com timed out. (connect timeout=10)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/datasets/datasets/json/json.py (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x1181e9940>, 'Connection to s3.amazonaws.com timed out. (connect timeout=10)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/littlely/projects/python_projects/pytorch_learning/nlp/dataset/transformers_datasets.py", line 12, in <module>
    dataset = load_dataset('json', data_files=["../../data/json.json"])
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/datasets/load.py", line 591, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/datasets/load.py", line 263, in prepare_module
    head_hf_s3(path, filename=name, dataset=dataset, max_retries=download_config.max_retries)
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 232, in head_hf_s3
    max_retries=max_retries,
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 523, in http_head
    max_retries=max_retries,
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 458, in _request_with_retry
    raise err
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 454, in _request_with_retry
    response = requests.request(verb.upper(), url, **params)
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/Users/littlely/myvirtual/tf2/lib/python3.7/site-packages/requests/adapters.py", line 504, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/datasets/datasets/json/json.py (Caused by ConnectTimeoutError(<urllib3.connection.HTTPSConnection object at 0x1181e9940>, 'Connection to s3.amazonaws.com timed out. (connect timeout=10)'))

Process finished with exit code 1

```

Why it want to connect a remote url when I load local datasets, and how can I fix it?


pyarrow.lib.ArrowInvalid: Column 1 named input_ids expected length 599 but got length 1500

Hi !
The error you have is due to the `input_ids` column not having the same number of examples as the other columns.
Indeed you're concatenating the `input_ids` at this line:

https://github.com/LuCeHe/GenericTools/blob/431835d8e13ec24dceb5ee4dc4ae58f0e873b091/KerasTools/lm_preprocessing.py#L134

However the other columns are kept unchanged, and therefore you end up with an `input_ids` column with 599 elements while the others columns like `attention_mask` have 1500.

To fix that you can instead concatenate them all using
```python
concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
```

Also you may need to drop the "text" column before applying `group_texts` since strings can't be concatenated with lists. You can drop it at the tokenization step:
```python
dset = dset.map(
    tokenize_function,
    batched=True,
    remove_columns=["text"]
)
```

I am trying to preprocess any dataset in this package with GPT-2 tokenizer, so I need to structure the datasets as long sequences of text without padding. I've been following a couple of your tutorials and here you can find the script that is failing right at the end

https://github.com/LuCeHe/GenericTools/blob/master/KerasTools/lm_preprocessing.py

In the last iteration of the last dset.map, it gives the error that I copied in the title. Another issue that I have, if I leave the batch_size set as 1000 in the last .map, I'm afraid it's going to lose most text, so I'm considering setting both writer_batch_size and batch_size to 300 K, but I'm not sure it's the best way to go.

Can you help me?
Thanks!


Doc2dial rc update to latest version

- update data loader and readme for latest version 1.0.1


Add CCAligned Multilingual Dataset

Hi !

We already have some datasets that can have many many configurations possible.
To be able to support that, we allow to subclass BuilderConfig to add as many additional parameters as you may need.
This way users can load any language they want. For example the [bible_para](https://github.com/huggingface/datasets/blob/master/datasets/bible_para/bible_para.py) dataset is a dataset for translation and therefore users should be able to provide any language pair. You can check how the subclass of BuilderConfig is defined [here](https://github.com/huggingface/datasets/blob/master/datasets/bible_para/bible_para.py#L49).

For testing, only the configurations defined in the `BUILDER_CONFIGS` class attribute are used.
All the other configs combinations are not tested, but they can be used by users. If a config doesn't already exist in `BUILDER_CONFIGS`, then it is created on the fly.
For example in [bible_para](https://github.com/huggingface/datasets/blob/master/datasets/bible_para/bible_para.py#L61), only 6 configs are defined in `BUILDER_CONFIGS`.

So what I would do in your case is have something like
```python

class CCAlignedConfig(datasets.BuilderConfig):
    def __init__(self, *args, documents_or_sentences=None, language_code=None, **kwargs):
        super().__init__(
            *args,
            name=f"{documents_or_sentences}-{language_code}",
            **kwargs,
        )
        self.documents_or_sentences = documents_or_sentences
        self.language_code = language_code
```
And of course, feel free to change/rename things if you want to. In particular I think we can improve the name of the parameter `documents_or_sentences`

Hi @lhoestq,

Thanks a lot! I don't know why I didn't think about that. :P
I'll make these changes and update.

Hi @lhoestq,

I have tested and added dummy files. Request you to review.

Also, does this mean BUILDER_CONFIGS is only needed while testing?

Hi @lhoestq,

Any changes required on this one?

Thanks,
Gunjan

Hi @lhoestq,

Sorry for the delay, I have added the changes from the review. For the ISO format language codes, I just selected the first two characters from the names, hoping those are correct. Let me know if you want me to verify :P

Thanks for taking the time to add such a detailed review. I'll keep all these changes in mind the next time I'm adding a dataset.

Thanks,
Gunjan

Hi @lhoestq,

I have changed the README, and added a single example per config. Even one example is long enough to make the files heavy. Hope that isn't an issue.

Thanks,
Gunjan

Hello,

I'm trying to add [CCAligned Multilingual Dataset](http://www.statmt.org/cc-aligned/). This has the potential to close #1756.

This dataset has two types - Document-Pairs, and Sentence-Pairs.

The datasets are huge, so I won't be able to test all of them. At the same time, a user might only want to download one particular language and not all. To provide this feature, `load_dataset`'s `**config_kwargs` should allow some random keyword args, in this case -`language_code`. This will be needed before the dataset is downloaded and extracted.

I'm expecting the usage to be something like -
`load_dataset('ccaligned_multilingual','documents',language_code='en_XX-af_ZA')`. Ofcourse, at a later stage we can provide just two character language codes. This also has an issue where one language has multiple files (`my_MM` and `my_MM_zaw` on the link), but before that the required functionality must be added to `load_dataset`.

It would be great if someone could either tell me an alternative way to do this, or point me to where changes need to be made, if any, apart from the `BuilderConfig` definition.

Additionally, I believe the tests will also have to be modified if this change is made, since it would not be possible to test for any random keyword arguments.

A decent way to go about this would be to provide all the options in a list/dictionary for `language_code` and use that to test the arguments. In essence, this is similar to the pre-trained checkpoint dictionary as `transformers`. That means writing dataset specific tests, or adding something new to dataset generation script to make it easier for everyone to add keyword arguments without having to worry about the tests.

Thanks,
Gunjan

Requesting @lhoestq / @yjernite to review.