rIsHu009/github-issues-updated huggingface dataset:
Title: Fix Blog Authorship Corpus dataset
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-07-20T15:44:50Z

This PR:
- Update the JSON metadata file, which previously was raising a `NonMatchingSplitsSizesError`
- Fix the codec of the data files (`latin_1` instead of `utf-8`), which previously was raising ` UnicodeDecodeError` for some files

Close #2679.

Comments:
Normally, I'm expecting errors from the validation of the README file... 😅 

That is:
```
=========================== short test summary info ============================
FAILED tests/test_dataset_cards.py::test_changed_dataset_card[blog_authorship_corpus]
==== 1 failed, 3182 passed, 2763 skipped, 16 warnings in 201.23s (0:03:21) =====
```

@lhoestq, apart from the dataset card, everything is OK with this PR: I tested it locally.
Title: Cache directories changed due to recent changes in how config kwargs are handled
Author: lhoestq (MEMBER)
State: closed
Created: 2021-07-20T14:37:57Z

Since #2659 I can see weird cache directory names with hashes in the config id, even though no additional config kwargs are passed. For example:

```python
from datasets import load_dataset_builder

c4_builder = load_dataset_builder("c4", "en")
print(c4_builder.cache_dir)
# /Users/quentinlhoest/.cache/huggingface/datasets/c4/en-174d3b7155eb68db/0.0.0/...

# instead of 
# /Users/quentinlhoest/.cache/huggingface/datasets/c4/en/0.0.0/...
```
This issue could be annoying since it would simply ignore old cache directories for users, and regenerate datasets

cc @stas00 this is what you experienced a few days ago

Comments:
Title: 5 duplicate datasets
Author: severo (COLLABORATOR)
State: closed
Created: 2021-07-20T14:25:00Z
Labels: bug

## Describe the bug

In 5 cases, I could find a dataset on Paperswithcode which references two Hugging Face datasets as dataset loaders. They are:

- https://paperswithcode.com/dataset/multinli -> https://huggingface.co/datasets/multi_nli and https://huggingface.co/datasets/multi_nli_mismatch
  
  <img width="838" alt="Capture d’écran 2021-07-20 à 16 33 58" src="https://user-images.githubusercontent.com/1676121/126342757-4625522a-f788-41a3-bd1f-2a8b9817bbf5.png">

- https://paperswithcode.com/dataset/squad -> https://huggingface.co/datasets/squad and https://huggingface.co/datasets/squad_v2
- https://paperswithcode.com/dataset/narrativeqa -> https://huggingface.co/datasets/narrativeqa and https://huggingface.co/datasets/narrativeqa_manual
- https://paperswithcode.com/dataset/hate-speech-and-offensive-language -> https://huggingface.co/datasets/hate_offensive and https://huggingface.co/datasets/hate_speech_offensive
- https://paperswithcode.com/dataset/newsph-nli -> https://huggingface.co/datasets/newsph and https://huggingface.co/datasets/newsph_nli

Possible solutions:
- don't fix (it works)
- for each pair of duplicate datasets, remove one, and create an alias to the other.

## Steps to reproduce the bug

Visit the Paperswithcode links, and look at the "Dataset Loaders" section

## Expected results

There should only be one reference to a Hugging Face dataset loader

## Actual results

Two Hugging Face dataset loaders

Comments:
Yes this was documented in the PR that added this hf->paperswithcode mapping (https://github.com/huggingface/datasets/pull/2404) and AFAICT those are slightly distinct datasets so I think it's a wontfix

For context on the paperswithcode mapping you can also refer to https://github.com/huggingface/huggingface_hub/pull/43 which contains a lot of background discussion 

Thanks for the antecedents. I close.
Title: Error when downloading C4
Author: Aktsvigun (NONE)
State: closed
Created: 2021-07-20T08:37:30Z
Labels: bug

Hi,
I am trying to download `en` corpus from C4 dataset. However, I get an error caused by validation files download (see image). My code is very primitive:
`datasets.load_dataset('c4', 'en')`

Is this a bug or do I have some configurations missing on my server? 
Thanks!


<img width="1014" alt="Снимок экрана 2021-07-20 в 11 37 17" src="https://user-images.githubusercontent.com/36672861/126289448-6e0db402-5f3f-485a-bf74-eb6e0271fc25.png">

Comments:
Hi Thanks for reporting !
It looks like these files are not correctly reported in the list of expected files to download, let me fix that ;)

Alright this is fixed now. We'll do a new release soon to make the fix available.

In the meantime feel free to simply pass `ignore_verifications=True` to `load_dataset` to skip this error

@lhoestq thank you for such a quick feedback!
Title: Increase json reader block_size automatically
Author: lhoestq (MEMBER)
State: closed
Created: 2021-07-19T14:51:14Z

Currently some files can't be read with the default parameters of the JSON lines reader.
For example this one:
https://huggingface.co/datasets/thomwolf/codeparrot/resolve/main/file-000000000006.json.gz

raises a pyarrow error:
```python
ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```

The block size that is used is the default one by pyarrow (related to this [jira issue](https://issues.apache.org/jira/browse/ARROW-9612)).

To fix this issue I changed the block_size to increase automatically if there is a straddling issue when parsing a batch of json lines.

By default the value is `chunksize // 32` in order to leverage multithreading, and it doubles every time a straddling issue occurs. The block_size is then reset for each file.

cc @thomwolf @albertvillanova

Comments:
Title: Parallelize ETag requests
Author: lhoestq (MEMBER)
State: closed
Created: 2021-07-19T13:30:42Z

Since https://github.com/huggingface/datasets/pull/2628 we use the ETag or the remote data files to compute the directory in the cache where a dataset is saved. This is useful in order to reload the dataset from the cache only if the remote files haven't changed.

In this I made the ETag requests parallel using multithreading. There is also a tqdm progress bar that shows up if there are more than 16 data files.

Comments:
Title: Fix potential DuplicatedKeysError in SQuAD
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-07-19T06:08:00Z

DONE:
- Fix potential DiplicatedKeysError by ensuring keys are unique.
- Align examples in the docs with SQuAD code.

We should promote as a good practice, that the keys should be programmatically generated as unique, instead of read from data (which might be not unique).

Comments:
Title: Fix potential DuplicatedKeysError in LibriSpeech
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-07-19T06:00:49Z

DONE:
- Fix unnecessary path join.
- Fix potential DiplicatedKeysError by ensuring keys are unique.

We should promote as a good practice, that the keys should be programmatically generated as unique, instead of read from data (which might be not unique).

Comments:
Title: Mesinesp development and training data sets have been added.
Author: aslihanuysall (NONE)
State: closed
Created: 2021-07-19T05:14:38Z

https://zenodo.org/search?page=1&size=20&q=mesinesp, Mesinesp has Medical Semantic Indexed records in Spanish.  Indexing is done using DeCS codes, a sort of Spanish equivalent to MeSH terms.
The Mesinesp (Spanish BioASQ track, see https://temu.bsc.es/mesinesp) development set has a total of 750 records.
The Mesinesp (Spanish BioASQ track, see https://temu.bsc.es/mesinesp) training set has a total of 369,368 records.

Comments:
It'll be new pull request with new commits.
Title: Using sharding to parallelize indexing
Author: ggdupont (CONTRIBUTOR)
State: open
Created: 2021-07-18T21:26:26Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
Creating an elasticsearch index on large dataset could be quite long and cannot be parallelized on shard (the index creation is colliding)

**Describe the solution you'd like**
When working on dataset shards, if an index already exists, its mapping should be checked and if compatible, the indexing process should continue with the shard data. 

Additionally, at the end of the process, the `_indexes` dict should be send back to the original dataset object (from which the shards have been created) to allow to use the index for later filtering on the whole dataset.

**Describe alternatives you've considered**
Each dataset shard could created independent partial indices. then on the whole dataset level, indices should be all referred in `_indexes` dict and be used in querying through `get_nearest_examples()`. The drawback is that the scores will be computed independently on the partial indices leading to inconsistent values for most scoring based on corpus level statistics (tf/idf, BM25).

**Additional context**
The objectives is to parallelize the index creation to speed-up the process (ie surcharging the ES server which is fine to handle large load) while later enabling search on the whole dataset.

Comments:
Title: Add Russian SuperGLUE
Author: slowwavesleep (CONTRIBUTOR)
State: closed
Created: 2021-07-17T17:41:28Z

Hi,

This adds the [Russian SuperGLUE](https://russiansuperglue.com/) dataset. For the most part I reused the code for the original SuperGLUE, although there are some relatively minor differences in the structure that I accounted for.

Comments:
Added the missing label classes and their explanations (to the best of my understanding)

Thanks a lot ! Once the last comment about the label names is addressed we can merge :)
Title: Use tqdm from tqdm_utils
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-07-17T17:06:35Z

This PR replaces `tqdm` from the `tqdm` lib with `tqdm` from `datasets.utils.tqdm_utils`. With this change, it's possible to disable progress bars just by calling `disable_progress_bar`. Note this doesn't work on Windows when using multiprocessing due to how global variables are shared between processes. Currently, there is no easy way to disable progress bars in a multiprocess setting on Windows (patching logging with `datasets.utils.logging.get_verbosity = lambda: datasets.utils.logging.NOTSET` doesn't seem to work as well), so adding support for this is a future goal. Additionally, this PR adds a unit ("ba" for batches) to the bar printed by `Dataset.to_json` (this change is motivated by https://github.com/huggingface/datasets/issues/2657).

Comments:
The current CI failure is due to modifications in the dataset script.

Merging since the CI is only failing because of dataset card issues, which is unrelated to this PR
Title: Adds CodeClippy dataset [WIP]
Author: arampacha (NONE)
State: closed
Created: 2021-07-17T13:32:04Z
Labels: dataset contribution

CodeClippy is an opensource code dataset scrapped from github during flax-jax-community-week
https://the-eye.eu/public/AI/training_data/code_clippy_data/

Comments:
Thanks for your contribution, @arampacha. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.

Sorry to resurrect a dead issue, but any chance the dataset will make it to HuggingFace? I would love to use it to finetune Llama 2 and HF makes this a breeze. Also happy to submit a PR prepping it for HF if that is needed.
Title: Adds APPS dataset to the hub [WIP]
Author: arampacha (NONE)
State: closed
Created: 2021-07-17T13:13:17Z
Labels: dataset contribution

A loading script for [APPS dataset](https://github.com/hendrycks/apps)

Comments:
Thanks for your contribution, @arampacha. Are you still interested in adding this dataset?

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
Title: [`to_json`] add multi-proc sharding support
Author: stas00 (CONTRIBUTOR)
State: closed
Created: 2021-07-16T19:41:50Z
Labels: enhancement

As discussed on slack it appears that `to_json` is quite slow on huge datasets like OSCAR.

I implemented sharded saving, which is much much faster - but the tqdm bars all overwrite each other, so it's hard to make sense of the progress, so if possible ideally this multi-proc support could be implemented internally in `to_json` via `num_proc` argument. I guess `num_proc` will be the number of shards?

I think the user will need to use this feature wisely, since too many processes writing to say normal style HD is likely to be slower than one process.

I'm not sure whether the user should be responsible to concatenate the shards at the end  or `datasets`, either way works for my needs.

The code I was using:

```
from multiprocessing import cpu_count, Process, Queue

[...]

filtered_dataset = concat_dataset.map(filter_short_documents, batched=True, batch_size=256, num_proc=cpu_count())

DATASET_NAME = "oscar"
SHARDS = 10
def process_shard(idx):
    print(f"Sharding {idx}")
    ds_shard = filtered_dataset.shard(SHARDS, idx, contiguous=True)
    # ds_shard = ds_shard.shuffle() # remove contiguous=True above if shuffling
    print(f"Saving {DATASET_NAME}-{idx}.jsonl")
    ds_shard.to_json(f"{DATASET_NAME}-{idx}.jsonl", orient="records", lines=True, force_ascii=False)

queue = Queue()
processes = [Process(target=process_shard, args=(idx,)) for idx in range(SHARDS)]
for p in processes:
    p.start()

for p in processes:
    p.join()
```

Thank you!

@lhoestq

Comments:
Hi @stas00, 
I want to work on this issue and I was thinking why don't we use `imap` [in this loop](https://github.com/huggingface/datasets/blob/440b14d0dd428ae1b25881aa72ba7bbb8ad9ff84/src/datasets/io/json.py#L99)? This way, using offset (which is being used to slice the pyarrow table) we can convert  pyarrow table to `json` using multiprocessing. I've a small code snippet for some clarity:
```
result = list(
            pool.imap(self._apply_df, [(offset, batch_size) for offset in range(0, len(self.dataset), batch_size)])
        )
```
`_apply_df` is a function which will return `batch.to_pandas().to_json(path_or_buf=None, orient="records", lines=True)` which is basically json version of the batched pyarrow table. Later on we can concatenate it to form json file? 

I think the only downside here is to write file from `imap` output (output would be a list and we'll need to iterate over it and write in a file) which might add a little overhead cost. What do you think about this?

Followed up in https://github.com/huggingface/datasets/pull/2747
Title: Load Dataset from the Hub (NO DATASET SCRIPT)
Author: lhoestq (MEMBER)
State: closed
Created: 2021-07-16T17:21:58Z

## Load the data from any Dataset repository on the Hub

This PR adds support for loading datasets from any dataset repository on the hub, without requiring any dataset script.

As a user it's now possible to create a repo and upload some csv/json/text/parquet files, and then be able to load the data in one line. Here is an example with the `allenai/c4` repository that contains a lot of compressed json lines files:
```python
from datasets import load_dataset

data_files = {"train": "en/c4-train.*.json.gz"}
c4 = load_dataset("allenai/c4", data_files=data_files, split="train", streaming=True)

print(c4.n_shards)
# 1024
print(next(iter(c4)))
# {'text': 'Beginners BBQ Class Takin...'}
```

By default it loads all the files, but as shown in the example you can choose the ones you want with unix style patterns.

Of course it's still possible to use dataset scripts since they offer the most flexibility.

## Implementation details

It uses `huggingface_hub` to list the files in a dataset repository.

If you provide a path to a local directory instead of a repository name, it works the same way but it uses `glob`.

Depending on the data files available, or passed in the `data_files` parameter, one of the available builders will be used among the csv, json, text and parquet builders.

Because of this, it's not possible to load both csv and json files at once. In this case you have to load them separately and then concatenate the two datasets for example.

## TODO

- [x] tests
- [x] docs
- [x] when huggingface_hub gets a new release, update the CI and the setup.py

Close https://github.com/huggingface/datasets/issues/2629

Comments:
This is ready for review now :)

I would love to have some feedback on the changes in load.py @albertvillanova. There are many changes so if you have questions let me know, especially on the `resolve_data_files` functions and on the changes in `prepare_module`.

And @thomwolf if you want to take a look at the documentation, feel free to share your suggestions :)

I took your comments into account thanks !
And I made `aiohttp` a required dependency :)

Just updated the documentation :)
[share_datasets.html](https://45532-250213286-gh.circle-artifacts.com/0/docs/_build/html/share_dataset.html)

Let me know if you have some comments

Merging this one :) 

We can try to integrate the changes in the docs to #2718 @stevhliu !

Baked this into the [docs](https://44335-250213286-gh.circle-artifacts.com/0/docs/_build/html/loading.html#hugging-face-hub) already, let me know if there is anything else I should add! :)
Title: Move checks from _map_single to map
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-07-16T13:53:33Z

The goal of this PR is to remove duplicated checks in the `map` logic to execute them only once whenever possible (`fn_kwargs`, `input_columns`, ...). Additionally, this PR improves the consistency (to align it with `input_columns`) of the `remove_columns` check by adding support for a single string value, which is then wrapped into a list.

Comments:
@lhoestq This one has been open for a while. Could you please take a look?

@lhoestq Ready for the final review!

I forgot to update the signature of `DatasetDict.map`, so did that now.
Title: Allow dataset config kwargs to be None
Author: lhoestq (MEMBER)
State: closed
Created: 2021-07-16T10:25:38Z

Close https://github.com/huggingface/datasets/issues/2658

The dataset config kwargs that were set to None we simply ignored.
This was an issue when None has some meaning for certain parameters of certain builders, like the `sep` parameter of the "csv" builder that allows to infer to separator.

cc @SBrandeis

Comments:
Title: Can't pass `sep=None` to load_dataset("csv", ...) to infer the separator via pandas.read_csv
Author: lhoestq (MEMBER)
State: closed
Created: 2021-07-16T10:05:44Z

When doing `load_dataset("csv", sep=None)`, the `sep` passed to `pd.read_csv` is still the default `sep=","` instead, which makes it impossible to make the csv loader infer the separator.

Related to https://github.com/huggingface/datasets/pull/2656

cc @SBrandeis

Comments:
Title: `to_json` reporting enhancements
Author: stas00 (CONTRIBUTOR)
State: open
Created: 2021-07-15T23:32:18Z
Labels: enhancement

While using `to_json` 2 things came to mind that would have made the experience easier on the user:

1. Could we have a `desc` arg for the tqdm use and a fallback to just `to_json` so that it'd be clear to the user what's happening? Surely, one can just print the description before calling json, but I thought perhaps it'd help to have it self-identify like you did for other progress bars recently.

2. It took me a while to make sense of the reported numbers:
```
 22%|██▏       | 1536/7076 [12:30:57<44:09:42, 28.70s/it]
```
So iteration here happens to be 10K samples, and the total is 70M records. But the user does't know that, so the progress bar is perfect, but the numbers it reports are meaningless until one discovers that 1it=10K samples. And one still has to convert these in the head - so it's not quick. Not exactly sure what's the best way to approach this, perhaps it can be part of `desc`? or report M or K, so it'd be built-in if it were to print, e.g.:
```
 22%|██▏       | 15360K/70760K [12:30:57<44:09:42, 28.70s/it]
```
or 
```
 22%|██▏       | 15.36M/70.76M [12:30:57<44:09:42, 28.70s/it]
```
(while of course remaining friendly to small datasets)

I forget if tqdm lets you add a magnitude identifier to the running count.

Thank you!

Comments:
Title: Change `from_csv` default arguments
Author: SBrandeis (CONTRIBUTOR)
State: closed
Created: 2021-07-15T14:09:06Z

Passing `sep=None` to pandas's `read_csv` lets pandas guess the CSV file's separator

This PR allows users to use this pandas's feature by passing `sep=None` to `Dataset.from_csv`:

```python
Dataset.from_csv(
    ...,
    sep=None
)
```

Comments:
This is not the default in pandas right ?
We try to align our CSV loader with the pandas API.

Moreover according to their documentation, the python parser is used when sep is None, which might not be the fastest one.

Maybe users could just specify `sep=None` themselves ?
In this case we should add some documentation about this
Title: Allow the selection of multiple columns at once
Author: Dref360 (CONTRIBUTOR)
State: closed
Created: 2021-07-15T13:30:45Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**

Similar to pandas, it would be great if we could select multiple columns at once.


**Describe the solution you'd like**
```python
my_dataset = ...  # Has columns ['idx', 'sentence', 'label']
idx, label = my_dataset[['idx', 'label']]
```

**Describe alternatives you've considered**
we can do `[dataset[col] for col in ('idx', 'label')]`

**Additional context**
This is of course very minor.

Comments:
Hi! I was looking into this and hope you can clarify a point. Your my_dataset variable would be of type DatasetDict which means the alternative you've described (dict comprehension) is what makes sense. 
Is there a reason why you wouldn't want to convert my_dataset to a pandas df if you'd like to use it like one? Please let me know if I'm missing something.

Hi! Sorry for the delay.

In this case, the dataset would be a `datasets.Dataset` and we want to select multiple columns, the `idx` and `label` columns for example.

My issue is that my dataset is too big for memory if I load everything into pandas.

Hello - any update on this? Thank you.

Please note that you could use the method `Dataset.select_columns`: https://huggingface.co/docs/datasets/v2.15.0/en/package_reference/main_classes#datasets.Dataset.select_columns

Thank you!
Title: Give a user feedback if the dataset he loads is streamable or not
Author: philschmid (CONTRIBUTOR)
State: open
Created: 2021-07-15T09:07:27Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
I would love to know if a `dataset` is with the current implementation streamable or not. 

**Describe the solution you'd like**
We could show a warning when a dataset is loaded with `load_dataset('...',streaming=True)` when its lot streamable, e.g. if it is an archive. 

**Describe alternatives you've considered**
Add a new metadata tag for "streaming"

Comments:
#self-assign

I understand it already raises a `NotImplementedError` exception, eg:

```
>>> dataset = load_dataset("journalists_questions", name="plain_text", split="train", streaming=True)

[...]
NotImplementedError: Extraction protocol for file at https://drive.google.com/uc?export=download&id=1CBrh-9OrSpKmPQBxTK_ji6mq6WTN_U9U is not implemented yet
```
Title: Add SD task for SUPERB
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-07-15T07:51:40Z
Labels: dataset request

Include the SD (Speaker Diarization) task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051) and `s3prl` [instructions](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#sd-speaker-diarization).

Steps:
- [x] Generate the LibriMix corpus
- [x] Prepare the corpus for diarization
- [x] Upload these files to the superb-data repo
- [x] Transcribe the corresponding s3prl processing of these files into our superb loading script
- [ ] README: tags + description sections

Related to #2619.

cc: @lewtun

Comments:
Note that this subset requires us to:

* generate the LibriMix corpus from LibriSpeech
* prepare the corpus for diarization

As suggested by @lhoestq we should perform these steps locally and add the prepared data to this public repo on the Hub: https://huggingface.co/datasets/superb/superb-data

Then we can use the URLs for the files to load the data in `superb`'s dataset loading script.

For consistency, I suggest we name the folders in `superb-data` in the same way as the configs in the dataset loading script - e.g. use `sd` for speech diarization in both places :)

@lewtun @lhoestq: 

I have already generated the LibriMix corpus and prepared the corpus for diarization. The output is 3 dirs (train, dev, test), each one containing 6 files: reco2dur  rttm  segments  spk2utt  utt2spk  wav.scp

Next steps:
- Upload these files to the superb-data repo
- Transcribe the corresponding s3prl processing of these files into our superb loading script

Note that processing of these files is a bit more intricate than usual datasets: https://github.com/s3prl/s3prl/blob/master/s3prl/downstream/diarization/dataset.py#L233
Title: Setting log level higher than warning does not suppress progress bar
Author: Isa-rentacs (NONE)
State: closed
Created: 2021-07-14T21:06:51Z
Labels: bug

## Describe the bug
I would like to disable progress bars for `.map` method (and other methods like `.filter` and `load_dataset` as well).
According to #1627 one can suppress it by setting log level higher than `warning`, however doing so doesn't suppress it with version 1.9.0.

I also tried to set `DATASETS_VERBOSITY` environment variable to `error` or `critical` but it also didn't work.

## Steps to reproduce the bug
```python
import datasets

from datasets.utils.logging import set_verbosity_error

set_verbosity_error()

def dummy_map(batch):
    return batch

common_voice_train = datasets.load_dataset("common_voice", "de", split="train")
common_voice_test = datasets.load_dataset("common_voice", "de", split="test")

common_voice_train.map(dummy_map)
```

## Expected results
- The progress bar for `.map` call won't be shown

## Actual results
- The progress bar for `.map` is still shown 

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Linux-5.4.0-1045-aws-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyArrow version: 4.0.1

Comments:
Hi,

you can suppress progress bars by patching logging as follows:
```python
import datasets
import logging
datasets.logging.get_verbosity = lambda: logging.NOTSET
# map call ...
```
EDIT: now you have to use `disable_progress_bar `

Thank you, it worked :)

See https://github.com/huggingface/datasets/issues/2528 for reference

Note also that you can disable the progress bar with

```python
from datasets.utils import disable_progress_bar
disable_progress_bar()
```

See https://github.com/huggingface/datasets/blob/8814b393984c1c2e1800ba370de2a9f7c8644908/src/datasets/utils/tqdm_utils.py#L84

Now the library officially recommends `set_progress_bar_enabled(False)`

```py
from datasets.utils import set_progress_bar_enabled

set_progress_bar_enabled(False)
```

source:

https://github.com/huggingface/datasets/blob/1fd47120ace13626c528367787ffa13e1a26e6c0/src/datasets/utils/tqdm_utils.py#L83-L88



From https://github.com/huggingface/datasets/pull/3897, `disable_progress_bar` is the function you should use

Now ``disable_progress_bar`` function is in ``datasets/src/datasets/utils/logging.py``.
https://github.com/huggingface/datasets/blob/aa555a299ad73c65e3f997a764e9d211675ab05d/src/datasets/utils/logging.py#L233-L236
And the method mentioned in https://github.com/huggingface/datasets/issues/2651#issuecomment-880270774 is not working now.
Title: [load_dataset] shard and parallelize the process
Author: stas00 (CONTRIBUTOR)
State: closed
Created: 2021-07-14T18:04:58Z
Labels: enhancement

- Some huge datasets take forever to build the first time. (e.g. oscar/en) as it's done in a single cpu core.
- If the build crashes, everything done up to that point gets lost

Request: Shard the build over multiple arrow files, which would enable:
- much faster build by parallelizing the build process
- if the process crashed, the completed arrow files don't need to be re-built again

Thank you!

@lhoestq

Comments:
I need the same feature for distributed training

I think @TevenLeScao is exploring adding multiprocessing in `GeneratorBasedBuilder._prepare_split` - feel free to post updates here :)

Posted a PR to address the building side, still needs something to load sharded arrow files + tests

Closing as this feature has been implemented in #5107
Title: adding progress bar / ETA for `load_dataset`
Author: stas00 (CONTRIBUTOR)
State: open
Created: 2021-07-14T17:34:39Z
Labels: enhancement

Please consider:
```
Downloading and preparing dataset oscar/unshuffled_deduplicated_en (download: 462.40 GiB, generated: 1.18 TiB, post-processed: Unknown size, total: 1.63 TiB) to cache/oscar/unshuffled_deduplicated_en/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2...
HF google storage unreachable. Downloading and preparing it from source
```
and no indication whatsoever of whether things work well or when it'll be done. It's important to have an estimated completion time for when doing slurm jobs since some instances have a cap on run-time.

I think for this particular job it sat for 30min in total silence and then after 30min it started generating:
```
897850 examples [07:24, 10286.71 examples/s]
```
which is already great!

Request: 
1. ETA - knowing how many hours to allocate for a slurm job
2. progress bar - helps to know things are working and aren't stuck and where we are at.

Thank you!

@lhoestq

Comments:
Is this done now? I see progress bars when using `load_dataset`.

There are progress bars when downloading data and when preparing them as Arrow files.

The "total silence" part mentioned in OP refer to checksums verifications which have had some changes in the latest release 2.10:
- they're disabled by default (other less costly verifications are still done like checking the generated dataset size)
- they're using a tqdm bar as well
Title: Add web_split dataset for Paraphase and Rephrase benchmark
Author: bhadreshpsavani (CONTRIBUTOR)
State: open
Created: 2021-07-14T14:24:36Z
Labels: enhancement

## Describe:
For getting simple sentences from complex sentence there are dataset and task like wiki_split that is available in hugging face datasets. This web_split is a very similar dataset. There some research paper which states that by combining these two datasets we if we train the model it will yield better results on both tests data.

This dataset is made from web NLG data.

All the dataset related details are provided in the below repository

Github link: https://github.com/shashiongithub/Split-and-Rephrase

Comments:
#take
Title: downloading of yahoo_answers_topics dataset failed
Author: vikrant7k (NONE)
State: closed
Created: 2021-07-14T12:31:05Z
Labels: bug

## Describe the bug
I get an error datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files when I try to download the yahoo_answers_topics dataset

## Steps to reproduce the bug
   self.dataset = load_dataset(
                'yahoo_answers_topics', cache_dir=self.config['yahoo_cache_dir'], split='train[:90%]')
# Sample code to reproduce the bug
   self.dataset = load_dataset(
                'yahoo_answers_topics', cache_dir=self.config['yahoo_cache_dir'], split='train[:90%]')

## Expected results
A clear and concise description of the expected results.


## Actual results
Specify the actual results or traceback.
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files

Comments:
Hi ! I just tested and it worked fine today for me.

I think this is because the dataset is stored on Google Drive which has a quota limit for the number of downloads per day, see this similar issue https://github.com/huggingface/datasets/issues/996 

Feel free to try again today, now that the quota was reset

Fixed once data URL was replaced:
- #4023
Title: load_dataset processing failed with OS error after downloading a dataset
Author: fake-warrior8 (NONE)
State: closed
Created: 2021-07-14T12:23:53Z
Labels: bug

## Describe the bug
After downloading a dataset like opus100, there is a bug that 
OSError: Cannot find data file.
Original error:
dlopen: cannot load any more object with static TLS

## Steps to reproduce the bug
```python
from datasets import load_dataset
this_dataset = load_dataset('opus100', 'af-en')
```

## Expected results
there is no error when running load_dataset.

## Actual results
Specify the actual results or traceback.

Traceback (most recent call last):
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 652, in _download_and_prep
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 989, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 952, in encode_example
    example = cast_to_python_objects(example)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 219, in cast_to_python_ob
    return _cast_to_python_objects(obj)[0]
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 165, in _cast_to_python_o
    import torch
  File "/home/anaconda3/lib/python3.6/site-packages/torch/__init__.py", line 188, in <module>
    _load_global_deps()
  File "/home/anaconda3/lib/python3.6/site-packages/torch/__init__.py", line 141, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/home/anaconda3/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "download_hub_opus100.py", line 9, in <module>
    this_dataset = load_dataset('opus100', language_pair)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 575, in download_and_prepa
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 658, in _download_and_prep
    + str(e)
OSError: Cannot find data file.
Original error:
dlopen: cannot load any more object with static TLS


## Environment info
- `datasets` version: 1.8.0
- Platform: Linux-3.13.0-32-generic-x86_64-with-debian-jessie-sid
- Python version: 3.6.6
- PyArrow version: 3.0.0

Comments:
Hi ! It looks like an issue with pytorch.

Could you try to run `import torch` and see if it raises an error ?

> Hi ! It looks like an issue with pytorch.
> 
> Could you try to run `import torch` and see if it raises an error ?

It works. Thank you!
Title: Support multi-worker with streaming dataset (IterableDataset).
Author: changjonathanc (CONTRIBUTOR)
State: open
Created: 2021-07-14T08:22:58Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
The current `.map` does not support multi-process, CPU can become bottleneck if the pre-processing is complex (e.g. t5 span masking).

**Describe the solution you'd like**
Ideally `.map` should support multi-worker like tfds, with `AUTOTUNE`.

**Describe alternatives you've considered**
A simpler solution is to shard the dataset and process it in parallel with pytorch dataloader. The shard does not need to be of equal size.
* https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset

**Additional context**

Comments:
Hi ! This is a great idea :)
I think we could have something similar to what we have in `datasets.Dataset.map`, i.e. a `num_proc` parameter that tells how many processes to spawn to parallelize the data processing. 

Regarding AUTOTUNE, this could be a nice feature as well, we could see how to add it in a second step

Any update on this feature request?

Not yet, I'm happy to provide some guidance if someone wants to give it a shot though.

The code that applies the `map` function is in `iterable_dataset.py`, in `MappedExamplesIterable.__iter__`
Title: Streaming for the Json loader
Author: lhoestq (MEMBER)
State: closed
Created: 2021-07-13T14:37:06Z

It was not using `open` in the builder. Therefore `pyarrow.json.read_json` was downloading the full file to start yielding rows.

Moreover, it appeared that `pyarrow.json.read_json` was not really suited for streaming as it was downloading too much data and failing if `block_size` was not properly configured (related to #2573).

So I switched to using `open` which is extended to support reading from remote file progressively, and I removed the pyarrow json reader which was not practical.
Instead, I'm using the classical `json.loads` from the standard library.

Comments:
A note is that I think we should add a few indicator of status (as mentioned by @stas00 in #2649), probably at the (1) downloading, (2) extracting and (3) reading steps. In particular when loading many very large files it's interesting to know a bit where we are in the process.

I tested locally, and the builtin `json` loader is 4x slower than `pyarrow.json`. Thanks for the comment @albertvillanova !

Therefore I switched back to using `pyarrow.json`, but only on the batch that is read. This way we don't have to deal with its `block_size`, and it only loads in memory one batch at a time.
Title: add image-classification task template
Author: nateraw (CONTRIBUTOR)
State: closed
Created: 2021-07-12T17:41:03Z

Snippet below is the tl;dr, but you can try it out directly here:

[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/nateraw/005c025d41f0e48ae3d4ee61c0f20b70/image-classification-task-template-demo.ipynb)

```python
from datasets import load_dataset

ds = load_dataset('nateraw/image-folder', data_files='PetImages/')
# DatasetDict({
#     train: Dataset({
#         features: ['file', 'labels'],
#         num_rows: 23410
#     })
# })

ds = ds.prepare_for_task('image-classification')
# DatasetDict({
#     train: Dataset({
#         features: ['image_file_path', 'labels'],
#         num_rows: 23410
#     })
# })
```

Comments:
Awesome!

Thanks for adding a new task template - great work @nateraw  🚀 !
Title: Progress bars are not properly rendered in Jupyter notebook
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-07-12T14:07:13Z
Labels: bug

## Describe the bug
The progress bars are not Jupyter widgets; regular progress bars appear (like in a terminal).

## Steps to reproduce the bug
```python
ds.map(tokenize, num_proc=10)
```

## Expected results
Jupyter widgets displaying the progress bars.

## Actual results
Simple plane progress bars.

cc: Reported by @thomwolf

Comments:
To add my experience when trying to debug this issue:

Seems like previously the workaround given [here](https://github.com/tqdm/tqdm/issues/485#issuecomment-473338308) worked around this issue. But with the latest version of jupyter/tqdm I still get terminal warnings that IPython tried to send a message from a forked process.

Hi @mludv, thanks for the hint!!! :) 

We will definitely take it into account to try to fix this issue... It seems somehow related to `multiprocessing` and `tqdm`...
Title: Load datasets from the Hub without requiring a dataset script
Author: lhoestq (MEMBER)
State: closed
Created: 2021-07-12T08:45:17Z

As a user I would like to be able to upload my csv/json/text/parquet/etc. files in a dataset repository on the Hugging Face Hub and be able to load this dataset with `load_dataset` without having to implement a dataset script.

Moreover I would like to be able to specify which file goes into which split using the `data_files` argument.

This feature should be compatible with private repositories and dataset streaming.

This can be implemented by checking the extension of the files in the dataset repository and then by using the right dataset builder that is already packaged in the library (csv/json/text/parquet/etc.)

Comments:
This is so cool, let us know if we can help with anything on the hub side (@Pierrci @elishowk) 🎉
Title: can't set verbosity for `metric.py`
Author: thomas-happify (NONE)
State: closed
Created: 2021-07-10T20:23:45Z
Labels: bug

## Describe the bug
```
[2021-07-10 20:13:11,528][datasets.utils.filelock][INFO] - Lock 139705371374976 acquired on /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.lock
[2021-07-10 20:13:11,529][datasets.arrow_writer][INFO] - Done writing 32 examples in 6100 bytes /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.
[2021-07-10 20:13:11,531][datasets.arrow_dataset][INFO] - Set __getitem__(key) output type to python objects for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
[2021-07-10 20:13:11,543][/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric.py][INFO] - Removing /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow
```
As you can see, `datasets` logging come from different places. 
`filelock`, `arrow_writer` & `arrow_dataset` comes from `datasets.*` which are expected 
However, `metric.py` logging comes from `/conda/envs/myenv/lib/python3.8/site-packages/datasets/`

So when setting `datasets.utils.logging.set_verbosity_error()`,  it still logs the last message which is annoying during evaluation. 

I had to do 
```
logging.getLogger("/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric").setLevel(logging.ERROR)
``` 
to fully mute these messages

## Expected results
it shouldn't log these messages when setting `datasets.utils.logging.set_verbosity_error()`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: tried both 1.8.0 & 1.9.0
- Platform: Ubuntu 18.04.5 LTS 
- Python version: 3.8.10
- PyArrow version: 3.0.0

Comments:
Thanks @thomas-happify for reporting and thanks @mariosasko for the fix.
Title: [Metrics] added wiki_split metrics
Author: bhadreshpsavani (CONTRIBUTOR)
State: closed
Created: 2021-07-10T14:51:50Z

Fixes: #2606

This pull request adds combine metrics for the wikisplit or English sentence split task

Reviewer: @patrickvonplaten

Comments:
Looks all good to me thanks :)
Just did some minor corrections in the docstring
Title: Integration with AugLy
Author: Darktex (NONE)
State: closed
Created: 2021-07-10T00:03:09Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
Facebook recently launched a library, [AugLy](https://github.com/facebookresearch/AugLy) , that has a unified API for augmentations for image, video and text.

It would be pretty exciting to have it hooked up to HF libraries so that we can make NLP models robust to misspellings or to punctuation, or emojis etc. Plus, with Transformers supporting more CV use cases, having augmentations support becomes crucial.

**Describe the solution you'd like**
The biggest difference between augmentations and preprocessing is that preprocessing happens only once, but you are running augmentations once per epoch. AugLy operates on text directly, so this breaks the typical workflow where we would run the tokenizer once, set format to pt tensors and be ready for the Dataloader.

**Describe alternatives you've considered**

One possible way of implementing these is to make a custom Dataset class where getitem(i) runs the augmentation and the tokenizer every time, though this would slow training down considerably given we wouldn't even run the tokenizer in batches.

Comments:
Hi,

you can define your own custom formatting with `Dataset.set_transform()` and then run the tokenizer with the batches of augmented data as follows:
```python
dset = load_dataset("imdb", split="train")  # Let's say we are working with the IMDB dataset
dset.set_transform(lambda ex: {"text": augly_text_augmentation(ex["text"])}, columns="text", output_all_columns=True)
dataloader = torch.utils.data.DataLoader(dset, batch_size=32)
for epoch in range(5):
    for batch in dataloader:
       tokenizer_output = tokenizer(batch.pop("text"), padding=True, truncation=True, return_tensors="pt")
       batch.update(tokenizer_output)
       output = model(**batch)
       ...
```

Preprocessing functions/augmentations, unless super generic, should be defined in separate libraries, so I'm closing this issue.
Title: Use prefix to allow exceed Windows MAX_PATH
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-07-09T16:39:53Z

By using this prefix, you can exceed the Windows MAX_PATH limit.

See: https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file?redirectedfrom=MSDN#win32-file-namespaces

Related to #2524, #2220.

Comments:
Does this mean the `FileNotFoundError` that avoids infinite loop can be removed?

Yes, I think so...

Or maybe we could leave it in case a relative path exceeds the MAX_PATH limit?

 > Or maybe we could leave it in case a relative path exceeds the MAX_PATH limit?

What about converting relative paths to absolute?

Nice ! Have you had a chance to test it on a windows machine with the max path limit enabled ? Afaik the CI doesn't have the path limit

Sure @lhoestq: I've tested on my machine... And this fixes most of the tests... 😅
Title: Add speech processing tasks
Author: lewtun (MEMBER)
State: closed
Created: 2021-07-09T16:07:29Z

This PR replaces the `automatic-speech-recognition` task category with a broader `speech-processing` category. 

The tasks associated with this category are derived from the [SUPERB benchmark](https://arxiv.org/abs/2105.01051), and ASR is included in this set.

Comments:
Are there any `task_categories:automatic-speech-recognition` dataset for which we should update the tags ?

> Are there any `task_categories:automatic-speech-recognition` dataset for which we should update the tags ?

Yes there's a few - I'll fix them tomorrow :)
Title: Add ASR task for SUPERB
Author: lewtun (MEMBER)
State: closed
Created: 2021-07-09T15:19:45Z

This PR starts building up the SUPERB benchmark by including the ASR task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051) and `s3prl` [instructions](https://github.com/s3prl/s3prl/tree/v0.2.0/downstream#asr-automatic-speech-recognition).

Usage:

```python
from datasets import load_dataset 

asr = load_dataset("superb", "asr")
# DatasetDict({
#     train: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 28539
#     })
#     validation: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 2703
#     })
#     test: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 2620
#     })
# })
```

I've used the GLUE benchmark as a guide for filling out the README.

To move fast during the evaluation PoC I propose to merge one task at a time, so we can continue building the training / evaluation framework in parallel.

Note: codewise this PR is ready for review - I'll add the missing YAML tags once #2620 is merged :)

Comments:
Wait until #2620 is merged before pushing the README tags in this PR

> Thanks!
> 
> One question: aren't you adding `task_templates` to the `_info` method (and to the `dataset_infos.json`?

great catch! i've now added the asr task template (along with a mapping from superb task -> template) and updated the `dataset_infos.json` :) 

> Good!
> 
> I have a suggested refactoring... Tell me what you think! :)

your approach is much more elegant - i've included your suggestions 🙏