rIsHu009/github-issues-updated huggingface dataset:
Title: Support `converters` in `CsvBuilder`
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-10-03T14:23:21Z

Add the `converters` param to `CsvBuilder`, to help in situations like [this one](https://discuss.huggingface.co/t/typeerror-in-load-dataset-related-to-a-sequence-of-strings/23545).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Fix broken URL's (GEM)
Author: manandey (CONTRIBUTOR)
State: closed
Created: 2022-10-03T13:13:22Z

This PR fixes the broken URL's in GEM. cc. @lhoestq, @albertvillanova

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5056). All of your documentation changes will be reflected on that endpoint.

Thanks, @manandey. We have removed all dataset scripts from this repo. Subsequent PRs should be opened directly on the Hugging Face Hub.
Title: Fix backward compatibility for dataset_infos.json
Author: lhoestq (MEMBER)
State: closed
Created: 2022-10-03T10:30:14Z

While working on https://github.com/huggingface/datasets/pull/5018 I noticed a small bug introduced in #4926 regarding backward compatibility for dataset_infos.json

Indeed, when a dataset repo had both dataset_infos.json and README.md, the JSON file was ignored. This is unexpected: in practice it should be ignored only if the README.md has a dataset_info field, which has precedence over the data in the JSON file.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Fix license/citation information of squadshifts dataset card
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-10-03T05:19:13Z
Labels: dataset contribution

This PR fixes the license/citation information of squadshifts dataset card, once the dataset owners have responded to our request for information:
- https://github.com/modestyachts/squadshifts-website/issues/1

Additionally, we have updated the mention in their website to our `datasets` library (they were referring old name `nlp`):
- https://github.com/modestyachts/squadshifts-website/pull/2#event-7500953009

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: added from_generator method to IterableDataset class.
Author: hamid-vakilzadeh (CONTRIBUTOR)
State: closed
Created: 2022-09-30T22:14:05Z

Hello, 

This resolves issues #4988.

I added a method `from_generator` to class `IterableDataset`.

I modified the `read` method of input stream generator to also return Iterable_dataset.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I added a test and moved the `streaming` param from `read` to `__init_`. Then, I also decided to update the `read` method of the rest of the packaged modules to account for this param. 

@hamid-vakilzadeh Are you OK with these changes? 

@mariosasko these all look great! Thanks for the updates.
Title: Revert task removal in folder-based builders
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-09-30T14:50:03Z

Reverts the removal of `task_templates` in the folder-based builders. I also added the `AudioClassifaction` task for consistency.

This is needed to fix https://github.com/huggingface/transformers/issues/19177.

I think we should soon deprecate and remove the current task API (and investigate if it's possible to integrate the `train eval index` API), but we need to update the Transformers examples before that so we don't break them.

cc @NielsRogge

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Restore saved format state in `load_from_disk`
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-09-30T12:40:07Z
Labels: bug

good first issue

Even though we save the `format` state in `save_to_disk`, we don't restore it in `load_from_disk`. We should fix that.
 
Reported here: https://discuss.huggingface.co/t/save-to-disk-loses-formatting-information/23815

Comments:
Hi, can I work on this?

Hi, sure! Let us know if you need some pointers/help.
Title: Add `kwargs` to `Dataset.from_generator`
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-09-30T12:24:27Z

Add the `kwargs` param to `from_generator` to align it with the rest of the `from_` methods (this param allows passing custom `writer_batch_size` for instance).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Fix bug with labels of eurlex config of lex_glue dataset
Author: iliaschalkidis (CONTRIBUTOR)
State: closed
Created: 2022-09-30T09:47:12Z
Labels: dataset contribution

Fix for a critical bug in the EURLEX dataset label list to make LexGLUE EURLEX results replicable. 

In LexGLUE (Chalkidis et al., 2022), the following is mentioned w.r.t. EUR-LEX: _"It supports four different label granularities, comprising 21, 127, 567, 7390 EuroVoc concepts, respectively. We use the 100 most frequent concepts from level 2 [...]”._ The current label list has all 127 labels, which leads to different (lower) results, as communicated by users.

Thanks!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@JamesLYC88 here is the fix! Thanks again!

Thanks, @albertvillanova. When do you expect that this change will take effect when someone downloads the dataset?

The change is immediately available now, since this change we made to our library:
- #4059
Title: Audiofolder creates empty Dataset if files same level as metadata
Author: msis (NONE)
State: closed
Created: 2022-09-29T19:17:23Z
Labels: bug

good first issue

hacktoberfest

## Describe the bug
When audio files are at the same level as the metadata (`metadata.csv` or `metadata.jsonl` ), the `load_dataset` returns a `DatasetDict` with no rows but the correct columns.

https://github.com/huggingface/datasets/blob/1ea4d091b7a4b83a85b2eeb8df65115d39af3766/docs/source/audio_dataset.mdx?plain=1#L88

## Steps to reproduce the bug
`metadata.csv`:
```csv
file_name,duration,transcription
./2063_fe9936e7-62b2-4e62-a276-acbd344480ce_1.wav,10.768,hello
```


```python
>>> audio_dataset = load_dataset("audiofolder", data_dir="/audio-data/")
>>> audio_dataset
DatasetDict({
    train: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
})
```

I've tried, with no success,:
- setting `split` to something else so I don't get a `DatasetDict`,
- removing the `./`,
- using `.jsonl`.


## Expected results
```
 Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 1
})
```

## Actual results
```
DatasetDict({
    train: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
    validation: Dataset({
        features: ['audio', 'duration', 'transcription'],
        num_rows: 0
    })
})
```

## Environment info
- `datasets` version: 2.5.1
- Platform: Linux-5.13.0-1025-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 9.0.0
- Pandas version: 1.5.0

Comments:
Hi! Unfortunately, I can't reproduce this behavior. Instead, I get `ValueError: audio at 2063_fe9936e7-62b2-4e62-a276-acbd344480ce_1.wav doesn't have metadata in /audio-data/metadata.csv`, which can be fixed by removing the `./` from the file name.

(Link to a Colab that tries to reproduce this behavior: https://colab.research.google.com/drive/1IhQzULYi0Van1xLrN_SddBX1JF7mLZZK?usp=sharing)

I think we can make the file name matching part more robust by replacing `file_name` with `os.path.normpath(file_name)`, to ignore "./" among other things, in these two places:
* https://github.com/huggingface/datasets/blob/85cd129bde605cd9acacdff0d065fc02e39e09b1/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L319
* https://github.com/huggingface/datasets/blob/85cd129bde605cd9acacdff0d065fc02e39e09b1/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L388

@mariosasko Some tests failed (see my PR). Any thoughts on that?

Yes, I mentioned the solution in my review.

I realized what I was doing wrong.

The documentation puts the files in a subfolder.
Once I have done that, it worked.

But l agree that this should be handled better if possible.
Title: Hendrycks Checksum
Author: DanielHesslow (NONE)
State: closed
Created: 2022-09-29T06:56:20Z
Labels: dataset bug

Hi, 
The checksum for [hendrycks_test](https://huggingface.co/datasets/hendrycks_test) does not compare correctly, I guess it has been updated on the remote.

```
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://people.eecs.berkeley.edu/~hendrycks/data.tar']
```

Comments:
Thanks for reporting, @DanielHesslow. We are fixing it. 

@albertvillanova thanks for taking care of this so quickly!

The dataset metadata is fixed. You can download it normally.
Title: `Dataset.unique` showing wrong output after filtering
Author: mxschmdt (CONTRIBUTOR)
State: closed
Created: 2022-09-28T16:20:35Z
Labels: bug

## Describe the bug
After filtering a dataset, and if no samples remain, `Dataset.unique` will return the unique values of the unfiltered dataset.

## Steps to reproduce the bug
```python
from datasets import Dataset

dataset = Dataset.from_dict({'id': [0]})
dataset = dataset.filter(lambda _: False)
print(dataset.unique('id'))
```

## Expected results
The above code should return an empty list since the dataset is empty.

## Actual results
```bash
[0]
```

## Environment info
- `datasets` version: 2.5.1
- Platform: Linux-5.18.19-100.fc35.x86_64-x86_64-with-glibc2.34
- Python version: 3.9.14
- PyArrow version: 7.0.0
- Pandas version: 1.3.5

Comments:
Hi! It seems like `flatten_indices` (called in `unique`) doesn't know how to handle empty indices mappings. I'm working on the fix.

Thanks, that was fast!
Title: Add oversampling strategy iterable datasets interleave
Author: ylacombe (CONTRIBUTOR)
State: closed
Created: 2022-09-28T10:10:23Z

Hello everyone,
Following the issue #4893 and the PR #4831, I propose here an oversampling strategy for a `IterableDataset` list.  
The `all_exhausted` strategy stops building the new dataset as soon as all samples in each dataset have been added at least once. 
It follows roughly the same logic behind #4831, namely:
- if ``probabilities`` is `None` and the strategy is `all_exhausted`, it simply performs a round robin interleaving that stops when the longest dataset is out of samples. Here the new dataset length will be $maxLengthDataset*nbDataset$.
- if ``probabilities`` is not `None` and the strategy is `all_exhausted`, it keeps trace of the datasets which were out of samples but continues to add them to the new dataset, and stops as soons as every dataset runs out of samples at least once.

In order to be consistent and also to align with the `Dataset` behavior, please note that the behavior of the default strategy (`first_exhausted`) has been changed. Namely, it really stops when a dataset is out of samples whereas it used to stop when receiving the `StopIteration` error.  
To give an example of the last note, with the following snippet:
```
>>> from tests.test_iterable_dataset import *
>>> d1 = IterableDataset(ExamplesIterable((lambda: (yield from [(i, {"a": i}) for i in [0, 1, 2]])), {}))
>>> d2 = IterableDataset(ExamplesIterable((lambda: (yield from [(i, {"a": i}) for i in [10, 11, 12, 13]])), {}))
>>> d3 = IterableDataset(ExamplesIterable((lambda: (yield from [(i, {"a": i}) for i in [20, 21, 22, 23, 24]])), {}))
>>> dataset = interleave_datasets([d1, d2, d3])
>>> [x["a"] for x in dataset]
```
The result here will then be `[10, 0, 11, 1, 2]` instead of `[10, 0, 11, 1, 2, 20, 12, 13]`.

I modified the behavior because I found it to be consistent with the under/oversampling approach and because it unified the undersampling and oversampling code, but I stay open to any suggestions.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Update README.md of yahoo_answers_topics dataset
Author: borgr (CONTRIBUTOR)
State: closed
Created: 2022-09-28T07:17:33Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5034). All of your documentation changes will be reflected on that endpoint.

Thanks, @borgr. We have removed all dataset scripts from this repo. Subsequent PRs should be opened directly on the Hugging Face Hub.

Do you mean to edit through "edit dataset card" button? because it just leads to a broken page...
https://huggingface.co/datasets/yahoo_answers_topics
![image](https://user-images.githubusercontent.com/6416600/193852796-009ba537-1e8f-4c8b-898a-8c4f817b86ee.png)
https://github.com/huggingface/datasets/tree/main/datasets/yahoo_answers_topics

Hi @borgr, good catch! I'm going to report the button leading to a broken link.

In the meantime, you can propose a PR to the `README.md` file using this link: https://huggingface.co/datasets/yahoo_answers_topics/blob/main/README.md
Title: new dataset type: single-label and multi-label video classification
Author: fcakyon (NONE)
State: open
Created: 2022-09-27T19:40:11Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
In my research, I am dealing with multi-modal (audio+text+frame sequence) video classification. It would be great if the datasets library supported generating multi-modal batches from a video dataset.

**Describe the solution you'd like**
Assume I have video files having single/multiple labels. I want to train a single/multi-label video classification model. I want datasets to support generating multi-modal batches (audio+frame sequence) from video files. Audio waveform and frame sequence can be extracted from each video clip then I can use any audio, image and video model from transformers library to extract features which will be fed into my model.

**Describe alternatives you've considered**
Currently, I am using https://github.com/facebookresearch/pytorchvideo dataloaders. There seems to be not much alternative.

**Additional context**
I am wiling to open a PR but don't know where to start.

Comments:
Hi ! You can in the `features` folder how we implemented the audio and image feature types.

We can have something similar to videos. What we need to decide:
- the video loading library to use
- the output format when a user accesses a video type object
- what parameters a `Video()` feature type needs

also cc @nateraw who also took a look at what we can do for video

@lhoestq @nateraw is there any progress on adding video classification datasets? 

Hi ! I think we just missing which lib we're going to use to decode the videos + which parameters must go in the `Video` type

Hmm. `decord` could be nice but it's no longer maintained [it seems](https://github.com/dmlc/decord/issues/214). 

pytorchvideo uses [pyav](https://github.com/PyAV-Org/PyAV) as the default decoder: https://github.com/facebookresearch/pytorchvideo/blob/c8d23d8b7e597586a9e2d18f6ed31ad8aa379a7a/pytorchvideo/data/labeled_video_dataset.py#L37

Also it would be great if `optionally` audio can also be decoded from the video as in pytorchvideo: https://github.com/facebookresearch/pytorchvideo/blob/c8d23d8b7e597586a9e2d18f6ed31ad8aa379a7a/pytorchvideo/data/labeled_video_dataset.py#L35

Here are the other decoders supported in pytorchvideo: https://github.com/facebookresearch/pytorchvideo/blob/c8d23d8b7e597586a9e2d18f6ed31ad8aa379a7a/pytorchvideo/data/encoded_video.py#L17


@sayakpaul I did do quite a bit of work on [this PR](https://github.com/huggingface/datasets/pull/4532) a while back to add a video feature. It's outdated, but uses my `encoded_video` [package](https://github.com/nateraw/encoded-video) under the hood, which is basically a wrapper around PyAV stolen from [pytorchvideo](https://github.com/facebookresearch/pytorchvideo/) that gets rid of the `torch` dependency. 

would be really great to get something like this in...it's just a really tricky and time consuming feature to add.
Title: Support hfh 0.10 implicit auth
Author: lhoestq (MEMBER)
State: closed
Created: 2022-09-27T18:37:49Z

In huggingface-hub 0.10 the `token` parameter is deprecated for dataset_info and list_repo_files in favor of use_auth_token.
Moreover if use_auth_token=None then the user's token is used implicitly.

I took those two changes into account

Close https://github.com/huggingface/datasets/issues/4990

TODO:
- [x] fix tests

We should wait hfh 0.10 to be relased first to make sure it works correctly before merging

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@lhoestq it is now released so you can move forward with it :) 

I took your comments into account @Wauplin :)
I also bumped the requirement to 0.2.0 because we're using `set_access_token`

cc @albertvillanova WDYT ? I edited the CI job to also check for our minimum supported version of hfh at the same time as the minimum pyarrow version

@lhoestq great, thanks ! :)
Title: Fast dataset iter
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-09-27T16:44:51Z

Use `pa.Table.to_reader` to make iteration over examples/batches faster in `Dataset.{__iter__, map}`

TODO:
* [x] benchmarking (the only benchmark for now - iterating over (single) examples of `bookcorpus` (75 mil examples) in Colab is approx. 2.3x faster)
* [x] check if iterating over bigger chunks + slicing to fetch individual examples in `_iter` yields better performance

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I ran some benchmarks (focused on the data fetching part of `__iter__`) and it seems like the combination `table.to_reader(batch_size)` + `RecordBatch.slice` performs the best ([script](https://gist.github.com/mariosasko/0248288a2e3a7556873969717c1fe52b) with the results). I think we can choose (implicit) `batch_size=10` in the final implementation to avoid having problems with fetching large examples.
Title: Fix import in `ClassLabel` docstring example
Author: alvarobartt (MEMBER)
State: closed
Created: 2022-09-27T11:35:29Z

This PR addresses a super-simple fix: adding a missing `import` to the `ClassLabel` docstring example, as it was formatted as `from datasets Features`, so it's been fixed to `from datasets import Features`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: passing parameters to the method passed to Dataset.from_generator()
Author: Basir-mahmood (NONE)
State: closed
Created: 2022-09-26T15:20:06Z
Labels: enhancement

Big thanks for providing dataset creation via a generator.
I want to ask whether there is any way that parameters can be passed to the method Dataset.from_generator() method, like as follows. 

```
from datasets import Dataset

def gen(param1):
    for idx in len(custom_dataset):
        yield custom_dataset[idx] + param1

ds = Dataset.from_generator(gen(param1))
```

Comments:
Hi! Yes, you can either use the `gen_kwargs` param in `Dataset.from_generator` (`ds = Dataset.from_generator(gen, gen_kwargs={"param1": val})`) or wrap the generator function with `functools.partial`
(`ds = Dataset.from_generator(functools.partial(gen, param1="val"))`) to pass custom parameters to it.
Title: patch CI_HUB_TOKEN_PATH with Path instead of str
Author: Wauplin (CONTRIBUTOR)
State: closed
Created: 2022-09-26T13:19:01Z

Should fix the tests for `huggingface_hub==0.10.0rc0` prerelease (see [failed CI](https://github.com/huggingface/datasets/actions/runs/3127805250/jobs/5074879144)). 
Related to [this thread](https://huggingface.slack.com/archives/C02V5EA0A95/p1664195165294559) (internal link).

Note: this should be a backward compatible fix (e.g. works also with previous versions of `huggingface_hub`)

I am not sure where to put the changes so feel free to cherry-pick the commit and close this one without merging.

cc @lhoestq

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Fix string features of xcsr dataset
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-09-26T11:55:36Z
Labels: dataset contribution

This PR fixes string features of `xcsr` dataset to avoid character splitting.

Fix #5023.

CC: @yangxqiao, @yuchenlin

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Text strings are split into lists of characters in xcsr dataset
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-09-26T11:11:50Z
Labels: dataset bug

## Describe the bug
Text strings are split into lists of characters.

Example for "X-CSQA-en":
```
{'id': 'd3845adc08414fda',
 'lang': 'en',
 'question': {'stem': ['T',
   'h',
   'e',
   ' ',
   'd',
   'e',
   'n',
   't',
   'a',
   'l',
   ' ',
   'o',
   'f',
   'f',
   'i',
   'c',
   'e',
   ' ',
   'h',
   'a',
   'n',
   'd',
   'l',
   'e',
   'd',
   ' ',
   'a',
   ' ',
   'l',
   'o',
   't',
   ' ',
   'o',
   'f',
   ' ',
   'p',
   'a',
   't',
   'i',
   'e',
   'n',
   't',
   's',
   ' ',
   'w',
   'h',
   'o',
   ' ',
   'e',
   'x',
   'p',
   'e',
   'r',
   'i',
   'e',
   'n',
   'c',
   'e',
   'd',
   ' ',
   't',
   'r',
   'a',
   'u',
   'm',
   'a',
   't',
   'i',
   'c',
   ' ',
   'm',
   'o',
   'u',
   't',
   'h',
   ' ',
   'i',
   'n',
   'j',
   'u',
   'r',
   'y',
   ',',
   ' ',
   'w',
   'h',
   'e',
   'r',
   'e',
   ' ',
   'w',
   'e',
   'r',
   'e',
   ' ',
   't',
   'h',
   'e',
   's',
   'e',
   ' ',
   'p',
   'a',
   't',
   'i',
   'e',
   'n',
   't',
   's',
   ' ',
   'c',
   'o',
   'm',
   'i',
   'n',
   'g',
   ' ',
   'f',
   'r',
   'o',
   'm',
   '?'],
  'choices': [{'label': ['A'], 'text': ['t', 'o', 'w', 'n']},
   {'label': ['B'], 'text': ['m', 'i', 'c', 'h', 'i', 'g', 'a', 'n']},
   {'label': ['C'], 'text': ['h', 'o', 's', 'p', 'i', 't', 'a', 'l']},
   {'label': ['D'], 'text': ['s', 'c', 'h', 'o', 'o', 'l', 's']},
   {'label': ['E'],
    'text': ['o',
     'f',
     'f',
     'i',
     'c',
     'e',
     ' ',
     'b',
     'u',
     'i',
     'l',
     'd',
     'i',
     'n',
     'g']}]},
 'answerKey': 'C'}

## Steps to reproduce the bug
```python
ds = load_dataset("datasets/xcsr", "X-CSQA-en", split="validation", streaming=True)
item = next(iter(ds))
item
```

## Expected results
```
{'id': 'd3845adc08414fda',
 'lang': 'en',
 'question': {'stem': 'The dental office handled a lot of patients who experienced traumatic mouth injury, where were these patients coming from?',
  'choices': {'label': ['A', 'B', 'C', 'D', 'E'],
   'text': ['town', 'michigan', 'hospital', 'schools', 'office building']}},
 'answerKey': 'C'}
```

Comments:
Title: Fix languages of X-CSQA configs in xcsr dataset
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-09-26T05:13:39Z
Labels: dataset contribution

Fix #5017.

CC: @yangxqiao, @yuchenlin

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks @lhoestq, I had missed that... 

thx for the super fast work @albertvillanova ! any estimate for when the relevant release will happen?

Thanks again 

@thesofakillers after a recent change in our library (see #4059), now fixes in all datasets are immediately accessible. You can try it:
```python
french = datasets.load_dataset("xcsr", "X-CSQA-fr")
```

Please note there is an additional fix to that dataset in progress (to be merged today):
- #5024
Title: Fix URLs of sbu_captions dataset
Author: donglixp (CONTRIBUTOR)
State: closed
Created: 2022-09-24T14:00:33Z
Labels: dataset contribution

Forbidden
You don't have permission to access /~vicente/sbucaptions/sbu-captions-all.tar.gz on this server.

Additionally, a 403 Forbidden error was encountered while trying to use an ErrorDocument to handle the request.

Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips PHP/5.4.16 mod_fcgid/2.3.9 mod_wsgi/3.4 Python/2.7.5 mod_perl/2.0.11 Perl/v5.16.3 Server at [www.cs.virginia.edu](mailto:csroot@virginia.edu) Port 443

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Update swiss judgment prediction
Author: JoelNiklaus (CONTRIBUTOR)
State: closed
Created: 2022-09-24T13:28:57Z
Labels: dataset contribution

Hi,

I updated the dataset to include additional data made available recently. When I test it locally, it seems to work. However, I get the following error with the dummy data creation: 
`Dummy data generation done but dummy data test failed since splits ['train', 'validation', 'test'] have 0 examples for config 'fr'`. Do you know why this could be the case?

Cheers,
Joel

Comments:
Thank you very much for the detailed review @albertvillanova!

I updated the PR with the requested changes. 

At the end, I had to manually fix the conflict, so that CI tests are launched.

PLEASE NOTE: you should first pull to incorporate the previous commit
```shell
git pull
```

_The documentation is not available anymore as the PR was closed or merged._

Thank you very much for the detailed feedback and your time @albertvillanova! 
Yes, thanks. My other datasets are already on the hub: https://huggingface.co/joelito
Title: Create all YAML dataset_info
Author: lhoestq (MEMBER)
State: closed
Created: 2022-09-23T18:08:15Z
Labels: dataset contribution

Following https://github.com/huggingface/datasets/pull/4926

Creates all the `dataset_info` YAML fields in the dataset cards
The JSON are also updated using the simplified backward compatible format added in https://github.com/huggingface/datasets/pull/4926

Needs https://github.com/huggingface/datasets/pull/4926 to be merged first

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5018). All of your documentation changes will be reflected on that endpoint.

Closing since https://github.com/huggingface/datasets/pull/4974 removed all the datasets scripts.

Individual PRs must be opened on the Hugging face Hub to add the YAML metadata
Title: xcsr: X-CSQA simply uses english for all alleged non-english data
Author: thesofakillers (NONE)
State: closed
Created: 2022-09-23T16:11:54Z
Labels: dataset bug

## Describe the bug
All the alleged non-english subcollections for the X-CSQA task in the [xcsr benchmark dataset ](https://huggingface.co/datasets/xcsr) seem to be copies of the english subcollection, rather than translations. This is in contrast to the data description:

> we automatically translate the original CSQA and CODAH datasets, which only have English versions, to 15 other languages, forming development and test sets for studying X-CSR
 
## Steps to reproduce the bug
```python
# let's say you want to load the french X-CSQA subcollection
french = datasets.load_dataset("xcsr", "X-CSQA-fr")
# for good measure, let's load english too
english = datasets.load_dataset("xcsr", "X-CSQA-en")

# let's inspect
"".join(english['test'][0]['question']['stem'])
# output: 'The people wanted to stop the parade, so what did they set up to thwart it?'
"".join(french['test'][0]['question']['stem'])
# output: 'The people wanted to stop the parade, so what did they set up to thwart it?'
# what? Why are they both in english?

# I've checked this for validation and train splits too, across many datapoints. It's all the same english dataset
# maybe i need to look better?
french['test'].unique('lang')
# output: ['en']
# no, it's all english

```

## Expected results
Accessing a subcollection in language X should return a subcollection containg samples in language X

## Actual results
Accessing a subcollection in language X returns a subcollection containing samples in English.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 2.5.1
- Platform: macOS-10.15.7-x86_64-i386-64bit
- Python version: 3.8.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.3

Comments:
Thanks for reporting, @thesofakillers. Good catch. We are fixing this.
Title: Fix tar extraction vuln
Author: lhoestq (MEMBER)
State: closed
Created: 2022-09-23T14:22:21Z

Fix for CVE-2007-4559

Description:
Directory traversal vulnerability in the (1) extract and (2) extractall functions in the tarfile
module in Python allows user-assisted remote attackers to overwrite arbitrary files via a .. (dot dot)
sequence in filenames in a TAR archive, a related issue to CVE-2001-1267.

I fixed it by using the solution proposed in https://stackoverflow.com/questions/10060069/safely-extract-zip-or-tar-using-python

It blocks extraction of files with an absolute path or double dots and symlinks.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Transfer dataset scripts to Hub
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-09-23T08:48:10Z

Before merging:
- #4974 

TODO:
- [x] Create label: ["dataset contribution"](https://github.com/huggingface/datasets/pulls?q=label%3A%22dataset+contribution%22)
- [x] Create project: [Datasets: Transfer datasets to Hub](https://github.com/orgs/huggingface/projects/22/)
- [x] PRs:
  - [x] Add dataset: we should recommend transfer all additions of datasets to the Hub, under the appropriate namespace; no more additions of datasets on GitHub
  - [x] Update dataset: in general, we should merge bug fixes; enhancements should be considered on a case-by-case basis, depending on whether there is a more suitable namespace on the Hub
- [ ] Issues

Finally:
- [x] #4974

Let me know what you think! :hugs:

Comments:
Sounds good ! Can I help with anything ?
Title: I need to read the custom dataset in conll format
Author: shell-nlp (NONE)
State: open
Created: 2022-09-23T07:49:42Z
Labels: enhancement

I need to read the custom dataset in conll format

Comments:
Hi! We don't currently have a builder for parsing custom `conll` datasets, but I guess we could add one as a packaged module (similarly to what [TFDS](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/dataset_builders/conll/conll_dataset_builder.py) did). @lhoestq @albertvillanova WDYT?

In the meantime, you can use `Dataset.from_generator` to create a dataset as follows:
```python
from datasets import Dataset

# 2009 version
INPUT_COLUMNS = "ID FORM LEMMA PLEMMA POS PPOS FEAT PFEAT HEAD PHEAD DEPREL PDEPREL".split()

def read_conll(file):
    example = {col: [] for col in INPUT_COLUMNS}
    idx = 0
    with open(file) as f:
        for line in f:
            if line.startswith("-DOCSTART-") or line == "\n" or not line:
                if example[next(iter(example))]:
                    yield idx, example
                idx += 1
                example = {col: [] for col in INPUT_COLUMNS}
            else:
                row_cols = line.split()
                for i, col in enumerate(example):
                    example[col] = row_cols[i].rstrip()

# (optional) pass custom features with `features=Features(...)`
dset = Dataset.from_generator(read_conll, gen_kwargs={"file": "path/to/conll/file"})   
``` 

I think we could add a dedicated builder if you think this format is general enough.




> I think we could add a dedicated builder if you think this format is general enough.

I think its functions are incomplete. It should have to_ Conll and from_ There are two methods of conll.
Title: would huggingface like publish cpp binding for datasets package ?
Author: mullerhai (NONE)
State: closed
Created: 2022-09-23T07:42:49Z
Labels: wontfix

HI:
I use cpp env libtorch, I like use hugggingface ,but huggingface not cpp binding, would you like publish cpp binding for it.
thanks

Comments:
Hi ! Can you share more information about your use case ? How could it help you to have cpp bindings versus using the python libraries ?

> Hi ! Can you share more information about your use case ? How could it help you to have cpp bindings versus using the python libraries ?

for example ,the huggingface  load_model() and load_dataset()  can  execute in cpp env

If it's a viable option for you, you can check [tch-rs](https://github.com/LaurentMazare/tch-rs) to load models in Rust. Regarding datasets, you can first download them in python and then use Arrow C++ or Rust to load them

If you are more adventurous, another option is to embed python calls inside c++ e.g. with `pybind11`.

> pybind11

I think it is not the best solution
Title: Force JSON format regardless of file naming on S3
Author: junwang-wish (NONE)
State: closed
Created: 2022-09-22T18:28:15Z
Labels: enhancement

I have a file on S3 created by Data Version Control, it looks like `s3://dvc/ac/badff5b134382a0f25248f1b45d7b2` but contains a json file. If I run 
```python
dataset = load_dataset(
    "json",
    data_files='s3://dvc/ac/badff5b134382a0f25248f1b45d7b2' 
)
```
It gives me 
```
InvalidSchema: No connection adapters were found for 's3://dvc/ac/badff5b134382a0f25248f1b45d7b2'
```
However, I cannot go ahead and change the names of the s3 file. Is there a way to "force" load a S3 url with certain decoder (JSON, CSV, etc.) regardless of s3 URL naming?

Comments:
Hi ! Support for URIs like `s3://...` is not implemented yet in `data_files=`. You can use the HTTP URL instead if your data is public in the meantime

Hi,
I want to make sure I understand this response. I have a set of files on S3 that are private for security reasons. Because they are not public files  I cannot read those files (many are parquet) into my hf notebooks in Kaggle? That can't be correct, can it? 

Hi ! There is a discussion at https://github.com/huggingface/datasets/issues/5281

Using the latest `datasets` 2.11 you can try passing fsspec URLs to private buckets to `data_files` in `load_dataset()`. Though this is still experimental and undocumented, so feedback is welcome. You may not have the best experience though, since anything related to performance and caching hasn't been tested properly yet.

closing this one since data_files supports fsspec (still experimental/untested/undocumented for s3 though)
Title: Add deprecation warning to multilingual_librispeech dataset card
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-09-22T11:41:59Z
Labels: dataset contribution

Besides the current deprecation warning in the script of `multilingual_librispeech`, this PR adds a deprecation warning to its dataset card as well.

The format of the deprecation warning is aligned with the one in the library documentation when docstrings contain the `<Deprecated/>` tag.

Related to:
- #4060

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Re-apply input columns change
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-09-21T15:09:01Z

Fixes the `filter` + `input_columns` combination, which is used in the `transformers` examples for instance.

Revert #5006 (which in turn reverts #4971)

Fix https://github.com/huggingface/datasets/issues/4858

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Revert input_columns change
Author: lhoestq (MEMBER)
State: closed
Created: 2022-09-21T13:49:20Z

Revert https://github.com/huggingface/datasets/pull/4971

Fix https://github.com/huggingface/datasets/issues/5005

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Merging this one and I'll check if it fixes the `transformers` CI before doing a patch release
Title: Release 2.5.0 breaks transformers CI
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-09-21T13:39:19Z
Labels: bug

## Describe the bug
As reported by @lhoestq:

> see https://app.circleci.com/pipelines/github/huggingface/transformers/47634/workflows/b491886b-e66e-4edb-af96-8b459e72aa25/jobs/564563
this is used here: [https://github.com/huggingface/transformers/blob/3b19c0317b6909e2d7f11b5053895ac55[…]torch/speech-pretraining/run_wav2vec2_pretraining_no_trainer.py](https://github.com/huggingface/transformers/blob/3b19c0317b6909e2d7f11b5053895ac55250e7da/examples/pytorch/speech-pretraining/run_wav2vec2_pretraining_no_trainer.py#L482-L488)

Comments:
Shall we revert https://github.com/huggingface/datasets/pull/4971 @mariosasko ?

And for consistency we can update IterableDataset.map later
Title: Remove license tag file and validation
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-09-21T12:35:14Z

As requested, we are removing the validation of the licenses from `datasets` because this is done on the Hub.

Fix #4994.

Related to:
- #4926, which is removing all the validation from `datasets`

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Fix missing use_auth_token in streaming docstrings
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-09-21T09:27:03Z

This PRs fixes docstrings:
- adds the missing `use_auth_token` param
- updates syntax of param types
- adds params to docstrings without them
- fixes return/yield types
- fixes syntax

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Dataset Viewer issue for loubnabnl/humaneval-x
Author: loubnabnl (NONE)
State: closed
Created: 2022-09-21T09:06:17Z
Labels: dataset-viewer

### Link

https://huggingface.co/datasets/loubnabnl/humaneval-x/viewer/

### Description

The dataset has subsets but the viewer gets stuck in the default subset even when I select another one (the data loading of the subsets works fine)

### Owner

Yes

Comments:
It's a bug! Thanks for reporting, I'm looking at it

Fixed.
Title: Add EmptyDatasetError
Author: lhoestq (MEMBER)
State: closed
Created: 2022-09-20T15:28:05Z

examples:

from the hub:
```python
Traceback (most recent call last):
  File "playground/ttest.py", line 3, in <module>
    print(load_dataset("lhoestq/empty"))
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1686, in load_dataset
    **config_kwargs,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1458, in load_dataset_builder
    data_files=data_files,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1171, in dataset_module_factory
    raise e1 from None
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1162, in dataset_module_factory
    download_mode=download_mode,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 760, in get_module
    else get_data_patterns_in_dataset_repository(hfh_dataset_info, self.data_dir)
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/data_files.py", line 678, in get_data_patterns_in_dataset_repository
    ) from None
datasets.data_files.EmptyDatasetError: The dataset repository at 'lhoestq/empty' doesn't contain any data file.
```

from local directory:
```python
Traceback (most recent call last):
  File "playground/ttest.py", line 3, in <module>
    print(load_dataset("playground/empty"))
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1686, in load_dataset
    **config_kwargs,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1458, in load_dataset_builder
    data_files=data_files,
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 1107, in dataset_module_factory
    path, data_dir=data_dir, data_files=data_files, download_mode=download_mode
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/load.py", line 625, in get_module
    else get_data_patterns_locally(base_path)
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/data_files.py", line 460, in get_data_patterns_locally
    raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data file") from None
datasets.data_files.EmptyDatasetError: The directory at playground/empty doesn't contain any data file
```

Close https://github.com/huggingface/datasets/issues/4995

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Don't add a tag on the Hub on release
Author: lhoestq (MEMBER)
State: closed
Created: 2022-09-20T13:54:57Z

Datasets with no namespace on the Hub have tags to redirect to the version of datasets where they come from.
I’m about to remove them all because I think it looks bad/unexpected in the UI and it’s not actually useful

Therefore I'm also disabling tagging.

Note that the CI job will be completely removed in https://github.com/huggingface/datasets/pull/4974 anyway

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Add support for parsing JSON files in array form
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-09-20T13:31:26Z

Support parsing JSON files in the array form (top-level object is an array). For simplicity, `json.load` is used for decoding. This means the entire file is loaded into memory. If requested, we can optimize this by introducing a param similar to `lines` in [`pandas.read_json`](https://pandas.pydata.org/docs/reference/api/pandas.read_json.html), which, if set to `True`, would allow us to read in chunks.

Fixes https://github.com/huggingface/datasets/issues/4963

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Dataset Viewer issue for Jean-Baptiste/wikiner_fr
Author: severo (COLLABORATOR)
State: closed
Created: 2022-09-20T12:32:07Z

### Link

https://huggingface.co/datasets/Jean-Baptiste/wikiner_fr

### Description

```
Error code:   StreamingRowsError
Exception:    FileNotFoundError
Message:      [Errno 2] No such file or directory: 'zip:/data/train::https:/huggingface.co/datasets/Jean-Baptiste/wikiner_fr/resolve/main/data.zip/state.json'
Traceback:    Traceback (most recent call last):
               File "/src/services/worker/src/worker/responses/first_rows.py", line 337, in get_first_rows_response
                 rows = get_rows(dataset, config, split, streaming=True, rows_max_number=rows_max_number, hf_token=hf_token)
               File "/src/services/worker/src/worker/utils.py", line 123, in decorator
                 return func(*args, **kwargs)
               File "/src/services/worker/src/worker/responses/first_rows.py", line 77, in get_rows
                 rows_plus_one = list(itertools.islice(ds, rows_max_number + 1))
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 718, in __iter__
                 for key, example in self._iter():
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 708, in _iter
                 yield from ex_iterable
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/iterable_dataset.py", line 112, in __iter__
                 yield from self.generate_examples_fn(**self.kwargs)
               File "/tmp/modules-cache/datasets_modules/datasets/Jean-Baptiste--wikiner_fr/683a580ba6ec769d508f7dfc603a651667b0ed3817b1ae5bfd45f97cc024923f/wikiner_fr.py", line 165, in _generate_examples
                 dataset = Dataset.load_from_disk(filepath)
               File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 1210, in load_from_disk
                 with open(Path(dataset_path, config.DATASET_STATE_JSON_FILENAME).as_posix(), encoding="utf-8") as state_file:
             FileNotFoundError: [Errno 2] No such file or directory: 'zip:/data/train::https:/huggingface.co/datasets/Jean-Baptiste/wikiner_fr/resolve/main/data.zip/state.json'
```

Is it an error with the dataset script, or the data itself, @huggingface/datasets?

https://huggingface.co/datasets/Jean-Baptiste/wikiner_fr/tree/main 

### Owner

No

Comments:
The script uses `Dataset.load_from_disk`, which as you can expect, doesn't work in streaming mode.

It would probably be more practical to load the dataset locally using `Dataset.load_from_disk` first and then `push_to_hub` to upload it in Parquet on the Hub

I've transferred this issue to the Hub repo: https://huggingface.co/datasets/Jean-Baptiste/wikiner_fr/discussions/3

I'm closing this.
Title: Get a specific Exception when the dataset has no data
Author: severo (COLLABORATOR)
State: closed
Created: 2022-09-20T09:31:59Z
Labels: enhancement

dataset-viewer

In the dataset viewer on the Hub (https://huggingface.co/datasets/glue/viewer), we would like (https://github.com/huggingface/moon-landing/issues/3882) to show a specific message when the repository lacks any data files.

In that case, instead of showing a complex traceback, we want to show a call to action to help the user upload data.

To do that, it would be very helpful to know for sure that the repository is missing any (supported) data files.

It could be done by raising a custom exception, for example, `NoDataError`.

Comments:
Title: delete the hardcoded license list in `datasets`
Author: julien-c (MEMBER)
State: closed
Created: 2022-09-20T09:14:41Z

> Feel free to delete the license list in `datasets` [...]
>
> Also FYI in #4926 I also removed all the validation steps anyway (language, license, types etc.)

_Originally posted by @lhoestq in https://github.com/huggingface/datasets/issues/4930#issuecomment-1238401662_

> [...], in my opinion we can just delete this file from `datasets`, the validation is happening hub-side anyways now?

_Originally posted by @julien-c in https://github.com/huggingface/datasets/issues/4930#issuecomment-1238390659_

Comments:
Title: fix: avoid casting tuples after Dataset.map
Author: szmoro (CONTRIBUTOR)
State: closed
Created: 2022-09-20T08:45:16Z

This PR updates features.py to avoid casting tuples to lists when reading the results of Dataset.map as suggested by @lhoestq [here](https://github.com/huggingface/datasets/issues/4676#issuecomment-1187371367) in https://github.com/huggingface/datasets/issues/4676.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Fix missing tags in dataset cards
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-09-20T06:42:07Z

Fix missing tags in dataset cards:
- aeslc
- empathetic_dialogues
- event2Mind
- gap
- iwslt2017
- newsgroup
- qa4mre
- scicite

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891
- #4896
- #4908
- #4921
- #4931
- #4979

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Running add_column() seems to corrupt existing sequence-type column info
Author: derek-rocheleau (NONE)
State: closed
Created: 2022-09-17T17:42:05Z
Labels: bug

I have a dataset that contains a column ("foo") that is a sequence type of length 4. So when I run .to_pandas() on it, the resulting dataframe correctly contains 4 columns - foo_0, foo_1, foo_2, foo_3. So the 1st row of the dataframe might look like:

ds = load_dataset(...)
df = ds.to_pandas()

df:
foo_0 | foo_1 | foo_2 | foo_3
0.0 | 1.0 | 2.0 | 3.0

If I run .add_column("new_col", data) on the dataset, and then .to_pandas() on the resulting new dataset, the resulting dataframe contains only 2 columns - foo, new_col. The values in column foo are lists of length 4, the 4 elements that should have been split into separate columns. Dataframe 1st row would be:

ds = load_dataset(...)
new_ds = ds.add_column("new_col", data)
df = new_ds.to_pandas()

df:
foo | new_col
[0.0, 1.0, 2.0, 3.0] | new_val

I've explored the 2 datasets in a debugger and haven't noticed any changes to any attributes related to the foo column, but I can't determine why the dataframes are so different.

Comments:
Nevermind, I was incorrect.
Title: Add `IterableDataset.from_generator` to the API
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-09-16T15:19:41Z
Labels: enhancement

good first issue

We've just added `Dataset.from_generator` to the API. It would also be cool to add `IterableDataset.from_generator` to support creating an iterable dataset from a generator.

cc @lhoestq

Comments:
#take

Thanks @hamid-vakilzadeh ! Let us know if you have some questions or if we can help

Thank you! I certainly will reach out if I need any help.