rIsHu009/github-issues-updated huggingface dataset:
Title: None Type error for swda datasets
Author: hannan72 (NONE)
State: closed
Created: 2022-08-29T07:05:20Z
Labels: bug

## Describe the bug
I got `'NoneType' object is not callable` error while calling the swda datasets.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("swda")
```

## Expected results
Run without error

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Python version: 3.8.10

Comments:
Thanks for reporting @hannan72 ! I couldn't reproduce the error on my side, can you share the full stack trace please ?

Thanks a lot for your response @lhoestq 
The problem is solved accidentally today and I don't know exactly why it was happened yesterday.
The issue can be closed.

Ok, let us know if you encounter the issue again ;)
Title: [LibriSpeech] Fix dev split local_extracted_archive for 'all' config
Author: sanchit-gandhi (CONTRIBUTOR)
State: closed
Created: 2022-08-27T10:04:57Z

We define the keys for the `_DL_URLS` of the dev split as `dev.clean` and `dev.other`:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L60-L61

These keys get forwarded to the `dl_manager` and thus the `local_extracted_archive`.

However, when calling `SplitGenerator` for the dev sets, we query the `local_extracted_archive` keys `validation.clean` and `validation.other`:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L212
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L219

The consequence of this is that the `local_extracted_archive` arg passed to `_generate_examples` is always `None`, as the keys `validation.clean` and `validation.other` do not exists in the `local_extracted_archive`.

When defining the `audio_file` in `_generate_examples`, since `local_extracted_archive` is always `None`, we always omit the `local_extracted_archive` path from the `audio_file` path, **even** if in non-streaming mode:
https://github.com/huggingface/datasets/blob/2e7142a3c6500b560da45e8d5128e320a09fcbd4/datasets/librispeech_asr/librispeech_asr.py#L259-L263

Thus, `audio_file` will only ever be the streaming path (`audio_file`, not `os.path.join(local_extracted_archive, audio_file)`).

This PR fixes the `.get()` keys for the `local_extracted_archive` for the dev splits.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This PR fixes a bug introduced in:
- #4184
Title: Fix CI reporting
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-08-26T17:16:30Z

Fix CI so that it reports defaults (failed and error) besides the custom (xfailed and xpassed) in the test summary.

This PR fixes a regression introduced by:
- #4845

This introduced the reporting of xfailed and xpassed, but wrongly removed the reporting of the defaults failed and error.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Name the default config `default`
Author: severo (COLLABORATOR)
State: closed
Created: 2022-08-26T16:16:22Z
Labels: enhancement

question

Currently, if a dataset has no configuration, a default configuration is created from the dataset name.

For example, for a dataset loaded from the hub repository, such as https://huggingface.co/datasets/user/dataset (repo id is `user/dataset`), the default configuration will be `user--dataset`.

It might be easier to handle to set it to `default`, or another reserved word.

Comments:
Addressed in #5331.
Title: Raise ManualDownloadError from get_dataset_config_info
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-08-26T15:45:56Z

This PRs raises a specific `ManualDownloadError` when `get_dataset_config_info` is called for a dataset that requires manual download.

Related to:
- #4898

CC: @severo

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Re-add code and und language tags
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-08-26T09:48:57Z

This PR fixes the removal of 2 language tags done by:
- #4882

The tags are:
- "code": this is not a IANA tag but needed
- "und": this is one of the special scoped tags removed by 0d53202b9abce6fd0358cb00d06fcfd904b875af
  - used in "mc4" and "udhr" datasets

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Dataset Viewer issue for timit_asr
Author: InayatUllah932 (NONE)
State: closed
Created: 2022-08-26T07:12:05Z

### Link

_No response_

### Description

_No response_

### Owner

_No response_

Comments:
Yes, the dataset viewer is based on `datasets`, and the following does not work:

```
>>> from datasets import get_dataset_split_names
>>> get_dataset_split_names('timit_asr')
Downloading builder script: 7.48kB [00:00, 6.69MB/s]
Traceback (most recent call last):
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 354, in get_dataset_config_info
    for split_generator in builder._split_generators(
  File "/home/slesage/.cache/huggingface/modules/datasets_modules/datasets/timit_asr/43f9448dd5db58e95ee48a277f466481b151f112ea53e27f8173784da9254fb2/timit_asr.py", line 117, in _split_generators
    data_dir = os.path.abspath(os.path.expanduser(dl_manager.manual_dir))
  File "/home/slesage/.pyenv/versions/3.9.6/lib/python3.9/posixpath.py", line 231, in expanduser
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 404, in get_dataset_split_names
    info = get_dataset_config_info(
  File "/home/slesage/hf/datasets-server/services/worker/.venv/lib/python3.9/site-packages/datasets/inspect.py", line 359, in get_dataset_config_info
    raise SplitsNotFoundError("The split names could not be parsed from the dataset config.") from err
datasets.inspect.SplitsNotFoundError: The split names could not be parsed from the dataset config.
```

cc @huggingface/datasets 

Due to license restriction, this dataset needs manual downloading of the original data.

This information is in the dataset card: https://huggingface.co/datasets/timit_asr
> The dataset needs to be downloaded manually from https://catalog.ldc.upenn.edu/LDC93S1

Maybe a better error message for datasets that need manual downloading? @severo 

Maybe we can raise a specific excpetion as done from `load_dataset`...

Yes, ideally something like https://github.com/huggingface/datasets/blob/main/src/datasets/builder.py#L81


The preview is now disabled (and a descriptive warning is displayed) for datasets requiring manual download. See:

![timit_asr-manual-download](https://user-images.githubusercontent.com/8515462/193578572-3d21b790-f848-4257-9e9b-7cab3d76a269.png)
Title: datasets generate large arrow file
Author: jax11235 (NONE)
State: closed
Created: 2022-08-26T05:51:16Z
Labels: bug

Checking the large file in disk, and found the large cache file in the cifar10 data directory:

![image](https://user-images.githubusercontent.com/18533904/186830449-ba96cdeb-0fe8-4543-994d-2abe7145933f.png)

As we know, the size of cifar10 dataset is ~130MB, but the cache file has almost 30GB size, there may be some problems here.

Comments:
Hi ! The cache files are the results of all the transforms you applied to the dataset using `map` for example.
Did you run a transform that could potentially blow up the size of the dataset ?

@lhoestq,
I don't remember, but I can't imagine what kind of transform may generate data that grow over 200 times in size. 
I think maybe it doesn' matter, it's just cache after all.
Title: Fix missing tags in dataset cards
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-08-25T16:41:43Z

Fix missing tags in dataset cards:
- anli
- coarse_discourse
- commonsense_qa
- cos_e
- ilist
- lc_quad
- web_questions
- xsum

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833
- #4891

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Add citation information to makhzan dataset
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-08-25T10:16:40Z

This PR adds the citation information to `makhzan` dataset, once they have replied to our request for that information:
- https://github.com/zeerakahmed/makhzan/issues/43

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Add citation to ro_sts and ro_sts_parallel datasets
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-08-25T09:51:06Z

This PR adds the citation information to `ro_sts_parallel` and `ro_sts_parallel` datasets, once they have replied our request for that information:
- https://github.com/dumitrescustefan/RO-STS/issues/4

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4892). All of your documentation changes will be reflected on that endpoint.
Title: Fix missing tags in dataset cards
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-08-25T09:14:17Z

Fix missing tags in dataset cards:
- aslg_pc12
- librispeech_lm
- mwsc
- opus100
- qasc
- quail
- squadshifts
- winograd_wsc

This PR partially fixes the missing tags in dataset cards. Subsequent PRs will follow to complete this task.

Related to:
- #4833

Comments:
Title: add Dataset.from_list
Author: sanderland (CONTRIBUTOR)
State: closed
Created: 2022-08-25T09:05:58Z

As discussed in #4885 

I initially added this bit at the end, thinking filling this field was necessary as it is done in from_dict. 
However, it seems the constructor takes care of filling info when it is empty.
```
if info.features is None:
  info.features = Features(
      {
          col: generate_from_arrow_type(coldata.type)
          for col, coldata in zip(pa_table.column_names, pa_table.columns)
      }
  )
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@albertvillanova it seems tests fail on pyarrow 6, perhaps from_pylist is a v7 method? How do you usually handle these version differences?
Added something that at least works
Title: Dataset Viewer issue for subjqa
Author: lewtun (MEMBER)
State: closed
Created: 2022-08-24T13:26:20Z
Labels: dataset-viewer

### Link

https://huggingface.co/datasets/subjqa

### Description

Getting the following error for this dataset:

```
Status code:   500
Exception:     Status500Error
Message:       2 or more items returned, instead of 1
```

Not sure what's causing it though 🤔 

### Owner

Yes

Comments:
It's a bug in the viewer, thanks for reporting it. We're hoping to update to a new version in the next few days which should fix it.

Fixed 

https://huggingface.co/datasets/subjqa

<img width="1040" alt="Capture d’écran 2022-09-08 à 10 23 26" src="https://user-images.githubusercontent.com/1676121/189073210-2a57ff88-8bb1-44bd-851e-0e75473cea3f.png">
Title: Add "cc-by-nc-sa-2.0" to list of licenses 
Author: osanseviero (CONTRIBUTOR)
State: closed
Created: 2022-08-24T13:11:49Z

Datasets side of https://github.com/huggingface/hub-docs/pull/285

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Sorry for the issue @albertvillanova! I think it's now fixed! :heart:
Title: Create dataset from list of dicts
Author: sanderland (CONTRIBUTOR)
State: closed
Created: 2022-08-24T10:01:24Z
Labels: enhancement

I often find myself with data from a variety of sources, and a list of dicts is very common among these.
However, converting this to a Dataset is a little awkward, requiring either

```Dataset.from_pandas(pd.DataFrame(formatted_training_data))```
Which can error out on some more exotic values as 2-d arrays for reasons that are not entirely clear
> ArrowInvalid: ('Can only convert 1-dimensional array values', 'Conversion failed for column labels with type object')

Alternatively:
```Dataset.from_dict({k: [s[k] for s in formatted_training_data] for k in  formatted_training_data[0].keys()})```
Which works, but is a little ugly.

**Describe the solution you'd like**
Either `.from_dict` accepting a list of dicts, or a `.from_records` function accepting such.

I am happy to PR this, just wanted to check you are happy to accept this I haven't missed something obvious, and which of the solutions would be prefered.

Comments:
Hi @sanderland, thanks for your enhancement proposal.

I agree with you that this would be useful.

Please note that under the hood, we use PyArrow tables as backend:
- The implementation  of `Dataset.from_dict` uses the PyArrow `Table.from_pydict`

Therefore, I would suggest:
- Implementing `Dataset.from_list` using the PyArrow `Table.from_pylist`

What do you think?
Let's see if other people have other suggestions...

Thanks for the quick and positive reply @albertvillanova! 
`from_list` seems sensible. Have opened a PR so we can discuss details there.

Resolved via #4890.
Title: Fix language tags resource file
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-08-24T06:06:01Z

This PR fixes/updates/adds ALL language tags from IANA (as of 2022-08-08).

This PR also removes all BCP47 suffixes (the languages file only contains language subtags, i.e. ISO 639 1 or 2 codes; no script/region/variant suffixes). See:
- #4753

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4882). All of your documentation changes will be reflected on that endpoint.
Title: Added names of less-studied languages
Author: BenjaminGalliot (CONTRIBUTOR)
State: closed
Created: 2022-08-23T19:32:38Z

Added names of less-studied languages (nru – Narua and jya – Japhug) for existing datasets.

Comments:
OK, I removed Glottolog codes and only added ISO 639-3 ones. The former are for the moment in corpus card description, language details, and in subcorpora names.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4880). All of your documentation changes will be reflected on that endpoint.
Title: Fix Citation Information section in dataset cards
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-08-23T18:06:43Z

Fix Citation Information section in dataset cards:
- cc_news
- conllpp
- datacommons_factcheck
- gnad10
- id_panl_bppt
- jigsaw_toxicity_pred
- kinnews_kirnews
- kor_sarcasm
- makhzan
- reasoning_bg
- ro_sts
- ro_sts_parallel
- sanskrit_classic
- telugu_news
- thaiqa_squad
- wiki_movies

This PR partially fixes the Citation Information section in dataset cards. Subsequent PRs will follow to complete this task.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4879). All of your documentation changes will be reflected on that endpoint.
Title: [not really a bug] `identical_ok` is deprecated in huggingface-hub's `upload_file`
Author: severo (COLLABORATOR)
State: closed
Created: 2022-08-23T17:09:55Z
Labels: help wanted

question

In the huggingface-hub dependency, the `identical_ok` argument has no effect in `upload_file` (and it will be removed soon)

See

https://github.com/huggingface/huggingface_hub/blob/43499582b19df1ed081a5b2bd7a364e9cacdc91d/src/huggingface_hub/hf_api.py#L2164-L2169

It's used here:

https://github.com/huggingface/datasets/blob/fcfcc951a73efbc677f9def9a8707d0af93d5890/src/datasets/dataset_dict.py#L1373-L1381

https://github.com/huggingface/datasets/blob/fdcb8b144ce3ef241410281e125bd03e87b8caa1/src/datasets/arrow_dataset.py#L4354-L4362

https://github.com/huggingface/datasets/blob/fdcb8b144ce3ef241410281e125bd03e87b8caa1/src/datasets/arrow_dataset.py#L4197-L4213

We should remove it.

Maybe the third code sample has an unexpected behavior since it uses the non-default value `identical_ok = False`, but the argument is ignored.

Comments:
Resolved via https://github.com/huggingface/datasets/pull/4937.
Title: Fix documentation card of covid_qa_castorini dataset
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-08-23T16:52:33Z

Fix documentation card of covid_qa_castorini dataset.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4877). All of your documentation changes will be reflected on that endpoint.
Title: Multiple dataloader memory error
Author: cyk1337 (NONE)
State: open
Created: 2022-08-23T08:59:50Z
Labels: bug

For the use of multiple datasets and tasks, we use around more than 200+ dataloaders, then pass it into `dataloader1, dataloader2, ..., dataloader200=accelerate.prepare(dataloader1, dataloader2, ..., dataloader200)`
It causes the memory error when generating batches. Any solutions to it?

```bash
  File "/home/xxx/my_code/src/utils/data_utils.py", line 54, in generate_batch
    x = next(iterator)
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/accelerate/data_loader.py", line 301, in __iter__
    for batch in super().__iter__():
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 28, in fetch
    data.append(next(self.dataset_iter))
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/accelerate/data_loader.py", line 249, in __iter__
    for element in self.dataset:
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 503, in __iter__
    for key, example in self._iter():
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 500, in _iter
    yield from ex_iterable
  File "/home/xxx/anaconda3/envs/pt1.7/lib/python3.7/site-packages/datasets/iterable_dataset.py", line 231, in __iter__
    new_key = "_".join(str(key) for key in keys)
MemoryError
```

Comments:
Hi!

200+ data loaders is a lot. Have you tried to reduce the number of datasets by concatenating/interleaving the ones with the same structure/task (the API is `{concatenate_datasets/interleave_datasets}([dset1, ..., dset_N])`)?

Hi @mariosasko, thank you for your reply. I tried pre-concatenating different datasets into one, but one key need is to keep each batch the same data type. Considering that the concatenate-then-segment operation for prefetched samples may span across different data types after concatenating/interleaving (cuz different data sources are mixed), any solution to remain the same data source for each batch?

@cyk1337 have you found any solutions to it?
@mariosasko I tried with interleave_datasets to sample batches from two large datasets (wikipedia alike) and it results in out-of-memory error during data loading (16gpus, >1TB physical memory). Do you have any idea about it?
Title: Docs for creating an audio dataset
Author: stevhliu (MEMBER)
State: closed
Created: 2022-08-23T01:07:09Z
Labels: documentation

This PR is a first draft of how to create audio datasets (`AudioFolder` and loading script). Feel free to let me know if there are any specificities I'm missing for this. 🙂

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Awesome thanks ! I think we can also encourage TAR archives as for image dataset scripts (feel free to copy paste some parts from there lol)

Thanks for all the great feedback @polinaeterna and @lhoestq! 🥰

I added all the other feedback, and I'll look into the `librivox-indonesia` script now!

If you don't mind, I'm taking over this PR since we'll do a release pretty soon

@lhoestq no, I do :D 

haha sorry ^^
Title: Fix: wmt datasets - fix CWMT zh subsets
Author: lhoestq (MEMBER)
State: closed
Created: 2022-08-22T16:42:09Z

Fix https://github.com/huggingface/datasets/issues/4575

TODO: run `datasets-cli test`:
- [x] wmt17
- [x] wmt18
- [x] wmt19

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4871). All of your documentation changes will be reflected on that endpoint.
Title: adding mafand to datasets
Author: dadelani (CONTRIBUTOR)
State: closed
Created: 2022-08-20T15:26:14Z
Labels: wontfix

I'm addding the MAFAND dataset by Masakhane based on the paper/repository below:

Paper: https://aclanthology.org/2022.naacl-main.223/
Code: https://github.com/masakhane-io/lafand-mt

Please, help merge this

Everything works except for creating dummy data file

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @dadelani, thanks for your awesome contribution!!! :heart: 

However, now we are using the Hub to add new datasets, instead of this GitHub repo. 

You could share this dataset under your Hub organization namespace: [Masakhane NLP](https://huggingface.co/masakhane). This way the dataset will be accessible using:
```python
ds = load_dataset("masakhane/mafand")
```

You have the procedure documented in our online docs: 
- [Create a dataset loading script](https://huggingface.co/docs/datasets/dataset_script)
- [Share](https://huggingface.co/docs/datasets/share)

Moreover, datasets shared on the Hub no longer need the dummy data files.

Please, feel free to ping me if you need any further guidance/support.

thank you for the comment. I have moved it to the Hub https://huggingface.co/datasets/masakhane/mafand

Great job, @dadelani!!

Please, note that in the README.md file, the YAML tags should be preceded and followed by three dashes `---`, so that they are properly parsed. See, e.g.: https://raw.githubusercontent.com/huggingface/datasets/main/templates/README.md

Also you could replace the line:
```
# Dataset Card for [Needs More Information]
```
with
```
# Dataset Card for MAFAND-MT
```

Great, thank you for the feedback. I have fixed both issues.
Title: amend docstring for dunder
Author: schafsam (NONE)
State: open
Created: 2022-08-19T19:09:15Z

display dunder method in docsting with underlines an not bold markdown.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4866). All of your documentation changes will be reflected on that endpoint.
Title: Dataset Viewer issue for MoritzLaurer/multilingual_nli
Author: MoritzLaurer (NONE)
State: closed
Created: 2022-08-19T14:55:20Z
Labels: dataset-viewer

### Link

_No response_

### Description

I've just uploaded a new dataset to the hub and the viewer does not work for some reason, see here: https://huggingface.co/datasets/MoritzLaurer/multilingual_nli

It displays the error: 
```
Status code:   400
Exception:     Status400Error
Message:       The dataset does not exist.
```

Weirdly enough the dataviewer works for an earlier version of the same dataset. The only difference is that it is smaller, but I'm not aware of other changes I have made: https://huggingface.co/datasets/MoritzLaurer/multilingual_nli_test

Do you know why the dataviewer is not working?

### Owner

_No response_

Comments:
Thanks for reporting  @MoritzLaurer.

Currently, the dataset preview is working properly: https://huggingface.co/datasets/MoritzLaurer/multilingual_nli

Please note that when a dataset is modified, it might take some time until the preview is completely updated.

@severo might it be worth adding a clearer error message, something like "The preview is updating, please retry later"?

Thanks for your response. You are right, its now working well. I had waited for 30 min or so and refreshed several times and thought there was some other error. Yeah, a different error message sounds like a good idea to avoid confusion. 

I'm closing this issue then.

> @severo might it be worth adding a clearer error message, something like "The preview is updating, please retry later"?

Yes, it's a known issue, and we're about to ship a better version
Title: Allow pathlib PoxisPath in Dataset.read_json
Author: changjonathanc (CONTRIBUTOR)
State: open
Created: 2022-08-19T12:59:17Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
```
from pathlib import Path
from datasets import Dataset
ds = Dataset.read_json(Path('data.json'))
```
causes an error
```
AttributeError: 'PosixPath' object has no attribute 'decode'
```

**Describe the solution you'd like**

It should be able to accept PosixPath and read the json from inside.

Comments:
This same error will occur using `ds = datasets.load_dataset('json', data_files=['test.jsonl'])`

@cccntu I want to make a quick fix for this, but I am struggling to find where the json dataset builder is. Do you know?

@vvvm23 I think you mean think:
```python
ds = datasets.load_dataset('json', data_files=[Path('test.jsonl')])
```
And the place you want to modify is here:
```
utils/file_utils.py:64, in is_remote_url(url_or_filename)
     63 def is_remote_url(url_or_filename: str) -> bool:
---> 64     parsed = urlparse(url_or_filename)
     65     return parsed.scheme in ("http", "https", "s3", "gs", "hdfs", "ftp")
```

Probably just need to check first if `url_or_filename` is [PathLike](https://docs.python.org/3/library/os.html#os.PathLike) and return False early.

Btw, I tried installing from main, and ran my code above and got a different error. Probably because the API have changed.
`AttributeError: module 'datasets' has no attribute 'read_json'`


> @vvvm23 I think you mean think:

You are correct, thanks!

> Probably just need to check first if url_or_filename is [PathLike](https://docs.python.org/3/library/os.html#os.PathLike) and return False early.

Is PathLike sufficient, or should I check the file exists here? Or is that handled later?

I think here we just want to avoid passing Path to urlparse. A simpler solution is to add a str() call and convert the input to string before passing to the next step. No need to check anything.

Above PR should do your first suggestion. Hope that works for you, as I am going on holiday and won't be able to change much :wink: 

I just came across this problem with `DatasetDict.from_parquet()`. The error still seems to originate from the same place:
```
  File "/usr/local/lib/python3.12/dist-packages/datasets/utils/file_utils.py", line 88, in is_relative_path
    return urlparse(url_or_filename).scheme == "" and not os.path.isabs(url_or_filename)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
```
The PR looks like it would still solve everything. Is there a reason this got delayed?
Title: TFDS wiki_dialog dataset to Huggingface dataset
Author: djaym7 (NONE)
State: closed
Created: 2022-08-18T23:06:30Z
Labels: dataset request

## Adding a Dataset
- **Name:** *Wiki_dialog*
- **Description: https://github.com/google-research/dialog-inpainting#:~:text=JSON%20object%2C%20for-,example,-%3A
- **Paper: https://arxiv.org/abs/2205.09073
- **Data: https://github.com/google-research/dialog-inpainting
- **Motivation:** *Research and Development on biggest corpus of dialog data*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).

Comments:
@albertvillanova any help ? The linked dataset is in beam format which is similar to wikipedia dataset in huggingface that you  scripted..

Nvm, I was able to port it to huggingface datasets, will upload to the hub soon

https://huggingface.co/datasets/djaym7/wiki_dialog

Thanks for the addition, @djaym7.
Title: Using disk for memory with the method `from_dict`
Author: HugoLaurencon (NONE)
State: open
Created: 2022-08-18T15:18:18Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
I start with an empty dataset. In a loop, at each iteration, I create a new dataset with the method `from_dict` (based on some data I load) and I concatenate this new dataset with the one at the previous iteration. After some iterations, I have an OOM error.

**Describe the solution you'd like**
The method `from_dict` loads the data in RAM. It could be good to add an option to use the disk instead.

**Describe alternatives you've considered**
To solve the problem, I have to do an intermediate step where I save the new datasets at each iteration with `save_to_disk`. Once it's done, I open them all and concatenate them.

Comments:
This issue was also causing an OOM in @nateraw 's workflow and shows again that behavior is confusing - we should definitely switch to using the disk IMO
Title: can't install using conda on Windows 10
Author: xoffey (NONE)
State: open
Created: 2022-08-17T19:57:37Z
Labels: bug

## Describe the bug
I wanted to install using conda or Anaconda navigator. That didn't work, so I had to install using pip.

## Steps to reproduce the bug
conda install -c huggingface -c conda-forge datasets

## Expected results
Should have indicated successful installation.

## Actual results
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
... took forever, so I cancelled it with ctrl-c

## Environment info
- `datasets` version: 2.4.0  # after installing with pip
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.9.12
- PyArrow version: 9.0.0
- Pandas version: 1.4.2
- conda version: 4.13.0

conda info

     active environment : base
    active env location : G:\anaconda2022
            shell level : 1
       user config file : C:\Users\michael\.condarc
 populated config files : C:\Users\michael\.condarc
          conda version : 4.13.0
    conda-build version : 3.21.8
         python version : 3.9.12.final.0
       virtual packages : __cuda=11.1=0
                          __win=0=0
                          __archspec=1=x86_64
       base environment : G:\anaconda2022  (writable)
      conda av data dir : G:\anaconda2022\etc\conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/pytorch/win-64
                          https://conda.anaconda.org/pytorch/noarch
                          https://conda.anaconda.org/huggingface/win-64
                          https://conda.anaconda.org/huggingface/noarch
                          https://conda.anaconda.org/conda-forge/win-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/anaconda-fusion/win-64
                          https://conda.anaconda.org/anaconda-fusion/noarch
                          https://repo.anaconda.com/pkgs/main/win-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/win-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/msys2/win-64
                          https://repo.anaconda.com/pkgs/msys2/noarch
          package cache : G:\anaconda2022\pkgs
                          C:\Users\michael\.conda\pkgs
                          C:\Users\michael\AppData\Local\conda\conda\pkgs
       envs directories : G:\anaconda2022\envs
                          C:\Users\michael\.conda\envs
                          C:\Users\michael\AppData\Local\conda\conda\envs
               platform : win-64
             user-agent : conda/4.13.0 requests/2.27.1 CPython/3.9.12 Windows/10 Windows/10.0.19044
          administrator : False
             netrc file : None
           offline mode : False

Comments:
Title: map() function removes columns when input_columns is not None
Author: pramodith (NONE)
State: closed
Created: 2022-08-16T20:42:30Z
Labels: bug

## Describe the bug
The map function, removes features from the dataset that are not present in the _input_columns_ list of columns, despite the columns being removed not mentioned in the _remove_columns_ argument.
## Steps to reproduce the bug
```python
from datasets import Dataset
ds = Dataset.from_dict({"a" : [1,2,3],"b" : [0,1,0], "c" : [2,4,5]})

def double(x,y):
  x = x*2
  y = y*2
  return {"d" : x, "e" : y}

ds.map(double, input_columns=["a","c"])
```

## Expected results
```
Dataset({
    features: ['a', 'b', 'c', 'd', 'e'],
    num_rows: 3
})
```
## Actual results
```
Dataset({
    features: ['a', 'c', 'd', 'e'],
    num_rows: 3
})
```

In this specific example feature **b** should not be removed.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: linux (colab)
- Python version: 3.7.13
- PyArrow version: 6.0.1

Comments:
Hi! Thanks for reporting! This looks like a bug. I've just opened a PR with the fix.

Awesome! Thank you. I'll close the issue once the PR gets merged. :-)

I guess we should reopen after the revert by:
- #5006
Title: No preprocessed wikipedia is working on huggingface/datasets
Author: aninrusimha (NONE)
State: closed
Created: 2022-08-16T13:55:33Z
Labels: bug

## Describe the bug
20220301 wikipedia dump has been deprecated, so now there is no working wikipedia dump on huggingface

https://huggingface.co/datasets/wikipedia

https://dumps.wikimedia.org/enwiki/

Comments:
Thanks for reporting @aninrusimha.

Please, note that the preprocessed datasets are still available, as described in the dataset card, e.g.: https://huggingface.co/datasets/wikipedia
```python
ds = load_dataset("wikipedia", "20220301.en")
``` 

This is working now, but I was getting an error a few days ago when running an existing script. Unfortunately I did not do a proper bug report, but for some reason I was unable to load the dataset due to a request being made to the wikimedia website. However, its working now. Thanks for the reply!
Title: file missing when load_dataset with openwebtext on windows
Author: xi-loong (NONE)
State: closed
Created: 2022-08-16T04:04:22Z
Labels: bug

## Describe the bug
0015896-b1054262f7da52a0518521e29c8e352c.txt is missing when I run run_mlm.py with openwebtext. I check the cache_path and can not find 0015896-b1054262f7da52a0518521e29c8e352c.txt. but I can find this file in the 17ecf461bfccd469a1fbc264ccb03731f8606eea7b3e2e8b86e13d18040bf5b3/urlsf_subset00-16_data.xz  with 7-zip.

## Steps to reproduce the bug
```sh
python run_mlm.py --model_type roberta --tokenizer_name roberta-base --dataset_name openwebtext --per_device_train_batch_size 8 --per_device_eval_batch_size 8 --do_train --do_eval --output_dir F:/model/roberta-base
```
or 
```python
from datasets import load_dataset
load_dataset("openwebtext", None, cache_dir=None, use_auth_token=None)
```

## Expected results
Loading is successful
## Actual results
Traceback (most recent call last):
  File "D:\Python\v3.8.5\lib\site-packages\datasets\builder.py", line 704, in download_and_prepare
    self._download_and_prepare(
  File "D:\Python\v3.8.5\lib\site-packages\datasets\builder.py", line 1227, in _download_and_prepare
    super()._download_and_prepare(dl_manager, verify_infos, check_duplicate_keys=verify_infos)
  File "D:\Python\v3.8.5\lib\site-packages\datasets\builder.py", line 795, in _download_and_prepare
    raise OSError(
OSError: Cannot find data file. 
Original error:
[Errno 22] Invalid argument: 'F://huggingface/datasets/downloads/extracted/0901d27f43b7e9ac0577da0d0061c8c632ba0b70ecd1b4bfb21562d9b7486faa/0015896-b1054262f7da52a0518521e29c8e352c.txt'

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.4.0
- Platform: windows
- Python version: 3.8.5 
- PyArrow version: 9.0.0

Comments:
I have tried to extract ```0015896-b1054262f7da52a0518521e29c8e352c.txt``` from ```17ecf461bfccd469a1fbc264ccb03731f8606eea7b3e2e8b86e13d18040bf5b3/urlsf_subset00-16_data.xz``` with 7-zip
and put the file into cache_path ```F://huggingface/datasets/downloads/extracted/0901d27f43b7e9ac0577da0d0061c8c632ba0b70ecd1b4bfb21562d9b7486faa```
there is still raise the same error and I  find the file  was removed from cache_path after I run the run_mlm.py with ```python run_mlm.py --model_type roberta --tokenizer_name roberta-base --dataset_name openwebtext --per_device_train_batch_size 8 --per_device_eval_batch_size 8 --do_train --do_eval --output_dir F:/model/roberta-base```.
Title: Dataset Viewer issue for super_glue
Author: wzsxxa (NONE)
State: closed
Created: 2022-08-16T01:34:56Z
Labels: dataset-viewer

### Link

https://huggingface.co/datasets/super_glue

### Description

can't view super_glue dataset on the web page

### Owner

_No response_

Comments:
Thanks for reporting @wzsxxa.

However the "super_glue" dataset is rendered properly by the Dataset preview: https://huggingface.co/datasets/super_glue
Title: Fix test of _get_extraction_protocol for TAR files
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-08-15T08:37:58Z

While working in another PR, I discovered an xpass test (a test that is supposed to xfail but nevertheless passes) when testing `_get_extraction_protocol`: https://github.com/huggingface/datasets/runs/7818845285?check_suite_focus=true
```
XPASS tests/test_streaming_download_manager.py::test_streaming_dl_manager_get_extraction_protocol_throws[https://foo.bar/train.tar] 
```

This PR:
- refactors the test so that it tests the raise of the exceptions instead of xfailing
- fixes the test for TAR files: it does not raise an exception, but returns "tar"
- fixes some tests wrongly named: exchange `test_streaming_dl_manager_get_extraction_protocol` with `test_streaming_dl_manager_get_extraction_protocol_gg_drive`

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Update documentation card of miam dataset
Author: PierreColombo (CONTRIBUTOR)
State: closed
Created: 2022-08-13T14:38:55Z

Hi !
Paper has been published at EMNLP.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Ahahah :D not sur how i broke something by updating the README :D 

Thanks for the fix @PierreColombo. 

Once a README is modified, our CI runs tests on it, requiring additional quality fixes, so that all READMEs are progressively improved and have some minimal tags/sections/information.

For this specific README file, the additional quality requirements of the CI are: https://github.com/huggingface/datasets/runs/7819924428?check_suite_focus=true
```
E           The following issues were found for the README at `/home/runner/work/datasets/datasets/datasets/miam/README.md`:
E           -	Section `Additional Information` is missing subsection: `Dataset Curators`.
E           -	Section `Additional Information` is missing subsection: `Contributions`.
E           -	`Additional Information` has an extra subsection: `Benchmark Curators`. Skipping further validation checks for this subsection as expected structure is unknown.
```

Thanks a lot Albert :)))
Title: Mark CI tests as xfail if Hub HTTP error
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-08-13T10:45:11Z

In order to make testing more robust (and avoid merges to master with red tests), we could mark tests as xfailed (instead of failed) when the Hub raises some temporary HTTP errors.

This PR:
- marks tests as xfailed only if the Hub raises a 500 error for:
  - test_upstream_hub
- makes pytest report the xfailed/xpassed tests.

More tests could also be marked if needed.

Examples of CI failures due to temporary Hub HTTP errors:
- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_multiple_files
  - https://github.com/huggingface/datasets/runs/7806855399?check_suite_focus=true
    `requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://hub-ci.huggingface.co/api/datasets/__DUMMY_TRANSFORMERS_USER__/test-16603108028233/commit/main (Request ID: aZeAQ5yLktoGHQYBcJ3zo)`
- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_no_token
  - https://github.com/huggingface/datasets/runs/7840022996?check_suite_focus=true
    `requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://s3.us-east-1.amazonaws.com/lfs-staging.huggingface.co/repos/81/e3/81e3b831fa9bf23190ec041f26ef7ff6d6b71c1a937b8ec1ef1f1f05b508c089/caae596caa179cf45e7c9ac0c6d9a9cb0fe2d305291bfbb2d8b648ae26ed38b6?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA4N7VTDGOZQA2IKWK%2F20220815%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220815T144713Z&X-Amz-Expires=900&X-Amz-Signature=5ddddfe8ef2b0601e80ab41c78a4d77d921942b0d8160bcab40ff894095e6823&X-Amz-SignedHeaders=host&x-id=PutObject`
- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_dict_to_hub_private
  - https://github.com/huggingface/datasets/runs/7835921082?check_suite_focus=true
    `requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://hub-ci.huggingface.co/api/repos/create (Request ID: gL_1I7i2dii9leBhlZen-) - Internal Error - We're working hard to fix that as soon as possible!`
- FAILED tests/test_upstream_hub.py::TestPushToHub::test_push_dataset_to_hub_custom_features_image_list
  - https://github.com/huggingface/datasets/runs/7835920900?check_suite_focus=true
  - This is not 500, but 404:
    `requests.exceptions.HTTPError: 404 Client Error: Not Found for url: [https://hub-ci.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/test-16605586458339.git/info/lfs/objects](https://hub-ci.huggingface.co/datasets/__DUMMY_TRANSFORMERS_USER__/test-16605586458339.git/info/lfs/objects/batch)`

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Add 'val' to  VALIDATION_KEYWORDS. 
Author: akt42 (CONTRIBUTOR)
State: closed
Created: 2022-08-13T06:49:41Z

This PR fixes #4839 by adding the word `"val"` to the `VALIDATION_KEYWORDS` so that the `load_dataset()` method with `imagefolder` (and probably, some other directives as well) reads folders named `"val"` as well.

I think the supported keywords have to be mentioned in the documentation as well, but I couldn't think of a proper place to add that.

Comments:
@mariosasko not sure about how the reviewing process works. Maybe you can have a look because we discussed this elsewhere?

Hi, thanks! 

Let's add one pattern with `val` to this test before merging: 
https://github.com/huggingface/datasets/blob/b88a656cf94c4ad972154371c83c1af759fde522/tests/test_data_files.py#L598

_The documentation is not available anymore as the PR was closed or merged._

@akt42 note that there is some info about splits keywords in the docs: https://huggingface.co/docs/datasets/main/en/repository_structure#split-names-keywords. I agree it's not clear that it applies not only to filenames, but to directories as well.

I think "val" should be now added to the documentation source file here: https://github.com/huggingface/datasets/blob/main/docs/source/repository_structure.mdx?plain=1#L98

@polinaeterna Thanks for notifying us that there is a list of supported keywords

I've added "val" to that list and a test.
Title: Update ted_talks_iwslt license to include ND
Author: cakiki (CONTRIBUTOR)
State: closed
Created: 2022-08-12T16:14:52Z

Excerpt from the paper's abstract: "Aside from its cultural and social relevance, this content, which is published under the Creative Commons BY-NC-ND license, also represents a precious language resource for the machine translation research community"

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Dataset Viewer issue for darragh/demo_data_raw3
Author: severo (COLLABORATOR)
State: open
Created: 2022-08-12T15:22:58Z

### Link

https://huggingface.co/datasets/darragh/demo_data_raw3

### Description

```
Exception:     ValueError
Message:       Arrow type extension<arrow.py_extension_type<pyarrow.lib.UnknownExtensionType>> does not have a datasets dtype equivalent.
```
reported by @NielsRogge 

### Owner

No

Comments:
do you have an idea of why it can occur @huggingface/datasets? The dataset consists of a single parquet file.

Thanks for reporting @severo.

I'm not able to reproduce that error. I get instead:
```
FileNotFoundError: [Errno 2] No such file or directory: 'orix/data/ChiSig/唐合乐-9-3.jpg'
```

Which pyarrow version are you using? Mine is 6.0.1. 

OK, I get now your error when not streaming.

OK!

If it's useful, the pyarrow version is 7.0.0:

https://github.com/huggingface/datasets-server/blob/487c39d87998f8d5a35972f1027d6c8e588e622d/services/worker/poetry.lock#L1537-L1543

Apparently, there is something weird with that Parquet file: its schema is:
```
images: extension<arrow.py_extension_type<pyarrow.lib.UnknownExtensionType>>
```

I have forced a right schema:
```python
from datasets import Features, Image, load_dataset

features = Features({"images": Image()})
ds = datasets.load_dataset("parquet", split="train", data_files="train-00000-of-00001.parquet", features=features)
```
and then recreated a new Parquet file:
```python
ds.to_parquet("train.parquet")
```

Now this Parquet file has the right schema:
```
images: struct<bytes: binary, path: string>
  child 0, bytes: binary
  child 1, path: string
```
and can be loaded normally:
```python
In [26]: ds = load_dataset("parquet", split="train", data_files="dataset.parquet")
n [27]: ds
Out[27]: 
Dataset({
    features: ['images'],
    num_rows: 20
})
```