sunny2309/githug-issues huggingface dataset:
load_dataset save_to_disk load_from_disk error

### Describe the bug

datasets version： 2.10.1
I `load_dataset `and `save_to_disk` sucessfully on windows10( **and I `load_from_disk(/LLM/data/wiki)` succcesfully on windows10**), and I copy the dataset `/LLM/data/wiki`
into a ubuntu system, but when I `load_from_disk(/LLM/data/wiki)` on ubuntu,  something weird happens:


```
load_from_disk('/LLM/data/wiki')
  File "/usr/local/miniconda3/lib/python3.8/site-packages/datasets/load.py", line 1874, in load_from_disk
    return DatasetDict.load_from_disk(dataset_path, keep_in_memory=keep_in_memory, storage_options=storage_options)
  File "/usr/local/miniconda3/lib/python3.8/site-packages/datasets/dataset_dict.py", line 1309, in load_from_disk
    dataset_dict[k] = Dataset.load_from_disk(
  File "/usr/local/miniconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1543, in load_from_disk
    fs_token_paths = fsspec.get_fs_token_paths(dataset_path, storage_options=storage_options)
  File "/usr/local/miniconda3/lib/python3.8/site-packages/fsspec/core.py", line 610, in get_fs_token_paths
    chain = _un_chain(urlpath0, storage_options or {})
  File "/usr/local/miniconda3/lib/python3.8/site-packages/fsspec/core.py", line 325, in _un_chain
    cls = get_filesystem_class(protocol)
  File "/usr/local/miniconda3/lib/python3.8/site-packages/fsspec/registry.py", line 232, in get_filesystem_class
    raise ValueError(f"Protocol not known: {protocol}")
ValueError: Protocol not known: /LLM/data/wiki
```
It seems that something went wrong on the arrow file?
How can I solve this , since currently I can not save_to_disk on ubuntu system

### Steps to reproduce the bug

datasets version： 2.10.1

### Expected behavior

datasets version： 2.10.1

### Environment info

datasets version： 2.10.1
Error loading wikitext data    raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).__name__} is not supported.")

I was trying to load  the wiki dataset, but i got this error 

    traindata = load_dataset('wikitext', 'wikitext-2-raw-v1', split='train')
  File "/home/aelkordy/.conda/envs/prune_llm/lib/python3.9/site-packages/datasets/load.py", line 1804, in load_dataset
    ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory)
  File "/home/aelkordy/.conda/envs/prune_llm/lib/python3.9/site-packages/datasets/builder.py", line 1108, in as_dataset
    raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).__name__} is not supported.")
NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
Different objects are returned from calls that should be returning the same kind of object.

### Describe the bug

        1.  dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir, split='train[:1%]')
        2.  dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=training_args.cache_dir)

The only difference I would expect these calls to have is the size of the dataset.

But, while 2. returns a dictionary with "train" key in it, 1. returns a dataset WITHOUT any initial "train" keyword.

Both calls are to be used within exactly the same context. They should return identically structured datasets of different size.

### Steps to reproduce the bug

See above.

### Expected behavior

Expect both calls to return the same structured Dataset structure but with different number of elements, i.e. call 1. should have 1% of the data of the call 2.0

### Environment info

Ubuntu 20.04
gcc 9.x.x.

It is really irrelevant.
Can't load ds = load_dataset("imdb")

### Describe the bug

I did `from datasets import load_dataset, load_metric` and then `ds = load_dataset("imdb")` and it gave me the error:
ExpectedMoreDownloadedFiles: {'http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz'}

I tried doing `ds = load_dataset("imdb",download_mode="force_redownload")` as well as reinstalling dataset. I still face this problem.

### Steps to reproduce the bug

1. from datasets import load_dataset, load_metric
2. ds = load_dataset("imdb")

### Expected behavior

It should load and give me this when I run `ds`

DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 25000
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 25000
    })
    unsupervised: Dataset({
        features: ['text', 'label'],
        num_rows: 50000
    })
})

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-5.4.0-164-generic-x86_64-with-glibc2.17
- Python version: 3.8.18
- Huggingface_hub version: 0.16.2
- PyArrow version: 13.0.0
- Pandas version: 2.0.2
Incorrect example code in 'Create a dataset' docs

### Describe the bug

On [this](https://huggingface.co/docs/datasets/create_dataset) page, the example code for loading in images and audio is incorrect.

Currently, examples are:

``` python
from datasets import ImageFolder
dataset = load_dataset("imagefolder", data_dir="/path/to/pokemon")
```
and 
``` python
from datasets import AudioFolder
dataset = load_dataset("audiofolder", data_dir="/path/to/folder")
```

I'm pretty sure the imports are wrong and should be:

``` python 
from datasets import load_dataset
dataset = load_dataset("audiofolder", data_dir="/path/to/folder")
```

I am happy to update this if this is right but just wanted to check before making any changes.

### Steps to reproduce the bug

Go to https://huggingface.co/docs/datasets/create_dataset

### Expected behavior

N/A

### Environment info

N/A
Fix UnboundLocalError if preprocessing returns an empty list

If this tokenization function is used with IterableDatasets and no sample is as big as the context length, `input_batch` will be an empty list.
```
def tokenize(batch, tokenizer, context_length):
    outputs = tokenizer(
        batch["text"],
        truncation=True,
        max_length=context_length,
        return_overflowing_tokens=True,
        return_length=True
    )
    input_batch = []
    for length, input_ids in zip(outputs["length"], outputs["input_ids"]):
        if length == context_length:
            input_batch.append(input_ids)
    return {"input_ids": input_batch}

dataset.map(tokenize, batched=True, batch_size=batch_size, fn_kwargs={"context_length": context_length, "tokenizer": tokenizer}, remove_columns=dataset.column_names)
```

This will throw the following error: UnboundLocalError: local variable 'batch_idx' referenced before assignment, because the for loop was not executed a single time

```
for batch_idx, example in enumerate(_batch_to_examples(transformed_batch)):
     yield new_key, example
current_idx += batch_idx + 1
```

Some of the possible solutions

```
for batch_idx, example in enumerate(_batch_to_examples(transformed_batch)):
     yield new_key, example
try:
     current_idx += batch_idx + 1
except:
     current_idx += 1
```

or 

```
batch_idx = 0
for batch_idx, example in enumerate(_batch_to_examples(transformed_batch)):
     yield new_key, example
current_idx += batch_idx + 1
```
Pin supported upper version of fsspec

Pin upper version of `fsspec` to avoid disruptions introduced by breaking changes (and the need of urgent patch releases with hotfixes) on each release on their side. See:
- #6331
- #6210 
- #5731
- #5617
- #5447

I propose that we explicitly test, introduce fixes and support each new `fsspec` version release.

CC: @LysandreJik
Replace deprecated license_file in setup.cfg

Replace deprecated license_file in `setup.cfg`.

See: https://github.com/huggingface/datasets/actions/runs/6610930650/job/17953825724?pr=6331
```
      /tmp/pip-build-env-a51hls20/overlay/lib/python3.8/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!
      
              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************
      
      !!
```
Temporarily pin fsspec < 2023.10.0

Temporarily pin fsspec < 2023.10.0 until permanent solution is found.

Hot fix #6330.

See: https://github.com/huggingface/datasets/actions/runs/6610904287/job/17953774987
```
...
ERROR tests/test_iterable_dataset.py::test_iterable_dataset_from_file - NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
= 373 failed, 2055 passed, 17 skipped, 8 warnings, 6 errors in 228.14s (0:03:48) =
```
Latest fsspec==2023.10.0 issue with streaming datasets

### Describe the bug

Loading a streaming dataset with this version of fsspec fails with the following error:

`NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet.`

I suspect the issue is with this PR 

https://github.com/fsspec/filesystem_spec/pull/1381

### Steps to reproduce the bug

1. Upgrade fsspec to version `2023.10.0`
2. Attempt to load a streaming dataset e.g. `load_dataset("laion/gpt4v-emotion-dataset", split="train", streaming=True)`
3. Observe the following exception:

```
  File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/datasets/load.py", line 2146, in load_dataset
    return builder_instance.as_streaming_dataset(split=split)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/datasets/builder.py", line 1318, in as_streaming_dataset
    raise NotImplementedError(
NotImplementedError: Loading a streaming dataset cached in a LocalFileSystem is not supported yet.
```

### Expected behavior

Should stream the dataset as normal.

### Environment info

datasets@main
fsspec==2023.10.0
FileNotFoundError when trying to load the downloaded dataset with `load_dataset(..., streaming=True)`

### Describe the bug

Hi, I'm trying to load the dataset `togethercomputer/RedPajama-Data-1T-Sample` with `load_dataset` in streaming mode, i.e., `streaming=True`, but `FileNotFoundError` occurs.

### Steps to reproduce the bug

I've downloaded the dataset and save it to the cache dir in advance. My hope is loading the files in offline environment and without taking too much hours to prepross the entire data before running into the training process.
So I try the following code to load the files streamingly
```py
dataset = load_dataset('togethercomputer/RedPajama-Data-1T-Sample', streaming=True)
print(next(iter(dataset['train'])))
```
Sadly, it raises the following:
```
FileNotFoundError: [Errno 2] No such file or directory: 'CURRENT_CODE_PATH/arxiv_sample.jsonl'
```
I've noticed that the dataset can be properly found in the begining
```
Using the latest cached version of the module from /root/.cache/huggingface/modules/datasets_modules/datasets/togethercomputer--RedPajama-Data-1T-Sample/6ea3bc8ec2e84ec6d2df1930942e9028ace8c5b9d9143823cf911c50bbd92039 (last modified on Sat Oct 21 20:12:57 2023) since it couldn't be found locally at togethercomputer/RedPajama-Data-1T-Sample., or remotely on the Hugging Face Hub.
```
But it seems that the paths couldn't be properly parsed when loading iteratively.
 
How should I fix this error. I've tried specifying `data_files` or `data_dir` as `.../arxiv_sample.jsonl` but none of them works.

Thanks.

### Expected behavior

Properly load the dataset.

### Environment info

`datasets==2.14.5`
Conversion to Arrow fails due to wrong type heuristic

### Describe the bug

I have a list of dictionaries with valid/JSON-serializable values. 

One key is the denominator for a paragraph. In 99.9% of cases its a number, but there are some occurences of '1a', '2b' and so on.

If trying to convert this list to a dataset with `Dataset.from_list()`, I always get
`ArrowInvalid: Could not convert '1' with type str: tried to convert to int64`, presumably because pyarrow tries to convert the keys to integers.

Is there any way to circumvent this and fix dtypes? I didn't find anything in the documentation.

### Steps to reproduce the bug

* create a list of dicts with one key being a string of an integer for the first few thousand occurences and try to convert to dataset.


### Expected behavior

There shouldn't be an error (e.g. some flag to turn off automatic str to numeric conversion).

### Environment info

- `datasets` version: 2.14.5
- Platform: Linux-5.15.0-84-generic-x86_64-with-glibc2.35
- Python version: 3.9.18
- Huggingface_hub version: 0.17.3
- PyArrow version: 13.0.0
- Pandas version: 2.1.1
Fix regex `get_data_files` formatting for base paths

With this pr https://github.com/huggingface/datasets/pull/6309, it is formatting the entire base path into regex, which results in the undesired formatting error `doesn't match the pattern` because of the line in `glob_pattern_to_regex`: `.replace("//", "/")`:
- Input: `hf://datasets/...`
- Output: `hf:/datasets/...`

This fix will only convert the `split_pattern` to regex and keep the `base_path` unchanged.

cc @albertvillanova hopefully this still works with your implementation
Dataset slice splits can't load training and validation at the same time

### Describe the bug

According to the [documentation](https://huggingface.co/docs/datasets/v2.14.5/loading#slice-splits) is should be possible to run the following command:

`train_test_ds = datasets.load_dataset("bookcorpus", split="train+test")`

to load the train and test sets from the dataset. 

However executing the equivalent code:

`speech_commands_v1 = load_dataset("superb", "ks", split="train+test")`

only yields the following output:

> Dataset({
>     features: ['file', 'audio', 'label'],
>     num_rows: 54175
> })

Where loading the dataset without the split argument yields:

> DatasetDict({
>     train: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 51094
>     })
>     validation: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 6798
>     })
>     test: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 3081
>     })
> })

Thus, the API seems to be broken in this regard. 

This is a bit annoying since I want to be able to use the split argument with `split="train[:10%]+test[:10%]"` to have smaller dataset to work with when validating my model is working correctly. 

### Steps to reproduce the bug

`speech_commands_v1 = load_dataset("superb", "ks", split="train+test")`

### Expected behavior

> DatasetDict({
>     train: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 51094
>     })
>     test: Dataset({
>         features: ['file', 'audio', 'label'],
>         num_rows: 3081
>     })
> })

### Environment info

```
import datasets
print(datasets.__version__)
```
> 2.14.5

```
import sys
print(sys.version)
```
> 3.9.17 (main, Jul  5 2023, 20:41:20) 
> [GCC 11.2.0]
Datasets.map is severely broken

### Describe the bug

Regardless of how many cores I used, I have 16 or 32 threads, map slows down to a crawl at around 80% done, lingers maybe until 97% extremely slowly and NEVER finishes the job. It just hangs.

After watching this for 27 hours I control-C out of it. Until the end one process appears to be doing something, but it never ends.

I saw some comments about fast tokenizers using Rust and all and tried different variations. NOTHING works.



### Steps to reproduce the bug

Running it without breaking the dataset into parts results in the same behavior. The loop was an attempt to see if this was a RAM issue. 

for idx in range(100):
    dataset = load_dataset("togethercomputer/RedPajama-Data-1T-Sample", cache_dir=cache_dir, split=f'train[{idx}%:{idx+1}%]')
    dataset = dataset.map(partial(tokenize_fn, tokenizer), batched=False, num_proc=1, remove_columns=["text", "meta"])
    dataset.save_to_disk(training_args.cache_dir + f"/training_data_{idx}")


### Expected behavior

I expect map to run at more or less the same speed it starts with and FINISH its processing. 

### Environment info

Python 3.8, same with 3.10 makes no difference.
Ubuntu 20.04,
sentiment140 dataset unavailable

### Describe the bug

loading the dataset using load_dataset("sentiment140") returns the following error
ConnectionError: Couldn't reach http://cs.stanford.edu/people/alecmgo/trainingandtestdata.zip (error 403)

### Steps to reproduce the bug

Run the following code (version should not matter).

```
from datasets import load_dataset
data = load_dataset("sentiment140")
```

### Expected behavior

The dataset should be loaded just like any other.
The main issue is that it is no longer hosted by stanford. It is still available from a [Google Drive Link](https://docs.google.com/file/d/0B04GJPshIjmPRnZManQwWEdTZjg/edit).

### Environment info

- `datasets` version: 2.14.5
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.10.8
- Huggingface_hub version: 0.17.3    
- PyArrow version: 13.0.0
- Pandas version: 2.1.1
Fix loading Hub datasets with CSV metadata file

Currently, the reading of the metadata file infers the file extension (.jsonl or .csv) from the passed filename. However, downloaded files from the Hub don't have file extension. For example:
- the original file: `hf://datasets/__DUMMY_TRANSFORMERS_USER__/test-dataset-5916a4-16977085077831/metadata.jsonl`
- corresponds to the downloaded path: `/tmp/pytest-of-username/pytest-46/cache/datasets/downloads/9f5374dbb470f711f6b89d66a5eec1f19cc96324b26bcbebe29138bda6cb20e6`, which does not have extension

In the case where the metadata file does not have an extension, the reader assumes it is a JSONL file, thus the reported error when trying to read a CSV file as a JSONL one: `ArrowInvalid: JSON parse error: Invalid value. in row 0`

This behavior was introduced by:
- #4837

This PR extracts the metadata file extension from the original filename (instead of the downloaded one) and passes it as a parameter to the read_metadata function.

Fix #6315.
Fix commit message formatting in multi-commit uploads

Currently, the commit message keeps on adding:

- `Upload dataset (part 00000-of-00002)`
- `Upload dataset (part 00000-of-00002) (part 00001-of-00002)`

Introduced in https://github.com/huggingface/datasets/pull/6269

This PR fixes this issue to have

- `Upload dataset (part 00000-of-00002)`
- `Upload dataset (part 00001-of-00002)`
docs: resolving namespace conflict, refactored variable 

In docs of about_arrow.md, in the below example code
![image](https://github.com/huggingface/datasets/assets/74114936/fc70e152-e15f-422e-949a-1c4c4c9aa116)
The variable name 'time' was being used in a way that could potentially lead to a namespace conflict with Python's built-in 'time' module. It is not a good convention and can lead to unintended variable shadowing for any user re-using the example code.
To ensure code clarity, and prevent potential naming conflicts renamed the variable 'time' to 'elapsed_time' in the example  code.
Add return_file_name in load_dataset

Proposition to fix #5806.

Added an optional parameter `return_file_name` in the dataset builder config. When set to `True`, the function will include the file name corresponding to the sample in the returned output.

There is a difference between arrow-based and folder-based datasets to return the file name:
- for arrow-based: a column is concatenated after the table is cast.
- for folder-based: `dataset.info.features` has the entry `file_name` and the original file name is passed to the `sample_metadata` dictionary. 

The difference in behavior might be a concern, also I do not know whether the `file_name` should return the original file path or the downloaded one for folder-based datasets.

I added some tests for the datasets that already had a test file.
Fix get_data_patterns for directories with the word data twice

Before the fix, `get_data_patterns` inferred wrongly the split name for paths with the word "data" twice:
- For the URL path: `hf://datasets/piuba-bigdata/articles_and_comments@f328d536425ae8fcac5d098c8408f437bffdd357/data/train-00001-of-00009.parquet` (note the org name `piuba-bigdata/` ending with `data/`)
- The inferred split name was: `articles_and_comments@f328d536425ae8fcac5d098c8408f437bffdd357/data/train` instead of `train`

This PR fixes this issue by passing the `base_path` (`hf://datasets/piuba-bigdata/articles_and_comments@f328d536425ae8fcac5d098c8408f437bffdd357`) to `_get_data_files_patterns` and prepending it to the regex split pattern (`data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9].*\\..*`).

Fix #6305.
Fix https://huggingface.co/datasets/piuba-bigdata/articles_and_comments/discussions/1
module 'resource' has no attribute 'error'

### Describe the bug

just run import:
`from datasets import load_dataset`

and then:
```

  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\__init__.py", line 22, in <module>
    from .arrow_dataset import Dataset
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\arrow_dataset.py", line 66, in <module>
    from .arrow_reader import ArrowReader
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\arrow_reader.py", line 30, in <module>
    from .download.download_config import DownloadConfig
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\download\__init__.py", line 10, in <module>
    from .streaming_download_manager import StreamingDownloadManager
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\download\streaming_download_manager.py", line 21, in <module>
    from ..filesystems import COMPRESSION_FILESYSTEMS
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\datasets\filesystems\__init__.py", line 8, in <module>
    import fsspec.asyn
  File "C:\ProgramData\anaconda3\envs\py310\lib\site-packages\fsspec\asyn.py", line 157, in <module>
    ResourceEror = resource.error
AttributeError: module 'resource' has no attribute 'error'

Process finished with exit code 1

```


and the error codes are:
```
try:
    import resource
except ImportError:
    resource = None
    ResourceError = OSError
else:
    ResourceEror = resource.error

```
1. miss spelling : "ResourceEror " should be "ResourceErorr"
2. module 'resource' has no attribute 'error'




### Steps to reproduce the bug

only one step:

`from datasets import load_dataset`

### Expected behavior

slove error: module 'resource' has no attribute 'error'

### Environment info

python=3.10

datasets==2.14.5