sjwsjw/github-issues huggingface dataset:
GitHub issue:   File "/home/zhizhou/anaconda3/envs/pytorch/lib/python3.10/site-packages/datasets/builder.py", line 992, in _download_and_prepare     if str(split_generator.split_info.name).lower() == "all": AttributeError: 'str' object has no attribute 'split_info'. Did you mean: 'splitlines'?
State: closed

Hi, I use the code below to load local file
```
    def _split_generators(self, dl_manager):
        # TODO: This method is tasked with downloading/extracting the data and defining the splits depending on the configuration
        # If several configurations are possible (listed in BUILDER_CONFIGS), the configuration selected by the user is in self.config.name

        # dl_manager is a datasets.download.DownloadManager that can be used to download and extract URLS
        # It can accept any type or nested list/dict and will give back the same structure with the url replaced with path to local files.
        # By default the archives will be extracted and a path to a cached folder where they are extracted is returned instead of the archive
        # urls = _URLS[self.config.name]
        data_dir = dl_manager.download_and_extract(_URLs)
        print(data_dir)
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN,
                # These kwargs will be passed to _generate_examples
                gen_kwargs={
                    "filepath": os.path.join(data_dir["train"]),
                    "split": "train",
                },
            ),
            datasets.SplitGenerator(
                name=datasets.Split.VALIDATION,
                # These kwargs will be passed to _generate_examples
                gen_kwargs={
                    "filepath": os.path.join(data_dir["dev"]),
                    "split": "dev",
                },
            ),
        ]
```
and error occured
```

Traceback (most recent call last):
  File "/home/zhizhou/data1/zhanghao/huggingface/FineTuning_Transformer/load_local_dataset.py", line 2, in <module>
    dataset = load_dataset("./QA_script.py",data_files='/home/zhizhou/.cache/huggingface/datasets/conversatiom_corps/part_file.json')
  File "/home/zhizhou/anaconda3/envs/pytorch/lib/python3.10/site-packages/datasets/load.py", line 1809, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/zhizhou/anaconda3/envs/pytorch/lib/python3.10/site-packages/datasets/builder.py", line 909, in download_and_prepare
    self._download_and_prepare(
  File "/home/zhizhou/anaconda3/envs/pytorch/lib/python3.10/site-packages/datasets/builder.py", line 1670, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/zhizhou/anaconda3/envs/pytorch/lib/python3.10/site-packages/datasets/builder.py", line 992, in _download_and_prepare
    if str(split_generator.split_info.name).lower() == "all":
AttributeError: 'str' object has no attribute 'split_info'. Did you mean: 'splitlines'?
```
Could you help me?
GitHub issue: Documentation links to examples are broken
State: closed

### Describe the bug

The links at the bottom of [add_dataset](https://huggingface.co/docs/datasets/v1.2.1/add_dataset.html) to examples of specific datasets are all broken, for example

- text classification: [ag_news](https://github.com/huggingface/datasets/blob/master/datasets/ag_news/ag_news.py) (original data are in csv files)

### Steps to reproduce the bug

Click on links to examples from latest documentation

### Expected behavior

Links should be up to date - it might be more stable to link to https://huggingface.co/datasets/ag_news/blob/main/ag_news.py

### Environment info

dataset v1.2.1
GitHub issue: Deprecate search API
State: open

The Search API only supports Faiss and ElasticSearch as vector stores, is somewhat difficult to maintain (e.g., it still doesn't support ElasticSeach 8.0, difficult testing, ...), does not have the best design (adds a bunch of methods to the `Dataset` class that are only useful after creating an index), the usage doesn't seem to be significant and is not integrated with the Hub. Since we have no plans/bandwidth to improve it and better alternatives such as `langchain` and `docarray` exist, I think it should be deprecated (and eventually removed).

If we decide to deprecate/remove it, the following usage instances need to be addressed:
* [Course](https://github.com/huggingface/course/blob/0018bb434204d9750a03592cb0d4e846093218d8/chapters/en/chapter5/6.mdx#L342 ) and [Blog](https://github.com/huggingface/blog/blob/4897c6f73d4492a0955ade503281711d01840e09/image-search-datasets.md?plain=1#L252) - calling the FAISS API directly should be OK in these instances as it's pretty simple to use for basic scenarios. Alternatively, we can use `langchain`, but this adds an extra dependency
*  [Transformers](https://github.com/huggingface/transformers/blob/50726f9ea7afc6113da617f8f4ca1ab264a5e28a/src/transformers/models/rag/retrieval_rag.py#L183) - we can use the FAISS API directly and store the index as a separate attribute (and instead of building the `wiki_dpr` index each time the dataset is generated, we can generate it once and push it to the Hub repo, and then read it from there

cc @huggingface/datasets @LysandreJik for the opinion
GitHub issue: load_dataset hangs on WSL
State: closed

### Describe the bug

load_dataset simply hangs. It happens once every ~5 times, and interestingly hangs for a multiple of 5 minutes (hangs for 5/10/15 minutes). Using the profiler in PyCharm shows that it spends the time at <method 'connect' of '_socket.socket' objects>. However, a local cache is available so I am not sure why socket is needed. ([profiler result](https://ibb.co/0Btbbp8))

It only happens on WSL for me. It works for native Windows and my MacBook. (cache quickly recognized and loaded within a second).



### Steps to reproduce the bug

I am using Ubuntu 22.04.2 LTS (GNU/Linux 5.15.90.1-microsoft-standard-WSL2 x86_64)

Python 3.10.10 (main, Mar 21 2023, 18:45:11) [GCC 11.2.0] on linux
>>> import datasets
>>> datasets.load_dataset('ai2_arc', 'ARC-Challenge')  # hangs for 5/10/15 minutes

### Expected behavior

cache quickly recognized and loaded within a second

### Environment info

Please let me know if I should provide more environment information.
GitHub issue: `map` function doesn't fully utilize `input_columns`.
State: closed

### Describe the bug

I wanted to select only some columns of data.
And I thought that's why the argument `input_columns` exists.
What I expected is like this:
If there are ["a", "b", "c", "d"] columns, and if I set `input_columns=["a", "d"]`, the data will have only ["a", "d"] columns.

But it doesn't select columns.
It preserves existing columns.
The main cause is `update` function of `dictionary` type `transformed_batch`.

https://github.com/huggingface/datasets/blob/682d21e94ab1e64c11b583de39dc4c93f0101c5a/src/datasets/iterable_dataset.py#L687-L691

`transformed_batch` gets all the columns by `transformed_batch = dict(batch)`.
Even `function_args` selects `input_columns`, `update` preserves columns other than `input_columns`.
I think it should take a new dictionary with columns in `input_columns` like this:
```
# transformed_batch = dict(batch)
# transformed_batch.update(self.function(*function_args, **self.fn_kwargs)

# This is what I think correct.
transformed_batch = self.function(*function_args, **self.fn_kwargs)
```

Let me know how to use `input_columns`.

### Steps to reproduce the bug

Described all above.

### Expected behavior

Described all above.

### Environment info

datasets: 2.12
python: 3.8
GitHub issue: DownloadConfig.proxies not work when load_dataset_builder calling HfApi.dataset_info  
State: open

### Describe the bug

```python
download_config = DownloadConfig(proxies={'https': '<my proxy>'})
builder = load_dataset_builder(..., download_config=download_config)
```
But, when getting the dataset_info from HfApi, the http requests not using the proxies.

### Steps to reproduce the bug

1. Setup proxies in DownloadConfig.
2. Call `load_dataset_build` with download_config.
3. Inspect the call stack in HfApi.dataset_info.

![image](https://github.com/huggingface/datasets/assets/138426806/33e538a8-2e22-4e63-b634-343febe5324b)

### Expected behavior

DownloadConfig.proxies works for getting dataset_info.

### Environment info

https://github.com/huggingface/datasets/commit/406b2212263c0d33f267e35b917f410ff6b3bc00
Python 3.11.4
GitHub issue: Argument type for map function changes when using `input_columns` for `IterableDataset`
State: closed

### Describe the bug

I wrote `tokenize(examples)` function as an argument for `map` function for `IterableDataset`.
It process dictionary type `examples` as a parameter.
It is used in `train_dataset = train_dataset.map(tokenize, batched=True)`
No error is raised.

And then, I found some unnecessary keys and values in `examples` so I added `input_columns` argument to `map` function to select keys and values.
It gives me an error saying
```
TypeError: tokenize() takes 1 positional argument but 3 were given.
```

The code below matters.
https://github.com/huggingface/datasets/blob/406b2212263c0d33f267e35b917f410ff6b3bc00/src/datasets/iterable_dataset.py#L687

For example, `inputs = {"a":1, "b":2, "c":3}`.
If `self.input_coluns` is `None`,
`inputs` is a dictionary type variable and `function_args` becomes a `list` of a single `dict` variable.
`function_args` becomes `[{"a":1, "b":2, "c":3}]`

Otherwise, lets say `self.input_columns = ["a", "c"]`
`[inputs[col] for col in self.input_columns]` results in `[1, 3]`.
I think it should be `[{"a":1, "c":3}]`.

I want to ask if the resulting format is intended.
Maybe I can modify `tokenize()` to have 2 parameters in this case instead of having 1 dictionary.
But this is confusing to me.
Or it should be fixed as `[{col:inputs[col] for col in self.input_columns}]`

### Steps to reproduce the bug

Run `map` function of `IterableDataset` with `input_columns` argument.

### Expected behavior

`function_args` looks better to have same format.
I think it should be `[{"a":1, "c":3}]`.

### Environment info

dataset version: 2.12
python: 3.8
GitHub issue: Use new hffs
State: closed

Thanks to @janineguo 's work in https://github.com/huggingface/datasets/pull/5919 which was needed to support HfFileSystem.

Switching to `HfFileSystem` will help implementing optimization in data files resolution

## Implementation details

I replaced all the from_hf_repo and from_local_or_remote in data_files.py to only use a new `from_patterns` which works for any fsspec path, including hf:// paths, https:// URLs and local paths. This simplifies the codebase since there is no logic duplication anymore when it comes to data files resolution.

I added `_prepare_path_and_storage_options` which returns the right storage_options to use given a path and a `DownloadConfig`. This is the only place where the logic depends on the filesystem type that must be used.

I also removed the `get_metadata_data_files_list ` and `get_patterns_and_data_files` functions added recently, since data files resolution is now handled using a common interface.

## New features

hf:// paths are now supported in data_files

## Breaking changes

DataFilesList and DataFilesDict:
- use `str` paths instead of `Union[Path, Url]`
- require posix paths for windows paths

close https://github.com/huggingface/datasets/issues/6017
GitHub issue: [docs] Update return statement of index search
State: closed

Clarifies in the return statement of the docstring that the retrieval score is `IndexFlatL2` by default (see [PR](https://github.com/huggingface/transformers/issues/24739) and internal Slack [convo](https://huggingface.slack.com/archives/C01229B19EX/p1689105179711689)), and fixes the formatting because multiple return values are not supported.
GitHub issue: Inconsistent "The features can't be aligned" error when combining map, multiprocessing, and variable length outputs
State: open

### Describe the bug

I'm using a dataset with map and multiprocessing to run a function that returned a variable length list of outputs. This output list may be empty. Normally this is handled fine, but there is an edge case that crops up when using multiprocessing. In some cases, an empty list result ends up in a dataset shard consisting of a single item. This results in a `The features can't be aligned` error that is difficult to debug because it depends on the number of processes/shards used.

I've reproduced a minimal example below. My current workaround is to fill empty results with a dummy value that I filter after, but this was a weird error that took a while to track down.

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.Dataset.from_list([{'idx':i} for i in range(60)])

def test_func(row, idx):
    if idx==58:
        return {'output': []}
    else:
        return {'output' : [{'test':1}, {'test':2}]}

# this works fine
test1 = dataset.map(lambda row, idx: test_func(row, idx), with_indices=True, num_proc=4)

# this fails
test2 = dataset.map(lambda row, idx: test_func(row, idx), with_indices=True, num_proc=32)
>ValueError: The features can't be aligned because the key output of features {'idx': Value(dtype='int64', id=None), 'output': Sequence(feature=Value(dtype='null', id=None), length=-1, id=None)} has unexpected type - Sequence(feature=Value(dtype='null', id=None), length=-1, id=None) (expected either [{'test': Value(dtype='int64', id=None)}] or Value("null").
```

The error occurs during the check

```python
_check_if_features_can_be_aligned([dset.features for dset in dsets])
```

When the multiprocessing splitting lines up just right with the empty return value, one of the `dset` in `dsets` will have a single item with an empty list value, causing the error.

### Expected behavior

Expected behavior is the result would be the same regardless of the `num_proc` value used.

### Environment info

Datasets version 2.11.0
Python 3.9.16
GitHub issue: Improve logging
State: closed

Adds the StreamHandler (as `hfh` and `transformers` do) to the library's logger to log INFO messages and logs the messages about "loading a cached result" (and some other warnings) as INFO

(Also removes the `leave=False` arg in the progress bars to be consistent with `hfh` and `transformers` - progress bars serve as an indicator that a result is not cached, so it makes more sense not to delete them)

Fix #2832, fix https://github.com/huggingface/datasets/issues/1948, fix https://github.com/huggingface/datasets/issues/5444
GitHub issue: Request to Share/Update Dataset Viewer Code
State: closed


Overview:
The repository (huggingface/datasets-viewer) was recently archived and when I tried to run the code, there was the error message "AttributeError: module 'datasets.load' has no attribute 'prepare_module'". I could not resolve the issue myself due to lack of documentation of that attribute. 

Request:
I kindly request the sharing of the code responsible for the dataset preview functionality or help with resolving the error. The dataset viewer on the Hugging Face website is incredibly useful since it is compatible with different types of inputs. It allows users to find datasets that meet their needs more efficiently. If needed, I am willing to contribute to the project by testing, documenting, and providing feedback on the dataset viewer code. 

Thank you for considering this request, and I look forward to your response.
GitHub issue: [FR] `map` should reuse unchanged columns from the previous dataset to avoid disk usage
State: open

### Feature request

Currently adding a new column with `map` will cause all the data in the dataset to be duplicated and stored/cached on the disk again. It should reuse unchanged columns. 

### Motivation

This allows having datasets with different columns but sharing some basic columns. Currently, these datasets would become too expensive to store and one would need some kind of on-the-fly join; which also doesn't seem implemented.

### Your contribution

_
GitHub issue: [FR] Transform Chaining, Lazy Mapping
State: open

### Feature request

Currently using a `map` call processes and duplicates the whole dataset, which takes both time and disk space.

The solution is to allow lazy mapping, which is essentially a saved chain of transforms that are applied on the fly whenever a slice of the dataset is requested.

The API should look like `map`, as `set_transform` changes the current dataset while `map` returns another dataset.

### Motivation

Lazy processing allows lower disk usage and faster experimentation.

### Your contribution

_
GitHub issue: Dataset.from_generator consistently freezes at ~1000 rows
State: closed

### Describe the bug

Whenever I try to create a dataset which contains images using `Dataset.from_generator`, it freezes around 996 rows. I suppose it has something to do with memory consumption, but there's more memory available. I

Somehow it worked a few times but mostly this makes the datasets library much more cumbersome to work with because generators are the easiest way to turn an existing dataset into a Hugging Face dataset.

I've let it run in the frozen state for way longer than it can possibly take to load the actual dataset.

Let me know if you have ideas how to resolve it!

### Steps to reproduce the bug

```python
from datasets import Dataset
import numpy as np

def gen():
    for row in range(10000):
        yield {"i": np.random.rand(512, 512, 3)}
        
Dataset.from_generator(gen)
# -> 90% of the time gets stuck around 1000 rows
```

### Expected behavior

Should continue and go through all the examples yielded by the generator, or at least throw an error or somehow communicate what's going on.

### Environment info

- `datasets` version: 2.8.0
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 12.0.1
- Pandas version: 1.5.1
GitHub issue: Misc improvements
State: closed

Contains the following improvements:

* fixes a "share dataset" link in README and modifies the "hosting" part in the disclaimer section
* updates `Makefile` to also run the style checks on `utils` and `setup.py`
* deletes a test for GH-hosted datasets (no longer supported)
* deletes `convert_dataset.sh` (outdated)
* aligns `utils/release.py` with `transformers` (the current version is outdated)
GitHub issue: Add KLUE-MRC metrics
State: closed

## Metrics for KLUE-MRC (Korean Language Understanding Evaluation — Machine Reading Comprehension)

Adding metrics for [KLUE-MRC](https://huggingface.co/datasets/klue).
KLUE-MRC is very similar to SQuAD 2.0 but has a slightly different format which is why I added metrics for KLUE-MRC.

Specifically, in the case of [LM Eval Harness](https://github.com/EleutherAI/lm-evaluation-harness), it leverages the scoring script of SQuAD to evaluate SQuAD 2.0 and KorQuAD. But the script isn't suitable for KLUE-MRC because KLUE-MRC is a bit different from SQuAD 2.0. And this is why I added the scoring script for KLUE-MRC.

- [x] All tests passed
- [x] Added a metric card (referred the metric card of SQuAD 2.0)
- [x] Compatibility test with [LM Eval Harness](https://github.com/EleutherAI/lm-evaluation-harness) passed

### References
- [KLUE: Korean Language Understanding Evaluation](https://datasets-benchmarks-proceedings.neurips.cc/paper_files/paper/2021/file/98dce83da57b0395e163467c9dae521b-Paper-round2.pdf)
- [KLUE on Hugging Face Datasets](https://huggingface.co/datasets/klue)
- #2416
GitHub issue: Getting a 409 error while loading xglue dataset
State: closed

### Describe the bug

Unable to load xglue dataset

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.load_dataset("xglue", "ntg")
```

> ConnectionError: Couldn't reach https://xglue.blob.core.windows.net/xglue/xglue_full_dataset.tar.gz (error 409)

### Expected behavior

Expected the dataset to load

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
GitHub issue: The current implementation has a potential bug in the sort method
State: closed

### Describe the bug

In the sort method，here's a piece of code

```python
# column_names: Union[str, Sequence_[str]]

# Check proper format of and for duplicates in column_names
if not isinstance(column_names, list):
    column_names = [column_names]
```

I get an error when I pass in a tuple based on the column_names type annotation, it will raise an errror.As in the example below, while the type annotation implies that a tuple can be passed.

```python
from datasets import load_dataset

dataset = load_dataset('glue', 'ax')['test']
dataset.sort(column_names=('premise', 'hypothesis'))
# Raise ValueError: Column '('premise', 'hypothesis')' not found in the dataset.
```

Of course, after I modified the tuple into a list, everything worked fine

Change the code to the following so there will be no problem

```python
# Check proper format of and for duplicates in column_names
if not isinstance(column_names, list):
    if isinstance(column_names, str):
        column_names = [column_names]
    else:
        column_names = list(column_names)
```

### Steps to reproduce the bug

```python
from datasets import load_dataset

dataset = load_dataset('glue', 'ax')['test']
dataset.sort(column_names=('premise', 'hypothesis'))
# Raise ValueError: Column '('premise', 'hypothesis')' not found in the dataset.
```

### Expected behavior

Passing tuple into column_names should be equivalent to passing list

### Environment info

- `datasets` version: 2.13.0
- Platform: macOS-13.1-arm64-arm-64bit
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.2
GitHub issue: extend the map function so it can wrap around long text that does not fit in the context window
State: open

### Feature request

I understand `dataset` provides a [`map`](https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L2849) function. This function in turn takes in a callable that is used to tokenize the text on which a model is trained. Frequently this text will not fit within a models's context window. In this case it would be useful to wrap around the text into multiple rows with each row fitting the model's context window. I tried to do it using this code as example which in turn I have borrowed from [here](https://stackoverflow.com/a/76343993/147530):

```
data = data.map(lambda samples: tokenizer(samples["text"], max_length=tokenizer.model_max_length, truncation=True, stride=4, return_overflowing_tokens=True), batched=True)
```

but running the code gives me this error:

```
File "/llm/fine-tune.py", line 117, in <module>
    data = data.map(lambda samples: tokenizer(samples["text"], max_length=tokenizer.model_max_length, truncation=True, stride=4, return_overflowing_tokens=True), batched=True)
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 580, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 545, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3087, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_dataset.py", line 3480, in _map_single
    writer.write_batch(batch)
  File "/llm/.env/lib/python3.9/site-packages/datasets/arrow_writer.py", line 556, in write_batch
    pa_table = pa.Table.from_arrays(arrays, schema=schema)
  File "pyarrow/table.pxi", line 3798, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 2962, in pyarrow.lib.Table.validate
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1 named input_ids expected length 394 but got length 447
```

The lambda function I have provided is correctly chopping up long text so it wraps around (and because of this 394 samples become 447 after wrap around) but the dataset `map` function does not like it.

### Motivation

please see above

### Your contribution

I'm afraid I don't have much knowledge to help
GitHub issue: ValueError: Table schema does not match schema used to create file
State: closed

### Describe the bug

Saving a dataset as parquet fails with a `ValueError: Table schema does not match schema used to create file`  if the dataset was obtained out of a `.select_columns()` call with columns selected out of order.

### Steps to reproduce the bug

```python
import datasets

dataset = datasets.Dataset.from_dict(
    {
        "x1": [1, 2, 3],
        "x2": [10, 11, 12],
    }
)

ds = dataset.select_columns(["x2", "x1"])

ds.to_parquet("demo.parquet")
```

```shell
>>>
ValueError: Table schema does not match schema used to create file: 
table:
x2: int64
x1: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x2": {"dtype": "int64", "_type": "V' + 53 vs. 
file:
x1: int64
x2: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x1": {"dtype": "int64", "_type": "V' + 53
```

--- 

I think this is because after the `.select_columns()` call with out of order columns, the output dataset features' schema ends up being out of sync with the schema of the arrow table backing it. 

```python
ds.features.arrow_schema
>>>
x1: int64
x2: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x1": {"dtype": "int64", "_type": "V' + 53

ds.data.schema
>>>
x2: int64
x1: int64
-- schema metadata --
huggingface: '{"info": {"features": {"x2": {"dtype": "int64", "_type": "V' + 53
```


So when we call `.to_parquet()`, the call behind the scenes to `datasets.io.parquet.ParquetDatasetWriter(...).write()` which initialises the backend `pyarrow.parquet.ParquetWriter` with `schema = self.dataset.features.arrow_schema` triggers `pyarrow` on write when [it checks](https://github.com/apache/arrow/blob/11b140a734a516e436adaddaeb35d23f30dcce44/python/pyarrow/parquet/core.py#L1086-L1090) that the `ParquetWriter` schema matches the schema of the table being written 🙌 

https://github.com/huggingface/datasets/blob/6ed837325cb539a5deb99129e5ad181d0269e050/src/datasets/io/parquet.py#L139-L141


### Expected behavior

The dataset gets successfully saved as parquet. 

*In the same way as it does if saving it as csv:

```python
import datasets

dataset = datasets.Dataset.from_dict(
    {
        "x1": [1, 2, 3],
        "x2": [10, 11, 12],
    }
)

ds = dataset.select_columns(["x2", "x1"])

ds.to_csv("demo.csv")
```

### Environment info

`python==3.11`
`datasets==2.13.1`
GitHub issue: `map` with any joblib backend
State: open

We recently enabled the (experimental) parallel backend switch for data download and extraction but not for `map` yet.

Right now we're using our `iflatmap_unordered` implementation for multiprocessing that uses a shared Queue to gather progress updates from the subprocesses and show a progress bar in the main process.

If a Queue implementation that would work on any joblib backend by leveraging the filesystem that is shared among workers, we can have `iflatmap_unordered` for joblib and therefore a `map` with any joblib backend with a progress bar !

Note that the Queue doesn't need to be that optimized though since we can choose a small frequency for progress updates (like 1 update per second).
GitHub issue: ConnectionError: Couldn't reach dataset_infos.json 
State: closed

### Describe the bug

I'm trying to load codeparrot/codeparrot-clean-train, but get the following error:

ConnectionError: Couldn't reach https://huggingface.co/datasets/codeparrot/codeparrot-clean-train/resolve/main/dataset_infos.json (ConnectionError(ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))))



### Steps to reproduce the bug

train_data = load_dataset('codeparrot/codeparrot-clean-train', split='train')


### Expected behavior

download the dataset

### Environment info

centos7
GitHub issue: Why max_shard_size is not supported in load_dataset and passed to download_and_prepare
State: closed

### Describe the bug

https://github.com/huggingface/datasets/blob/a8a797cc92e860c8d0df71e0aa826f4d2690713e/src/datasets/load.py#L1809

What I can to is break the `load_dataset` and use `load_datset_builder` + `download_and_prepare` instead.

### Steps to reproduce the bug

https://github.com/huggingface/datasets/blob/a8a797cc92e860c8d0df71e0aa826f4d2690713e/src/datasets/load.py#L1809

### Expected behavior

Users can define the max shard size.

### Environment info

datasets==2.13.1
GitHub issue: AutoSharding IterableDataset's when num_workers > 1
State: open

### Feature request


Minimal Example

```
import torch
from datasets import IterableDataset

d = IterableDataset.from_file(<file_name>)
dl = torch.utils.data.dataloader.DataLoader(d,num_workers=3)

for sample in dl:
    print(sample)

```

Warning:
Too many dataloader workers: 2 (max is dataset.n_shards=1). Stopping 1 dataloader workers.
To parallelize data loading, we give each process some shards (or data sources) to process. Therefore it's unnecessary to have a number of workers greater than dataset.n_shards=1. To enable more parallelism, please split the dataset in more files than 1.

Expected Behavior:
Dataset is sharded each cpu uses subset (contiguously - so you can do checkpoint loading/saving) 

### Motivation

I have a lot of unused cpu's and would like to be able to shard iterable datasets with pytorch's dataloader when num_workers > 1. This is for a very large single file. I am aware that we can use the `split_dataset_by_node` to ensure that each node (for distributed) gets different shards, but we should extend it so that this also continues for multiple workers. 

### Your contribution

If someone points me to what needs to change, I can create a PR.
GitHub issue: 404 on Datasets Documentation Page
State: closed

### Describe the bug

Getting a 404 from the Hugging Face Datasets docs page:
https://huggingface.co/docs/datasets/index


### Steps to reproduce the bug

1. Go to URL https://huggingface.co/docs/datasets/index
2. Notice 404 not found

### Expected behavior

URL should either show docs or redirect to new location

### Environment info

hugginface.co
GitHub issue: Only two cores are getting used in sagemaker with pytorch 3.10 kernel
State: closed

### Describe the bug

When using the newer pytorch 3.10 kernel, only 2 cores are being used by huggingface filter and map functions.  The Pytorch 3.9 kernel would use as many cores as specified in the num_proc field.

We have solved this in our own code by placing the following snippet in the code that is called inside subprocesses:

```os.sched_setaffinity(0, {i for i in range(1000)})```

The problem, as near as we can tell, us that once upon a time, cpu affinity was set using a bitmask ("0xfffff" and the like), and affinity recently changed to a list of processors rather than to using the mask.  As such, only processors 1 and 17 are shown to be working in htop.
![Selection_072](https://github.com/huggingface/datasets/assets/107141022/04c5a824-5321-4531-afca-7bc84dff36b4)


When running functions via `map`, the above resetting of affinity works to spread across the cores.  When using `filter`, however, only two cores are active.

### Steps to reproduce the bug

Repro steps:

1.  Create an aws sagemaker instance
2. use the pytorch 3_10 kernel
3. Load a dataset
4. run a filter operation
5. watch as only 2 cores are used when num_proc > 2
6. run a map operation
7. watch as only 2 cores are used when num_proc > 2
8. run a map operation with processor affinity reset inside the function called via map
9. Watch as all cores run



### Expected behavior

All specified cores are used via the num_proc argument.

### Environment info

AWS sagemaker with the following init script run in the terminal after instance creation:

conda init bash
bash
conda activate pytorch_p310
pip install Wand PyPDF pytesseract datasets seqeval pdfplumber transformers pymupdf sentencepiece timm donut-python accelerate optimum xgboost
python -m pip install 'git+https://github.com/facebookresearch/detectron2.git'
sudo yum -y install htop
sudo yum -y update
sudo yum -y install wget libstdc++ autoconf automake libtool autoconf-archive pkg-config gcc gcc-c++ make libjpeg-devel libpng-devel libtiff-devel zlib-devel
GitHub issue: Viewing dataset card returns “502 Bad Gateway”
State: closed

The url is: https://huggingface.co/datasets/Confirm-Labs/pile_ngrams_trigrams

I am able to successfully view the “Files and versions” tab: [Confirm-Labs/pile_ngrams_trigrams at main](https://huggingface.co/datasets/Confirm-Labs/pile_ngrams_trigrams/tree/main)

Any help would be appreciated! Thanks! I hope this is the right place to report an issue like this.
GitHub issue: Avoid stuck map operation when subprocesses crashes
State: closed

I've been using Dataset.map() with `num_proc=os.cpu_count()` to leverage multicore processing for my datasets, but from time to time I get stuck processes waiting forever. Apparently, when one of the subprocesses is abruptly killed (OOM killer, segfault, SIGKILL, etc), the main process keeps waiting for the async task sent to that child process to finish.

It seems to be easy to reproduce the issue with the following script:

```
import os
from datasets import Dataset, Features, Value


def do_stuck(item):
    os.kill(os.getpid(), 9)

data = {
    "col1": list(range(5)),
    "col2": list(range(5)),
}

ds = Dataset.from_dict(
    data,
    features=Features({
        "col1": Value("int64"),
        "col2": Value("int64"),
    }),
)

print(ds.map(do_stuck, num_proc=4))
```

This is an old behavior in Python, which apparently was fixed a few years ago in `concurrent.futures.ProcessPoolExecutor` ([ref](https://bugs.python.org/issue9205)), but not in `multiprocessing.pool.Pool` / `multiprocess.pool.Pool`, which is used by `Dataset.map` ([ref](https://bugs.python.org/issue22393)).

This PR is an idea to try to detect when a child process gets killed, and raises a `RuntimeError` warning the dataset.map() caller.

EDIT: Related proposal for future improvement: https://github.com/huggingface/datasets/discussions/5977
GitHub issue: Streaming Dataset behind Proxy - FileNotFoundError
State: closed

### Describe the bug

When trying to stream a dataset i get the following error after a few minutes of waiting.

```
FileNotFoundError: https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/n_files.json
If the repo is private or gated, make sure to log in with `huggingface-cli login`.
```

I have already set the proxy environment variables. Downloading a Dataset without streaming works as expected.
Still i suspect that this is connected to being behind a proxy.

Is there a way to set the proxy for streaming datasets? Possibly a keyword argument that gets passed to ffspec?

### Steps to reproduce the bug

This is the code i use.

```
import os
os.environ['http_proxy'] = "http://example.com:xxxx" 
os.environ['https_proxy'] = "http://example.com:xxxx" 


from datasets import load_dataset

ds = load_dataset("facebook/voxpopuli", name="de", streaming=True)
```

### Expected behavior

I would expect the streaming functionality to use the set proxy settings.

### Environment info


- `datasets` version: 2.13.0
- Platform: Linux-5.15.0-73-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- PyArrow version: 11.0.0
- Pandas version: 2.0.2
GitHub issue: Filter unsupported extensions
State: closed

I used a regex to filter the data files based on their extension for packaged builders.

I tried and a regex is 10x faster that using `in` to check if the extension is in the list of supported extensions.

Supersedes https://github.com/huggingface/datasets/pull/5850

Close https://github.com/huggingface/datasets/issues/5849

I also did a small change to favor the parquet module in case of a draw in the extension counter.
GitHub issue: Common Voice datasets still need `use_auth_token=True`
State: closed

### Describe the bug

We don't need to pass `use_auth_token=True` anymore to download gated datasets or models, so the following should work if correctly logged in.

```py
from datasets import load_dataset

load_dataset("mozilla-foundation/common_voice_6_1", "tr", split="train+validation")
```

However it throws an error - probably because something weird is hardcoded into the dataset loading script.

### Steps to reproduce the bug

1.) 
```
huggingface-cli login
```

2.) Make sure that you have accepted the license here:
https://huggingface.co/datasets/mozilla-foundation/common_voice_6_1

3.) Run:
```py
from datasets import load_dataset

load_dataset("mozilla-foundation/common_voice_6_1", "tr", split="train+validation")
```

4.)  You'll get:

```
File ~/hf/lib/python3.10/site-packages/datasets/builder.py:963, in DatasetBuilder._download_and_prepare(self, dl_manager, verification_mode, **prepare_split_kwargs)
    961 split_dict = SplitDict(dataset_name=self.name)
    962 split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 963 split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    965 # Checksums verification
    966 if verification_mode == VerificationMode.ALL_CHECKS and dl_manager.record_checksums:

File ~/.cache/huggingface/modules/datasets_modules/datasets/mozilla-foundation--common_voice_6_1/f4d7854c466f5bd4908988dbd39044ec4fc634d89e0515ab0c51715c0127ffe3/common_voice_6_1.py:150, in CommonVoice._split_generators(self, dl_manager)
    148 hf_auth_token = dl_manager.download_config.use_auth_token
    149 if hf_auth_token is None:
--> 150     raise ConnectionError(
    151         "Please set use_auth_token=True or use_auth_token='<TOKEN>' to download this dataset"
    152     )
    154 bundle_url_template = STATS["bundleURLTemplate"]
    155 bundle_version = bundle_url_template.split("/")[0]

ConnectionError: Please set use_auth_token=True or use_auth_token='<TOKEN>' to download this dataset
```

### Expected behavior

One should not have to pass `use_auth_token=True`. Also see discussion here: https://github.com/huggingface/blog/pull/1243#discussion_r1235131150

### Environment info

```
- `datasets` version: 2.13.0
- Platform: Linux-6.2.0-76060200-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.0.dev0
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
```
GitHub issue: Config name / split name lost after map with multiproc
State: open

### Describe the bug

Performing a `.map` method on a dataset loses it's config name / split name only if run with multiproc

### Steps to reproduce the bug

```python
from datasets import Audio, load_dataset
from transformers import AutoFeatureExtractor
import numpy as np

# load dummy dataset
libri = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean")

# make train / test splits
libri = libri["validation"].train_test_split(seed=42, shuffle=True, test_size=0.1)

# example feature extractor
model_id = "ntu-spml/distilhubert"
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, do_normalize=True, return_attention_mask=True)

sampling_rate = feature_extractor.sampling_rate

libri = libri.cast_column("audio", Audio(sampling_rate=sampling_rate))

max_duration = 30.0

def preprocess_function(examples):
    audio_arrays = [x["array"] for x in examples["audio"]]
    inputs = feature_extractor(
        audio_arrays,
        sampling_rate=feature_extractor.sampling_rate,
        max_length=int(feature_extractor.sampling_rate * max_duration),
        truncation=True,
        return_attention_mask=True,
    )
    return inputs

# single proc map
libri_encoded = libri.map(
    preprocess_function, remove_columns=["audio", "file"], batched=True, num_proc=1
)

print(10 * "=" ,"Single processing", 10 * "=")
print("Config name before: ", libri["train"].config_name, " Split name before: ", libri["train"].split)
print("Config name after: ", libri_encoded["train"].config_name, " Split name after: ", libri_encoded["train"].split)

# multi proc map
libri_encoded = libri.map(
    preprocess_function, remove_columns=["audio", "file"], batched=True, num_proc=2
)

print(10 * "=" ,"Multi processing", 10 * "=")
print("Config name before: ", libri["train"].config_name, " Split name before: ", libri["train"].split)
print("Config name after: ", libri_encoded["train"].config_name, " Split name after: ", libri_encoded["train"].split)
```

**Print Output:**
```
========== Single processing ==========
Config name before:  clean  Split name before:  validation
Config name after:  clean  Split name after:  validation
========== Multi processing ==========
Config name before:  clean  Split name before:  validation
Config name after:  None  Split name after:  None
```

=> we can see that the config/split names are lost in the multiprocessing setting




### Expected behavior

Should retain both config / split names in the multiproc setting

### Environment info

- `datasets` version: 2.13.1.dev0
- Platform: Linux-5.15.0-67-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.2
GitHub issue: Issue with train_test_split maintaining the same underlying PyArrow Table
State: open

### Describe the bug

I've been using the train_test_split method in the datasets module to split my HuggingFace Dataset into separate training, validation, and testing subsets. However, I've noticed an issue where the split datasets appear to maintain the same underlying PyArrow Table.

### Steps to reproduce the bug

1. Load any dataset  ```dataset = load_dataset("lhoestq/demo1")``` 
2. Try the next code:
```python
from datasets import Dataset, DatasetDict

train_size = 0.6

split_train = dataset["train"].train_test_split(
    train_size=train_size,
)

separate_dataset_dict = DatasetDict({
    "train": split_train["train"],
    "test": split_train["test"],
})
```
3.  The next code ```print(separate_dataset_dict)``` when printing the dataset it gives the indication that they have 3 and 2 rows respectively.
4. But the next code: 
 ```python
print(len(separate_dataset_dict["train"].data['id']))
print(len(separate_dataset_dict["test"].data['id'])) 
```

 Indicates that both tables still have 5 rows.

### Expected behavior

However, I've noticed that train_test_split["train"].data, test_val_split["train"].data, and test_val_split["test"].data are identical, suggesting that they all point to the same underlying PyArrow Table. This means that the split datasets are not independent, as I expected.

I believe this is a bug in the train_test_split implementation, as I would expect this function to return datasets with separate underlying PyArrow Tables. Could you please help me understand if this is expected behavior, or if there's a workaround to create truly independent split datasets?

I would appreciate any assistance with this issue. Thank you.

### Environment info

I tried in Colab:

- `datasets` version: 2.13.0
- Platform: Windows-10-10.0.22621-SP0
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1

and my PC:

- `datasets` version: 2.13.0
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 9.0.0
- Pandas version: 1.5.3
GitHub issue: IterableDataset: split by node and map may preprocess samples that will be skipped anyway
State: open

              There are two ways an iterable dataset can be split by node:
1. if the number of shards is a factor of number of GPUs: in that case the shards are evenly distributed per GPU
2. otherwise, each GPU iterate on the data and at the end keeps 1 sample out of n(GPUs) - skipping the others.

In case 2. it's therefore possible to have the same examples passed to `prepare_dataset` for each GPU.

This doesn't sound optimized though, because it runs the preprocessing on samples that won't be used in the end.

Could you open a new issue so that we can discuss about this and find a solution ?

_Originally posted by @lhoestq in https://github.com/huggingface/datasets/issues/5360#issuecomment-1592729051_
GitHub issue: read metric glue.py from local file 
State: closed

### Describe the bug

Currently, The server is off-line.  I am using the glue metric from the local file downloaded from the hub. 
I download / cached datasets using `load_dataset('glue','sst2', cache_dir='/xxx')` to cache them and then in the off-line mode, I use `load_dataset('xxx/glue.py','sst2', cache_dir='/xxx')`. I can successfully reuse cached datasets.

My problem is about the load_metric. 
When I run `load_dataset('xxx/glue_metric.py','sst2',cache_dir='/xxx')` , it returns 

`  File "xx/lib64/python3.9/site-packages/datasets/utils/deprecation_utils.py", line 46, in wrapper
    return deprecated_function(*args, **kwargs)
  File "xx//lib64/python3.9/site-packages/datasets/load.py", line 1392, in load_metric
    metric = metric_cls(
TypeError: 'NoneType' object is not callable`

Thanks in advance for help! 
### Steps to reproduce the bug

N/A

### Expected behavior

N/A

### Environment info

`datasets == 2.12.0`
GitHub issue: Bad error message when trying to download gated dataset
State: closed

### Describe the bug

When I attempt to download a model from the Hub that is gated without being logged in, I get a nice error message. E.g.:

E.g.
```sh
Repository Not Found for url: https://huggingface.co/api/models/DeepFloyd/IF-I-XL-v1.0.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password..
Will try to load from local cache.
```

If I do the same for a gated dataset on the Hub, I'm not gated a nice error message IMO:

```sh
File ~/hf/lib/python3.10/site-packages/fsspec/implementations/http.py:430, in HTTPFileSystem._info(self, url, **kwargs)
    427     except Exception as exc:
    428         if policy == "get":
    429             # If get failed, then raise a FileNotFoundError
--> 430             raise FileNotFoundError(url) from exc
    431         logger.debug(str(exc))
    433 return {"name": url, "size": None, **info, "type": "file"}

FileNotFoundError: https://huggingface.co/datasets/mozilla-foundation/common_voice_13_0/resolve/main/n_shards.json
```

### Steps to reproduce the bug

```
huggingface-cli logout
```

and then:

```py
from datasets import load_dataset, Audio

# English
stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "en", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
en_sample = next(iter(stream_data))["audio"]["array"]

# Swahili
stream_data = load_dataset("mozilla-foundation/common_voice_13_0", "sw", split="test", streaming=True)
stream_data = stream_data.cast_column("audio", Audio(sampling_rate=16000))
sw_sample = next(iter(stream_data))["audio"]["array"]
```

### Expected behavior

Better error message

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.12.0
- Platform: Linux-6.2.0-76060200-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.16.0.dev0
- PyArrow version: 11.0.0
- Pandas version: 1.5.3