sjwsjw/github-issues huggingface dataset:
GitHub issue: Calling shuffle on a IterableDataset with streaming=True, gives "ValueError: cannot reshape array"
State: closed

### Describe the bug

When calling shuffle on a IterableDataset with streaming=True, I get the following error:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/administrator/Documents/Projects/huggingface/jax-diffusers-sprint-consistency-models/virtualenv/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 937, in __iter__
    for key, example in ex_iterable:
  File "/home/administrator/Documents/Projects/huggingface/jax-diffusers-sprint-consistency-models/virtualenv/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 627, in __iter__
    for x in self.ex_iterable:
  File "/home/administrator/Documents/Projects/huggingface/jax-diffusers-sprint-consistency-models/virtualenv/lib/python3.10/site-packages/datasets/iterable_dataset.py", line 138, in __iter__
    yield from self.generate_examples_fn(**kwargs_with_shuffled_shards)
  File "/home/administrator/.cache/huggingface/modules/datasets_modules/datasets/mnist/fda16c03c4ecfb13f165ba7e29cf38129ce035011519968cdaf74894ce91c9d4/mnist.py", line 111, in _generate_examples
    images = np.frombuffer(f.read(), dtype=np.uint8).reshape(size, 28, 28)
ValueError: cannot reshape array of size 59992 into shape (60000,28,28)

```
Tested with the fashion_mnist and mnist datasets

### Steps to reproduce the bug

Code to reproduce
```python
from datasets import load_dataset

SHUFFLE_SEED = 42
SHUFFLE_BUFFER_SIZE = 10_000

dataset = load_dataset('fashion_mnist', streaming=True).shuffle(seed=SHUFFLE_SEED, buffer_size=SHUFFLE_BUFFER_SIZE)
next(iter(dataset['train']))
```

### Expected behavior

A random item from the dataset and no error

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.0-69-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 2.0.0
GitHub issue: ImportError: cannot import name 'DeprecatedEnum' from 'datasets.utils.deprecation_utils' 
State: closed

### Describe the bug

The module moved to new place?

### Steps to reproduce the bug

in the import step,

```python
from datasets.utils.deprecation_utils import DeprecatedEnum
```

error:

```
ImportError: cannot import name 'DeprecatedEnum' from 'datasets.utils.deprecation_utils' 
```

### Expected behavior

import successfully

### Environment info

python==3.9.16
datasets==1.18.3
GitHub issue: Minor tqdm fixes
State: closed

`GeneratorBasedBuilder`'s TQDM bars were not used as context managers. This PR fixes that (missed these bars in https://github.com/huggingface/datasets/pull/5560). 

Also, this PR modifies the single-proc `save_to_disk` to fix the issue with the TQDM bar not accumulating the progress in the multi-shard setting (again, this bug was introduced by me in the linked PR 😎)
GitHub issue: [IterableDatasets] Add column followed by interleave datasets gives bogus outputs
State: closed

### Describe the bug

If we add a new column to our iterable dataset using the hack described in #5752, when we then interleave datasets the new column is pinned to one value.

### Steps to reproduce the bug

What we're going to do here is:
1. Load an iterable dataset in streaming mode (`original_dataset`)
2. Add a new column to this dataset using the hack in #5752 (`modified_dataset_1`)
3. Create another new dataset by adding a column with the same key but different values (`modified_dataset_2`)
4. Interleave our new datasets (`modified_dataset_1` + `modified_dataset_2`)
5. Check the value of our newly added column (`new_column`)

```python
from datasets import load_dataset

# load an iterable dataset
original_dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)

# now add a new column to our streaming dataset using our hack from 5752
name = "new_column"
column = [f"new dataset 1, row {i}" for i in range(50)]

new_features = original_dataset.features.copy()
new_features[name] = new_features["file"]  # I know that "file" has the right column type to match our new feature

def add_column_fn(example, idx):
    if name in example:
        raise ValueError(f"Error when adding {name}: column {name} is already in the dataset.")
    return {name: column[idx]}

modified_dataset_1 = original_dataset.map(add_column_fn, with_indices=True, features=new_features)

# now create a second modified dataset using the same trick
column = [f"new dataset 2, row {i}" for i in range(50)]

def add_column_fn(example, idx):
    if name in example:
        raise ValueError(f"Error when adding {name}: column {name} is already in the dataset.")
    return {name: column[idx]}

modified_dataset_2 = original_dataset.map(add_column_fn, with_indices=True, features=new_features)

# interleave these datasets
interleaved_dataset = interleave_datasets([modified_dataset_1, modified_dataset_2])

# now check what the value of the added column is
for i, sample in enumerate(interleaved_dataset):
    print(sample["new_column"])
    if i == 10:
        break
```
**Print Output:**
```
new dataset 2, row 0
new dataset 2, row 0
new dataset 2, row 1
new dataset 2, row 1
new dataset 2, row 2
new dataset 2, row 2
new dataset 2, row 3
new dataset 2, row 3
new dataset 2, row 4
new dataset 2, row 4
new dataset 2, row 5
```
We see that we only get outputs from our second dataset.

### Expected behavior

We should interleave between dataset 1 and 2 and increase in row value:
```
new dataset 1, row 0
new dataset 2, row 0
new dataset 1, row 1
new dataset 2, row 1
new dataset 1, row 2
new dataset 2, row 2
...
```

### Environment info

- datasets version: 2.10.2.dev0
- Platform: Linux-4.19.0-23-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.16
- Huggingface_hub version: 0.13.3
- PyArrow version: 10.0.1
- Pandas version: 1.5.2
GitHub issue: Streaming dataset looses `.feature` method after `.add_column`
State: open

### Describe the bug

After appending a new column to a streaming dataset using `.add_column`, we can no longer access the list of dataset features using the `.feature` method.

### Steps to reproduce the bug

```python
from datasets import load_dataset

original_dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)
print(original_dataset.features.keys())

# now add a new column to our streaming dataset
modified_dataset = original_dataset.add_column("new_column", ["some random text" for _ in range(50)])
print(modified_dataset.features.keys())
```
**Print Output:**
```
dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id'])
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[1], line 8
      6 # now add a new column to our streaming dataset
      7 modified_dataset = original_dataset.add_column("new_column", ["some random text" for _ in range(50)])
----> 8 print(modified_dataset.features.keys())

AttributeError: 'NoneType' object has no attribute 'keys'
```

We see that we get the features for the original dataset, but not the modified one with the added column.


### Expected behavior

Features should be persevered after adding a new column, i.e. calling:
```python
print(modified_dataset.features.keys())
```

Should return:
```
dict_keys(['file', 'audio', 'text', 'speaker_id', 'chapter_id', 'id', 'new_column'])
```

### Environment info

- `datasets` version: 2.10.2.dev0
- Platform: Linux-4.19.0-23-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.16
- Huggingface_hub version: 0.13.3
- PyArrow version: 10.0.1
- Pandas version: 1.5.2
GitHub issue: Consistent ArrayXD Python formatting + better NumPy/Pandas formatting
State: closed

Return a list of lists instead of a list of NumPy arrays when converting the variable-shaped `ArrayXD` to Python. Additionally, improve the NumPy conversion by returning a numeric NumPy array when the offsets are equal or a NumPy object array when they aren't, and allow converting the variable-shaped `ArrayXD` to Pandas.

(Reported in https://github.com/huggingface/datasets/issues/5719#issuecomment-1507579671)
GitHub issue: Fail to create datasets from a generator when using Google Big Query
State: closed

### Describe the bug

Creating a dataset from a generator using `Dataset.from_generator()` fails if the generator is the [Google Big Query Python client](https://cloud.google.com/python/docs/reference/bigquery/latest). The problem is that the Big Query client is not pickable. And the function `create_config_id` tries to get a hash of the generator by pickling it. So the following error is generated:

```
_pickle.PicklingError: Pickling client objects is explicitly not supported.
Clients have non-trivial state that is local and unpickleable.
```


### Steps to reproduce the bug

1. Install the big query client and datasets `pip install google-cloud-bigquery datasets`
2. Run the following code:

```py
from datasets import Dataset
from google.cloud import bigquery

client = bigquery.Client()

# Perform a query.
QUERY = (
    'SELECT name FROM `bigquery-public-data.usa_names.usa_1910_2013` '
    'WHERE state = "TX" '
    'LIMIT 100')
query_job = client.query(QUERY)  # API request
rows = query_job.result()  # Waits for query to finish

ds = Dataset.from_generator(rows)

for r in ds:
    print(r)
```

### Expected behavior

Two options:
1. Ignore the pickle errors when computing the hash
2. Provide a scape hutch so that we can avoid calculating the hash for the generator. For example, allowing to provide a hash from the user. 

### Environment info

python 3.9
google-cloud-bigquery         3.9.0
datasets                      2.11.0
GitHub issue: [BUG] With Pandas 2.0.0, `load_dataset` raises `TypeError: read_csv() got an unexpected keyword argument 'mangle_dupe_cols'`
State: closed

The `load_dataset` function with Pandas `1.5.3` has no issue (just a FutureWarning) but crashes with Pandas `2.0.0`.
For your convenience, I opened a draft Pull Request to fix it quickly: https://github.com/huggingface/datasets/pull/5745

---

* The FutureWarning mentioned above: 
```
FutureWarning: the 'mangle_dupe_cols' keyword is deprecated and will be removed in a future version. Please take steps to stop the use of 'mangle_dupe_cols'
```
GitHub issue: dataclass.py in virtual environment is   overriding the stdlib module "dataclasses"
State: closed

### Describe the bug

"e:\Krish_naik\FSDSRegression\venv\Lib\dataclasses.py" is overriding the stdlib module "dataclasses"

### Steps to reproduce the bug

module issue

### Expected behavior

 overriding the stdlib module "dataclasses"

### Environment info

VS code
GitHub issue: Fix CI mock filesystem fixtures
State: closed

This PR fixes the fixtures of our CI mock filesystems.

Before, we had to pass `clobber=True` to `fsspec.register_implementation` to overwrite the still present previously added "mock" filesystem. That meant that the mock filesystem fixture was not working properly, because the previously added "mock" filesystem, should have been deleted by the fixture.

This PR fixes the mock filesystem fixtures, so that the "mock" filesystem is properly deleted from the inner `fsspec` registry.

Tests were added to check the correct behavior of the mock filesystem fixtures.

Related to:
- #5733
GitHub issue: weird result during dataset split when data path starts with `/data`
State: open

### Describe the bug

The regex defined here https://github.com/huggingface/datasets/blob/f2607935c4e45c70c44fcb698db0363ca7ba83d4/src/datasets/utils/py_utils.py#L158 

will cause a weird result during dataset split when data path starts with `/data`

### Steps to reproduce the bug

1.  clone dataset into local path
```
cd /data/train/raw/
git lfs clone  https://huggingface.co/datasets/deepmind/code_contests.git

ls /data/train/raw/code_contests
# README.md  data  dataset_infos.json

ls /data/train/raw/code_contests/data
# test-00000-of-00001-9c49eeff30aacaa8.parquet
# train-[0-9]+-of-[0-9]+-xx.parquet
# valid-00000-of-00001-5e672c5751f060d3.parquet

```

2. loading data from local 
```
from datasets import load_dataset
dataset = load_dataset('/data/train/raw/code_contests')

FileNotFoundError: Unable to resolve any data file that matches '['data/train/raw/code_contests/data/train-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*']' at /data/train/raw/code_contests with any supported extension
```

weird path `data/train/raw/code_contests/data/train-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*`

While dive deep into `LocalDatasetModuleFactoryWithoutScript` defined in [load.py](https://github.com/huggingface/datasets/blob/f2607935c4e45c70c44fcb698db0363ca7ba83d4/src/datasets/load.py#L627) and _get_data_files_patterns https://github.com/huggingface/datasets/blob/f2607935c4e45c70c44fcb698db0363ca7ba83d4/src/datasets/data_files.py#L228. I found the weird behavior caused by `string_to_dict`

3. check `string_to_dict`
```
p = '/data/train/raw/code_contests/data/test-00000-of-00001-9c49eeff30aacaa8.parquet'
split_pattern = 'data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'

string_to_dict(p, split_pattern)
# {'split': 'train/raw/code_contests/data/test'}

p = '/data2/train/raw/code_contests/data/test-00000-of-00001-9c49eeff30aacaa8.parquet'
string_to_dict(p, split_pattern)
{'split': 'test'}
```

go deep into string_to_dict https://github.com/huggingface/datasets/blob/f2607935c4e45c70c44fcb698db0363ca7ba83d4/src/datasets/utils/py_utils.py#L158. 

4. test the regex:
<img width="680" alt="image" src="https://user-images.githubusercontent.com/1772912/231351129-75179f01-fb9f-4f12-8fa9-0dfcc3d5f3bd.png">

<img width="679" alt="image" src="https://user-images.githubusercontent.com/1772912/231351025-009f3d83-2cf3-4e15-9ed4-6b9663dcb2ee.png">




### Expected behavior

statement in `steps to reproduce the bug`

3. check `string_to_dict`
```
p = '/data/train/raw/code_contests/data/test-00000-of-00001-9c49eeff30aacaa8.parquet'
split_pattern = 'data/{split}-[0-9][0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*'

string_to_dict(p, split_pattern)
# {'split': 'train/raw/code_contests/data/test'}

p = '/data2/train/raw/code_contests/data/test-00000-of-00001-9c49eeff30aacaa8.parquet'
string_to_dict(p, split_pattern)
{'split': 'test'}
```


### Environment info

- linux(debian)
- python 3.7
- datasets 2.8.0
GitHub issue: load_dataset("text","dataset.txt") loads the wrong dataset!
State: closed

### Describe the bug

I am trying to load my own custom text dataset using the load_dataset function.  My dataset is a bunch of ordered text, think along the lines of shakespeare plays.  However, after I load the dataset and I inspect it, the dataset is a table with a bunch of latitude and longitude values!  What in the world??

### Steps to reproduce the bug

my_dataset = load_dataset("text","TextFile.txt")
my_dataset

### Expected behavior

I expected the dataset to contain the actual data from the text document that I used.

### Environment info

Google Colab
GitHub issue: ClassLabel Error 
State: closed

### Describe the bug

I still getting the error "call() takes 1 positional argument but 2 were given" even after ensuring that the value being passed to the label object is a single value and that the ClassLabel object has been created with the correct number of label classes

### Steps to reproduce the bug

from datasets import ClassLabel, Dataset

1. Create the ClassLabel object with 3 label values and their corresponding names
label_test = ClassLabel(num_classes=3, names=["label_1", "label_2", "label_3"])

2. Define a dictionary with text and label fields
data = {
    'text': ['text_1', 'text_2', 'text_3'],
    'label': [1, 2, 3],
}

3. Create a Hugging Face dataset from the dictionary
dataset = Dataset.from_dict(data)
print(dataset.features)
4. Map the label values to their corresponding label names using the label object
dataset = dataset.map(lambda example: {'text': example['text'], 'label': label_test(example['label'])})

5. Print the resulting dataset
print(dataset)

### Expected behavior

I hope my label type is class label instead int. 

### Environment info

python 3.9
google colab
GitHub issue: FORCE_REDOWNLOAD raises "Directory not empty" exception on second run
State: open

### Describe the bug

Running `load_dataset(..., download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)` twice raises a `Directory not empty` exception on the second run.

### Steps to reproduce the bug

I cannot test this on datasets v2.11.0 due to #5711, but this happens in v2.10.1.

1. Set up a script `my_dataset.py` to generate and load an offline dataset.
2. Load it with 
    ```python
            ds = datasets.load_dataset(path=/path/to/my_dataset.py,
                                       name='toy',
                                       data_dir=/path/to/my_dataset.py,
                                       cache_dir=cache_dir,
                                       download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD,
                                       )
    ```
    It loads fine
    ```
    Dataset my_dataset downloaded and prepared to /path/to/cache/toy-..e05e/1.0.0/...5b4c. Subsequent calls will reuse this data.
    ```
3. Try to load it again with the same snippet and the splits are generated, but at the end of the loading process it raises the error
    ```
    2023-04-11 12:10:19,965: DEBUG:	open file: /path/to/cache/toy-..e05e/1.0.0/...5b4c.incomplete/dataset_info.json
    Traceback (most recent call last):
      File "<string>", line 2, in <module>
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/load.py", line 1782, in load_dataset
        builder_instance.download_and_prepare(
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/builder.py", line 852, in download_and_prepare
        with incomplete_dir(self._output_dir) as tmp_output_dir:
      File "/path/to/conda/environment/lib/python3.10/contextlib.py", line 142, in __exit__
        next(self.gen)
      File "/path/to/conda/environment/lib/python3.10/site-packages/datasets/builder.py", line 826, in incomplete_dir
        shutil.rmtree(dirname)
      File "/path/to/conda/environment/lib/python3.10/shutil.py", line 730, in rmtree
        onerror(os.rmdir, path, sys.exc_info())
      File "/path/to/conda/environment/lib/python3.10/shutil.py", line 728, in rmtree
        os.rmdir(path)
    OSError: [Errno 39] Directory not empty: '/path/to/cache/toy-..e05e/1.0.0/...5b4c'
    ```


### Expected behavior

Regenerate the dataset from scratch and reload it.

### Environment info

- `datasets` version: 2.10.1
- Platform: Linux-4.18.0-483.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.8
- PyArrow version: 11.0.0
- Pandas version: 1.5.2
GitHub issue: Implement sharding on merged iterable datasets
State: closed

This PR allows sharding of merged iterable datasets.

Merged iterable datasets with for instance the `interleave_datasets` command are comprised of multiple sub-iterable, one for each dataset that has been merged.

With this PR, sharding a merged iterable will result in multiple merged datasets each comprised of sharded sub-iterable, ensuring that there is no duplication of data.

As a result it is now possible to set any amount of workers in the dataloader as long as it is lower or equal to the lowest amount of shards amongst the datasets. Before it had to be set to 0.

I previously talked about this issue on the forum [here](https://discuss.huggingface.co/t/interleaving-iterable-dataset-with-num-workers-0/35801)
GitHub issue: Unpin fsspec
State: closed

In `fsspec--2023.4.0` default value for clobber when registering an implementation was changed from True to False. See:
- https://github.com/fsspec/filesystem_spec/pull/1237

This PR recovers previous behavior by passing clobber True when registering mock implementations.

This PR also removes the temporary pin introduced by:
- #5731

Fix #5734.
GitHub issue: Enwik8 should support the standard split
State: closed

### Feature request

The HuggingFace Datasets library currently supports two BuilderConfigs for Enwik8. One config yields individual lines as examples, while the other config yields the entire dataset as a single example. Both support only a monolithic split: it is all grouped as "train". 

The HuggingFace Datasets library should include a BuilderConfig for Enwik8 with train, validation, and test sets derived from the first 90 million bytes, next 5 million bytes, and last 5 million bytes, respectively. This Enwik8 split is standard practice in LM papers, as elaborated and motivated below. 

### Motivation

Enwik8 is commonly split into 90M, 5M, 5M consecutive bytes. This is done in the Transformer-XL [codebase](https://github.com/kimiyoung/transformer-xl/blob/44781ed21dbaec88b280f74d9ae2877f52b492a5/getdata.sh#L34), and is additionally mentioned in the Sparse Transformers [paper](https://arxiv.org/abs/1904.10509) and the Compressive Transformers [paper](https://arxiv.org/abs/1911.05507). This split is pretty much universal among language modeling papers. 

One may obtain the splits by manual wrangling, using the data yielded by the ```enwik8-raw``` BuilderConfig. However, this undermines the seamless functionality of the library: one must slice the single raw example, extract it into three tensors, and wrap each in a separate dataset. 

This becomes even more of a nuisance if using the current Enwik8 HuggingFace dataset as a TfdsDataSource with [SeqIO](https://github.com/google/seqio), where a pipeline of preprocessors is typically included in a SeqIO Task definition, to be applied immediately after loading the data with TFDS. 

### Your contribution

Supporting this functionality in HuggingFace Datasets will only require an additional BuilderConfig for Enwik8 and a few additional lines of code. I will submit a PR.
GitHub issue: The order of data split names is nondeterministic
State: closed

After this CI error: https://github.com/huggingface/datasets/actions/runs/4639528358/jobs/8210492953?pr=5718
```
FAILED tests/test_data_files.py::test_get_data_files_patterns[data_file_per_split4] - AssertionError: assert ['random', 'train'] == ['train', 'random']
  At index 0 diff: 'random' != 'train'
  Full diff:
  - ['train', 'random']
  + ['random', 'train']
```
I have checked locally and found out that the data split order is nondeterministic.

This is caused by the use of `set` for sharded splits.
GitHub issue: Fallback JSON Dataset loading does not load all values when features specified manually
State: closed

### Describe the bug

The fallback JSON dataset loader located here:

https://github.com/huggingface/datasets/blob/1c4ec00511868bd881e84a6f7e0333648d833b8e/src/datasets/packaged_modules/json/json.py#L130-L153

does not load the values of features correctly when features are specified manually and not all features have a value in the first entry of the dataset. I'm pretty sure this is not supposed to be expected bahavior?

To fix this you'd have to change this line:

https://github.com/huggingface/datasets/blob/1c4ec00511868bd881e84a6f7e0333648d833b8e/src/datasets/packaged_modules/json/json.py#L140

To pass a schema to pyarrow which has the same structure as the features argument passed to the load_dataset() method.

### Steps to reproduce the bug

Consider a dataset JSON like this:

```
[
{
	"instruction": "Do stuff",
	"output": "Answer stuff"
},
{
	"instruction": "Do stuff2",
	"input": "Additional Input2",
	"output": "Answer stuff2"
}
]
```

Using this code to load the dataset:

```
from datasets import load_dataset, Features, Value

features = {
    "instruction": Value("string"),
    "input": Value("string"),
    "output": Value("string")
}
features = Features(features)

ds = load_dataset("json", data_files="./ds.json", features=features)

for row in ds["train"]:
    print(row)
```

we get a dataset that looks like this:

| **Instruction** | **Input**          | **Output**      |
|-----------------|--------------------|-----------------|
| "Do stuff"      | None               | "Answer Stuff"  |
| "Do stuff2"     | None               | "Answer Stuff2" |

### Expected behavior

The input column should contain values other than None for dataset entries that have the "input" attribute set:

| **Instruction** | **Input**          | **Output**      |
|-----------------|--------------------|-----------------|
| "Do stuff"      | None               | "Answer Stuff"  |
| "Do stuff2"     | "Additional Input2" | "Answer Stuff2" |

### Environment info

Python 3.10.10
Datasets 2.11.0
Windows 10
GitHub issue: How to limit the number of examples in dataset, for testing?
State: closed

### Describe the bug

I am using this command:
`data = load_dataset("json", data_files=data_path)`
However, I want to add a parameter, to limit the number of loaded examples to be 10, for development purposes, but can't find this simple parameter.

### Steps to reproduce the bug

In the description.

### Expected behavior

To be able to limit the number of examples

### Environment info

Nothing special
GitHub issue: Calling datasets.load_dataset("text" ...) results in a wrong split.
State: open

### Describe the bug

When creating a text dataset, the training split should have the bulk of the examples by default. Currently, testing does.

### Steps to reproduce the bug

I have a folder with 18K text files in it. Each text file essentially consists in a document or article scraped from online. Calling the following codeL
```
folder_path = "/home/cyril/Downloads/llama_dataset"
data = datasets.load_dataset("text", data_dir=folder_path)
data.save_to_disk("/home/cyril/Downloads/data.hf")

data = datasets.load_from_disk("/home/cyril/Downloads/data.hf")
print(data)
```

Results in the following split:
```
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 2114
    })
    test: Dataset({
        features: ['text'],
        num_rows: 200882
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 152
    })
})
```

It seems to me like the train/test/validation splits are in the wrong order since test split >>>> train_split

### Expected behavior

Train split should have the bulk of the training examples.

### Environment info

datasets 2.11.0, python 3.10.6
GitHub issue: Array2D feature creates a list of list instead of a numpy array
State: closed

### Describe the bug

I'm not sure if this is expected behavior or not. When I create a 2D array using `Array2D`, the data has list type instead of numpy array. I think it should not be the expected behavior especially when I feed a numpy array as input to the data creation function. Why is it converting my array into a list?
Also if I change the first dimension of the `Array2D` shape to None, it's returning array correctly.

### Steps to reproduce the bug

Run this code:
```py
from datasets import Dataset, Features, Array2D
import numpy as np

# you have to change the first dimension of the shape to None to make it return an array
features = Features(dict(seq=Array2D((2,2), 'float32')))  
ds = Dataset.from_dict(dict(seq=[np.random.rand(2,2)]), features=features)
a = ds[0]['seq']
print(a)
print(type(a))
```

The following will be printed in stdout:
```
[[0.8127174377441406, 0.3760348856449127], [0.7510159611701965, 0.4322739541530609]]
<class 'list'>
```

### Expected behavior

Each indexed item should be a list or numpy array.  Currently, `Array((2,2))` yields a list but `Array((None,2))` yields an array.

### Environment info

- `datasets` version: 2.11.0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.9.13
- Huggingface_hub version: 0.13.4
- PyArrow version: 11.0.0
- Pandas version: 1.4.4
GitHub issue: Errror when saving to disk a dataset of images
State: open

### Describe the bug

Hello!

I have an issue when I try to save on disk my dataset of images. The error I get is:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1442, in save_to_disk
    for job_id, done, content in Dataset._save_to_disk_single(**kwargs):
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1473, in _save_to_disk_single
    writer.write_table(pa_table)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/arrow_writer.py", line 570, in write_table
    pa_table = embed_table_storage(pa_table)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 2268, in embed_table_storage
    arrays = [
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 2269, in <listcomp>
    embed_array_storage(table[name], feature) if require_storage_embed(feature) else table[name]
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 1817, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 1817, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 2142, in embed_array_storage
    return feature.embed_storage(array)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/features/image.py", line 269, in embed_storage
    storage = pa.StructArray.from_arrays([bytes_array, path_array], ["bytes", "path"], mask=bytes_array.is_null())
  File "pyarrow/array.pxi", line 2766, in pyarrow.lib.StructArray.from_arrays
  File "pyarrow/array.pxi", line 2961, in pyarrow.lib.c_mask_inverted_from_obj
TypeError: Mask must be a pyarrow.Array of type boolean
```

My dataset is around 50K images, is this error might be due to a bad image?

Thanks for the help.

### Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset("imagefolder", data_dir="/path/to/dataset")
dataset["train"].save_to_disk("./myds", num_shards=40)
```

### Expected behavior

Having my dataset properly saved to disk.

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- PyArrow version: 11.0.0
- Pandas version: 2.0.0
GitHub issue: Handle empty audio
State: closed

Some audio paths exist, but they are empty, and an error will be reported when reading the audio path.How to use the filter function to avoid the empty audio path?
when a audio is empty, when do resample , it will break:
`array, sampling_rate = sf.read(f) array = librosa.resample(array, orig_sr=sampling_rate, target_sr=self.sampling_rate)`
GitHub issue: Return Numpy Array (fixed length) Mode, in __get_item__, Instead of List
State: closed

### Feature request

There are old known issues, but they can be easily forgettable problems in multiprocessing with pytorch-dataloader:

Too high usage of RAM or shared-memory in pytorch when we set num workers > 1 and returning type of dataset or dataloader is "List" or "Dict".
https://github.com/pytorch/pytorch/issues/13246 

With huggingface datasets, unfortunately, the default return type is the list, so the problem is raised too often if we do not set anything for the issue.


However, this issue can be released when the returning output is fixed in length.
Therefore, I request the mode, returning outputs with fixed length (e.g. numpy array) rather than list.

The design would be good when we load datasets as
```python
load_dataset(..., with_return_as_fixed_tensor=True)
```

### Motivation

The general solution for this issue is already in the comments:  https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662 

: Numpy or Pandas seems not to have problems, while both have the string type.
(I'm not sure that the sequence of huggingface datasets can solve this problem as well)


### Your contribution

 I'll read it ! thanks
GitHub issue: Fix xnumpy_load for .npz files
State: closed

PR:
- #5626 

implemented support for streaming `.npy` files by using `numpy.load`.

However, it introduced a bug when used with `.npz` files, within a context manager:
```
ValueError: seek of closed file
```
or in streaming mode:
```
ValueError: I/O operation on closed file.
```

This PR fixes the bug and tests for both `.npy` and `.npz` files.

Fix #5711.
GitHub issue: ArrowNotImplementedError when loading dataset from the hub
State: closed

### Describe the bug

Hello,

I have created a dataset by using the image loader. Once the dataset is created I try to download it and I get the error:
```
Traceback (most recent call last):
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 1860, in _prepare_split_single
    for _, table in generator:
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 69, in _generate_tables
    for batch_idx, record_batch in enumerate(
  File "pyarrow/_parquet.pyx", line 1323, in iter_batches
  File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/load.py", line 1791, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 891, in download_and_prepare
    self._download_and_prepare(
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 986, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 1748, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 1893, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

Create the dataset and push it to the hub:
```python
from datasets import load_dataset

dataset = load_dataset("imagefolder", data_dir="/path/to/dataset")
dataset.push_to_hub("org/dataset-name", private=True, max_shard_size="1GB")
```

Then use it:
```python
from datasets import load_dataset

dataset = load_dataset("org/dataset-name")
```

### Expected behavior

To properly download and use the pushed dataset.

Something else to note is that I specified to have shards of 1GB max, but at the end, for the train set, it is an almost 7GB single file that is pushed.

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- PyArrow version: 11.0.0
- Pandas version: 2.0.0
GitHub issue: OSError: Memory mapping file failed: Cannot allocate memory
State: closed

### Describe the bug

Hello, I have a series of datasets each of 5 GB, 600 datasets in total. So together this makes 3TB. 

When I trying to load all the 600 datasets into memory, I get the above error message. 

Is this normal because I'm hitting the max size of memory mapping of the OS? 


Thank you

```terminal
0_21/cache-e9c42499f65b1881.arrow
load_hf_datasets_from_disk:  82%|████████████████████████████████████████████████████████████████████████████████████████████████████▍                     | 494/600 [07:26<01:35,  1.11it/s]
Traceback (most recent call last):
  File "example_load_genkalm_dataset.py", line 35, in <module>
    multi_ds.post_process(max_node_num=args.max_node_num,max_seq_length=args.max_seq_length,delay=args.delay)
  File "/home/geng/GenKaLM/src/dataloader/dataset.py", line 142, in post_process
    genkalm_dataset = GenKaLM_Dataset.from_hf_dataset(path_or_name=ds_path, max_seq_length=self.max_seq_length,
  File "/home/geng/GenKaLM/src/dataloader/dataset.py", line 47, in from_hf_dataset
    hf_ds = load_from_disk(path_or_name)
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/load.py", line 1848, in load_from_disk
    return Dataset.load_from_disk(dataset_path, keep_in_memory=keep_in_memory, storage_options=storage_options)
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1549, in load_from_disk
    arrow_table = concat_tables(
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/table.py", line 1805, in concat_tables
    tables = list(tables)
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1550, in <genexpr>
    table_cls.from_file(Path(dataset_path, data_file["filename"]).as_posix())
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/table.py", line 1065, in from_file
    table = _memory_mapped_arrow_table_from_file(filename)
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/table.py", line 50, in _memory_mapped_arrow_table_from_file
    memory_mapped_stream = pa.memory_map(filename)
  File "pyarrow/io.pxi", line 950, in pyarrow.lib.memory_map
  File "pyarrow/io.pxi", line 911, in pyarrow.lib.MemoryMappedFile._open
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: Memory mapping file failed: Cannot allocate memory
```








### Steps to reproduce the bug

Sorry I can not provide a reproducible code as the data is stored on my server and it's too large to share.

### Expected behavior

I expect the 3TB of data can be fully mapped to memory

### Environment info


- `datasets` version: 2.9.0
- Platform: Linux-4.15.0-204-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyArrow version: 11.0.0
- Pandas version: 1.0.1
GitHub issue: Manually dataset info made not taken into account
State: closed

### Describe the bug

Hello,

I'm manually building an image dataset with the `from_dict` approach. I also build the features with the `cast_features` methods. Once the dataset is created I push it on the hub, and a default `dataset_infos.json` file seems to have been automatically added to the repo in same time. Hence I update it manually with all the missing info, but when I download the dataset the info are never updated.

Former `dataset_infos.json` file:
```
{"default": {
    "description": "",
    "citation": "",
    "homepage": "",
    "license": "",
    "features": {
        "image": {
            "_type": "Image"
        },
        "labels": {
            "names": [
                "Fake",
                "Real"
            ],
            "_type": "ClassLabel"
        }
    },
    "splits": {
        "validation": {
            "name": "validation",
            "num_bytes": 901010094.0,
            "num_examples": 3200,
            "dataset_name": null
        },
        "train": {
            "name": "train",
            "num_bytes": 901010094.0,
            "num_examples": 3200,
            "dataset_name": null
        }
    },
    "download_size": 1802008414,
    "dataset_size": 1802020188.0,
    "size_in_bytes": 3604028602.0
}}
```

After I update it manually it looks like:
```
{
   "bstrai--deepfake-detection":{
      "description":"",
      "citation":"",
      "homepage":"",
      "license":"",
      "features":{
         "image":{
            "decode":true,
            "id":null,
            "_type":"Image"
         },
         "labels":{
            "num_classes":2,
            "names":[
               "Fake",
               "Real"
            ],
            "id":null,
            "_type":"ClassLabel"
         }
      },
      "supervised_keys":{
         "input":"image",
         "output":"labels"
      },
      "task_templates":[
         {
            "task":"image-classification",
            "image_column":"image",
            "label_column":"labels"
         }
      ],
      "config_name":null,
      "splits":{
         "validation":{
            "name":"validation",
            "num_bytes":36627822,
            "num_examples":123,
            "dataset_name":"deepfake-detection"
         },
         "train":{
            "name":"train",
            "num_bytes":901023694,
            "num_examples":3200,
            "dataset_name":"deepfake-detection"
         }
      },
      "download_checksums":null,
      "download_size":937562209,
      "dataset_size":937651516,
      "size_in_bytes":1875213725
   }
}
```

Anything I should do to have the new infos in the `dataset_infos.json` to be taken into account? Or it is not possible yet?

Thanks!

### Steps to reproduce the bug

-

### Expected behavior

-

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- PyArrow version: 11.0.0
- Pandas version: 2.0.0
GitHub issue: Dataset sizes are in MiB instead of MB in dataset cards
State: closed

As @severo reported in an internal discussion (https://github.com/huggingface/moon-landing/issues/5929):

Now we show the dataset size:
- from the dataset card (in the side column)
- from the datasets-server (in the viewer)

But, even if the size is the same, we see a mismatch because the viewer shows MB, while the info from the README generally shows MiB (even if it's written MB -> https://huggingface.co/datasets/blimp/blob/main/README.md?code=true#L1932)

<img width="664" alt="Capture d’écran 2023-04-04 à 10 16 01" src="https://user-images.githubusercontent.com/1676121/229730887-0bd8fa6e-9462-46c6-bd4e-4d2c5784cabb.png">

TODO: Values to be fixed in: `Size of downloaded dataset files:`, `Size of the generated dataset:` and `Total amount of disk used:`
- [x] Bulk edit on the Hub to fix this in all canonical datasets
- [x] Bulk PR on the Hub to fix ancient canonical datasets that were moved to organizations
GitHub issue: Support categorical data types for Parquet
State: closed

### Feature request

Huggingface datasets does not seem to support categorical / dictionary data types for Parquet as of now. There seems to be a `TODO` in the code for this feature but no implementation yet. Below you can find sample code to reproduce the error that is currently thrown when attempting to read a Parquet file with categorical columns:

```python
import pandas as pd
import pyarrow.parquet as pq

from datasets import load_dataset

# Create categorical sample DataFrame
df = pd.DataFrame({'type': ['foo', 'bar']}).astype('category')
df.to_parquet('data.parquet')

# Read back as pyarrow table
table = pq.read_table('data.parquet')
print(table.schema)
# type: dictionary<values=string, indices=int32, ordered=0>

# Load with huggingface datasets
load_dataset('parquet', data_files='data.parquet')
```

Error:
```
Traceback (most recent call last):                                                                                                                                                                                                        
  File ".venv/lib/python3.10/site-packages/datasets/builder.py", line 1875, in _prepare_split_single
    writer.write_table(table)
  File ".venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 566, in write_table
    self._build_writer(inferred_schema=pa_table.schema)
  File ".venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 379, in _build_writer
    inferred_features = Features.from_arrow_schema(inferred_schema)                                                                                     
  File ".venv/lib/python3.10/site-packages/datasets/features/features.py", line 1622, in from_arrow_schema
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File ".venv/lib/python3.10/site-packages/datasets/features/features.py", line 1622, in <dictcomp>
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File ".venv/lib/python3.10/site-packages/datasets/features/features.py", line 1361, in generate_from_arrow_type
    raise NotImplementedError  # TODO(thom) this will need access to the dictionary as well (for labels). I.e. to the py_table
NotImplementedError
```

### Motivation

Categorical data types, as offered by Pandas and implemented with the `DictionaryType` dtype in `pyarrow` can significantly reduce dataset size and are a handy way to turn textual features into numerical representations and back. Lack of support in Huggingface datasets greatly reduces compatibility with a common Pandas / Parquet feature.

### Your contribution

I could provide a PR. However, it would be nice to have an initial complexity estimate from one of the core developers first.
GitHub issue: Getting next item from IterableDataset took forever.
State: closed

### Describe the bug

I have a large dataset, about 500GB. The format of the dataset is parquet. 

I then load the dataset and try to get the first item
```python
def get_one_item():
    dataset = load_dataset("path/to/datafiles", split="train", cache_dir=".", streaming=True)
    dataset = dataset.filter(lambda example: example['text'].startswith('Ar'))
    print(next(iter(dataset)))
```

However, this function never finish. I waited ~10mins, the function was still running so I killed the process. I'm now using `line_profiler` to profile how long it would take to return one item. I'll be patient and wait for as long as it needs. 

I suspect the filter operation is the reason why it took so long. Can I get some possible reasons behind this?

### Steps to reproduce the bug

Unfortunately without my data files, there is no way to reproduce this bug.

### Expected behavior

With `IteralbeDataset`, I expect the first item to be returned instantly.

### Environment info

- datasets version: 2.11.0 
- python: 3.7.12