xyx138/github-issues huggingface dataset:
Fix the environment variable for huggingface cache

Resolve  #6256. As far as I tested, `HF_DATASETS_CACHE` was ignored and I could not specify the cache directory at all except for the default one by this environment variable. `HF_HOME` has worked. Perhaps the recent change on file downloading by `huggingface_hub` could affect this bug.

In my testing, I could not specify the cache directory even by `load_dataset("dataset_name" cache_dir="...")`. It might be another issue. I also welcome any advice to solve this issue.
ConnectionError: Couldn't reach 'allenai/c4' on the Hub (ConnectionError)数据集下不下来，怎么回事

### Describe the bug

from datasets import load_dataset
print("11")
traindata = load_dataset('ptb_text_only', 'penn_treebank', split='train')
print("22")
valdata = load_dataset('ptb_text_only',
                           'penn_treebank',
                           split='validation')

### Steps to reproduce the bug

1

### Expected behavior

1

### Environment info

1
Support of num_workers (multiprocessing) in map for IterableDataset

### Feature request

Currently, IterableDataset doesn't support setting num_worker in .map(), which results in slow processing here. Could we add support for it? As .map() can be run in the batch fashion (e.g., batch_size is default to 1000 in datasets), it seems to be doable for IterableDataset as the regular Dataset.

### Motivation

Improving data processing efficiency

### Your contribution

Testing
Add repeat() for iterable datasets

### Feature request

It would be useful to be able to straightforwardly repeat iterable datasets indefinitely, to provide complete control over starting and ending of iteration to the user.

An IterableDataset.repeat(n) function could do this automatically

### Motivation

This feature was discussed in this issue https://github.com/huggingface/datasets/issues/7147, and would resolve the need to use the hack of interleave datasets with probability 0 as a simple way to achieve this functionality.

An additional benefit might be the simplification of the use of iterable datasets in a distributed setting:
If the user can assume that datasets will repeat indefinitely, then issues around different numbers of samples appearing on different devices (e.g. https://github.com/huggingface/datasets/issues/6437, https://github.com/huggingface/datasets/issues/6594, https://github.com/huggingface/datasets/issues/6623, https://github.com/huggingface/datasets/issues/6719) can potentially be straightforwardly resolved by simply doing:

ids.repeat(None).take(n_samples_per_epoch)


### Your contribution

I'm not familiar enough with the codebase to assess how straightforward this would be to implement.

If it might be very straightforward, I could possibly have a go.
Solution to issue: #7080 Modified load_dataset function, so that it prompts the user to select a dataset when subdatasets or splits (train, test) are available

# Feel free to give suggestions please..

### This PR is raised because of issue: https://github.com/huggingface/datasets/issues/7080

![image](https://github.com/user-attachments/assets/8fbc604f-f0a5-4a59-a63e-aa4c26442c83)

### This PR gives solution to https://github.com/huggingface/datasets/issues/7080

1. Checking whether the dataset has splits or subdatasets.
2. Printing the available splits/subdatasets.
3. Asking the user to choose which one to load.
4. Loading only the selected dataset based on the user's input.

### Key Changes:

1. Available Splits/Subdatasets: The code checks for available splits/subdatasets using builder_instance.info.splits.keys().
2. User Prompt: If splits are found, it prints them out and prompts the user to select one.
3. Loading Based on User Input: The dataset is loaded based on the user's choice.

This way, the dataset loading function will interactively prompt the user to select which subdataset or split they want to load instead of automatically loading all of them.
Datasets conflicts with fsspec 2024.9

### Describe the bug

Installing both in latest versions are not possible

`pip install "datasets==3.0.1" "fsspec==2024.9.0"`

But using older version of datasets is ok

`pip install "datasets==1.24.4" "fsspec==2024.9.0"`



### Steps to reproduce the bug

`pip install "datasets==3.0.1" "fsspec==2024.9.0"`


### Expected behavior

install both versions.

### Environment info

debian 11.
python 3.10.15
CI benchmarks are broken

Since Aug 30, 2024, CI benchmarks are broken: https://github.com/huggingface/datasets/actions/runs/11108421214/job/30861323975
```
{"level":"error","message":"Resource not accessible by integration","name":"HttpError","request":{"body":"{\"body\":\"<details>\\n<summary>Show benchmarks</summary>\\n\\nPyArrow==8.0.0\\n\\n<details>\\n<summary>Show updated benchmarks!</summary>\\n\\n### Benchmark: benchmark_array_xd.json\\n\\n| metric | read_batch_formatted_as_numpy after write_array2d |
...
"headers":{"accept":"application/vnd.github.v3+json","authorization":"token [REDACTED]","content-type":"application/json; charset=utf-8","user-agent":"octokit-rest.js/18.0.0 octokit-core.js/3.6.0 Node.js/16.20.2 (linux; x64)"},"method":"POST","request":{"agent":{"_events":{},"_eventsCount":2,"cache":
...
"response":{"data":{"documentation_url":"https://docs.github.com/rest/issues/comments#create-an-issue-comment","message":"Resource not accessible by integration","status":"403"},
...
"stack":"HttpError: Resource not accessible by integration\n    at /usr/lib/node_modules/@dvcorg/cml/node_modules/@octokit/request/dist-node/index.js:86:21\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at async Job.doExecute (/usr/lib/node_modules/@dvcorg/cml/node_modules/bottleneck/light.js:405:18)","status":403}
```
Pin dill<0.3.9 to fix CI

Pin dill<0.3.9 to fix CI for deps-latest.

Note that dill-0.3.9 was released yesterday Sep 29, 2024:
- https://pypi.org/project/dill/0.3.9/
- https://github.com/uqfoundation/dill/releases/tag/0.3.9

Fix #7183.
Support features in metadata configs

Support features in metadata configs, like:
```
configs:
  - config_name: default
    features:
      - name: id
        dtype: int64
      - name:  name
        dtype: string
      - name: score
        dtype: float64
```

This will allow to avoid inference of data types.

Currently, we allow passing this information in the `dataset_info` (instead of `configs`) field, but this is not intuitive and it is not properly documented.

TODO:
- [ ] Document usage
Memory leak when wrapping datasets into PyTorch Dataset without explicit deletion

### Describe the bug

I've encountered a memory leak when wrapping the HuggingFace dataset into a PyTorch Dataset. The RAM usage constantly increases during iteration if items are not explicitly deleted after use.








### Steps to reproduce the bug


Steps to reproduce:

Create a PyTorch Dataset wrapper for 'nebula/cc12m':
````
from torch.utils.data import Dataset
from tqdm import tqdm
from datasets import load_dataset
from torchvision import transforms

Image.MAX_IMAGE_PIXELS = None

class CC12M(Dataset):
    def __init__(self, path_or_name='nebula/cc12m', split='train', transform=None, single_caption=True):
        self.raw_dataset = load_dataset(path_or_name)[split]

        if transform is None:
            self.transform = transforms.Compose([
                transforms.Resize((224, 224)),
                transforms.CenterCrop(224),
                transforms.ToTensor(),
                transforms.Normalize(
                    mean=[0.48145466, 0.4578275, 0.40821073],
                    std=[0.26862954, 0.26130258, 0.27577711]
                )
            ])
        else:
            self.transform = transforms.Compose(transform)

        self.single_caption = single_caption
        self.length = len(self.raw_dataset)

    def __len__(self):
        return self.length

    def __getitem__(self, index):
        item = self.raw_dataset[index]
        caption = item['txt']
        with io.BytesIO(item['webp']) as buffer:
            image = Image.open(buffer).convert('RGB')
            if self.transform:
                image = self.transform(image)
        # del item  # Uncomment this line to prevent the memory leak
        return image, caption
````

Iterate through the dataset without the del item line in __getitem__.

Observe RAM usage increasing constantly.

Add del item at the end of __getitem__:

```
def __getitem__(self, index):
    item = self.raw_dataset[index]
    caption = item['txt']
    with io.BytesIO(item['webp']) as buffer:
        image = Image.open(buffer).convert('RGB')
        if self.transform:
            image = self.transform(image)
    del item  # This line prevents the memory leak
    return image, caption
```


Iterate through the dataset again and observe that RAM usage remains stable.



### Expected behavior


Expected behavior:
RAM usage should remain stable during iteration without needing to explicitly delete items.

Actual behavior:
RAM usage constantly increases unless items are explicitly deleted after use

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-4.18.0-513.5.1.el8_9.x86_64-x86_64-with-glibc2.28
- Python version: 3.12.4
- `huggingface_hub` version: 0.24.6
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1
[FSTimeoutError] load_dataset

### Describe the bug

When using `load_dataset`to load [HuggingFaceM4/VQAv2](https://huggingface.co/datasets/HuggingFaceM4/VQAv2), I am getting `FSTimeoutError`. 

### Error

```
TimeoutError: 

The above exception was the direct cause of the following exception:

FSTimeoutError                            Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/fsspec/asyn.py](https://klh9mr78js-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240924-060116_RC00_678132060#) in sync(loop, func, timeout, *args, **kwargs)
     99     if isinstance(return_result, asyncio.TimeoutError):
    100         # suppress asyncio.TimeoutError, raise FSTimeoutError
--> 101         raise FSTimeoutError from return_result
    102     elif isinstance(return_result, BaseException):
    103         raise return_result

FSTimeoutError:
```

It usually fails around 5-6 GB.

<img width="847" alt="Screenshot 2024-09-26 at 9 10 19 PM" src="https://github.com/user-attachments/assets/ff91995a-fb55-4de6-8214-94025d6c8470">


### Steps to reproduce the bug

To reproduce it, run this in colab notebook:

```
!pip install -q -U datasets

from datasets import load_dataset
ds = load_dataset('HuggingFaceM4/VQAv2', split="train[:10%]")
```



### Expected behavior

It should download properly.

### Environment info

Using Colab Notebook.
Support JSON lines with missing columns

Support JSON lines with missing columns.

Fix #7169.

The implemented test raised:
```
datasets.table.CastError: Couldn't cast
age: int64
to
{'age': Value(dtype='int32', id=None), 'name': Value(dtype='string', id=None)}
because column names don't match
```

Related to:
- #7160 
- #7162
Error Mapping on sd3, sdxl and upcoming flux controlnet training scripts in diffusers

### Describe the bug

```
Map:   6%|██████                                                                                                   | 8000/138120 [19:27<5:16:36,  6.85 examples/s]
Traceback (most recent call last):
  File "/workspace/diffusers/examples/controlnet/train_controlnet_sd3.py", line 1416, in <module>
    main(args)
  File "/workspace/diffusers/examples/controlnet/train_controlnet_sd3.py", line 1132, in main
    train_dataset = train_dataset.map(compute_embeddings_fn, batched=True, new_fingerprint=new_fingerprint)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/arrow_dataset.py", line 560, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
                                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/arrow_dataset.py", line 3035, in map
    for rank, done, content in Dataset._map_single(**dataset_kwargs):
  File "/usr/local/lib/python3.11/dist-packages/datasets/arrow_dataset.py", line 3461, in _map_single
    writer.write_batch(batch)
  File "/usr/local/lib/python3.11/dist-packages/datasets/arrow_writer.py", line 567, in write_batch
    self.write_table(pa_table, writer_batch_size)
  File "/usr/local/lib/python3.11/dist-packages/datasets/arrow_writer.py", line 579, in write_table
    pa_table = pa_table.combine_chunks()
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 4387, in pyarrow.lib.Table.combine_chunks
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/usr/local/lib/python3.11/dist-packages/accelerate/commands/launch.py", line 1174, in launch_command
    simple_launcher(args)
  File "/usr/local/lib/python3.11/dist-packages/accelerate/commands/launch.py", line 769, in simple_launcher
  ```

### Steps to reproduce the bug

The dataset has no problem training on sd1.5 controlnet train script



### Expected behavior

Script not randomly erroing with error above

### Environment info

- `datasets` version: 3.0.0
- Platform: Linux-6.5.0-44-generic-x86_64-with-glibc2.35
- Python version: 3.11.9
- `huggingface_hub` version: 0.25.1
- PyArrow version: 17.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.6.1


training on A100
Set explicit seed in iterable dataset ddp shuffling example 

### Describe the bug

In the examples section of the iterable dataset docs https://huggingface.co/docs/datasets/en/package_reference/main_classes#datasets.IterableDataset
the ddp example shuffles without seeding

```python
from datasets.distributed import split_dataset_by_node
ids = ds.to_iterable_dataset(num_shards=512)
ids = ids.shuffle(buffer_size=10_000)  # will shuffle the shards order and use a shuffle buffer when you start iterating
ids = split_dataset_by_node(ds, world_size=8, rank=0)  # will keep only 512 / 8 = 64 shards from the shuffled lists of shards when you start iterating
dataloader = torch.utils.data.DataLoader(ids, num_workers=4)  # will assign 64 / 4 = 16 shards from this node's list of shards to each worker when you start iterating
for example in ids:
    pass
```

This code would - I think - raise an error due to the lack of an explicit seed:

https://github.com/huggingface/datasets/blob/2eb4edb97e1a6af2ea62738ec58afbd3812fc66e/src/datasets/iterable_dataset.py#L1707-L1711


### Steps to reproduce the bug

Run example code

### Expected behavior

Add explicit seeding to example code

### Environment info

latest datasets
Dataset viewer not working! Failure due to more than 32 splits.

Hello guys, 

I have a dataset and I didn't know I couldn't upload more than 32 splits. Now, my dataset viewer is not working. I don't have the dataset locally on my node anymore and recreating would take a week. And I have to publish the dataset coming Monday. I read about the practice, how I can resolve it and avoid this issue in the future. But, at the moment I need a hard fix for two of my datasets. 

And I don't want to mess or change anything and allow everyone in public to see the dataset and interact with it. Can you please help me?

https://huggingface.co/datasets/laion/Wikipedia-X
https://huggingface.co/datasets/laion/Wikipedia-X-Full
Datasets Unknown Keyword Argument Error - task_templates

### Describe the bug

Issue

```python
from datasets import load_dataset
examples = load_dataset('facebook/winoground', use_auth_token=<YOUR USER ACCESS TOKEN>)
```

Gives error

```
TypeError: DatasetInfo.__init__() got an unexpected keyword argument 'task_templates'
```

A simple downgrade to lower `datasets v 2.21.0` solves it.




### Steps to reproduce the bug

1. `pip install datsets`
2.
```python
from datasets import load_dataset
examples = load_dataset('facebook/winoground', use_auth_token=<YOUR USER ACCESS TOKEN>)
```



### Expected behavior

Should load the dataset correctly.

### Environment info

- Datasets version  `3.0.0`
- `transformers` version: 4.45.0.dev0
- Platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.35
- Python version: 3.12.4
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.5
- Accelerate version: 0.35.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
Bug: Error when downloading mteb/mtop_domain

### Describe the bug

When downloading the dataset "mteb/mtop_domain", ran into the following error:

```
Traceback (most recent call last):
  File "/share/project/xzy/test/test_download.py", line 3, in <module>
    data = load_dataset("mteb/mtop_domain", "en", trust_remote_code=True)
  File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 2606, in load_dataset
    builder_instance = load_dataset_builder(
  File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 2277, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1923, in dataset_module_factory
    raise e1 from None
  File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1896, in dataset_module_factory
    ).get_module()
  File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1507, in get_module
    local_path = self.download_loading_script()
  File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1467, in download_loading_script
    return cached_path(file_path, download_config=download_config)
  File "/opt/conda/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 211, in cached_path
    output_path = get_from_cache(
  File "/opt/conda/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 689, in get_from_cache
    fsspec_get(
  File "/opt/conda/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 395, in fsspec_get
    fs.get_file(path, temp_file.name, callback=callback)
  File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/hf_file_system.py", line 648, in get_file
    http_get(
  File "/opt/conda/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 578, in http_get
    raise EnvironmentError(
OSError: Consistency check failed: file should be of size 2191 but has size 2190 ((…)ets/mteb/mtop_domain@main/mtop_domain.py).
We are sorry for the inconvenience. Please retry with `force_download=True`.
If the issue persists, please let us know by opening an issue on https://github.com/huggingface/huggingface_hub.
```
Try to download through HF datasets directly but got the same error as above.
```python
from datasets import load_dataset

data = load_dataset("mteb/mtop_domain", "en")
```

### Steps to reproduce the bug

```python
from datasets import load_dataset

data = load_dataset("mteb/mtop_domain", "en", force_download=True)
```
With and without `force_download=True` both ran into the same error.

### Expected behavior

Should download the dataset successfully.

### Environment info

- datasets version: 2.21.0
- huggingface-hub version: 0.24.6
IterableDataset strange deadlock

### Describe the bug

```
import datasets
import torch.utils.data


num_shards = 1024


def gen(shards):
    for shard in shards:
        if shard < 25:
            yield {"shard": shard}


def main():
    dataset = datasets.IterableDataset.from_generator(
        gen,
        gen_kwargs={"shards": list(range(num_shards))},
    )
    dataset = dataset.shuffle(buffer_size=1)
    dataset = datasets.interleave_datasets(
        [dataset, dataset], probabilities=[1, 0], stopping_strategy="all_exhausted"
    )
    dataset = dataset.shuffle(buffer_size=1)

    dataloader = torch.utils.data.DataLoader(
        dataset,
        batch_size=8,
        num_workers=8,
    )

    for i, batch in enumerate(dataloader):
        print(batch)
        if i >= 10:
            break
    print()


if __name__ == "__main__":
    for _ in range(100):
        main()
```

### Steps to reproduce the bug

Running the script above, at some point it will freeze.

- Changing `num_shards` from 1024 to 25 avoids the issue
- Commenting out the final shuffle avoids the issue
- Commenting out the interleave_datasets call avoids the issue

As an aside, if you comment out just the final shuffle, the output from interleave_datasets is not shuffled at all even though there's the shuffle before it. So something about that shuffle config is not being propagated to interleave_datasets.

### Expected behavior

The script should not freeze.

### Environment info

- `datasets` version: 3.0.0
- Platform: macOS-14.6.1-arm64-arm-64bit
- Python version: 3.12.5
- `huggingface_hub` version: 0.24.7
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1


I observed this with 2.21.0 initially, then tried upgrading to 3.0.0 and could still repro.
Fix key error in webdataset

I was running into

```
example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}
KeyError: 'png'
```

The issue is that a filename may have multiple "." e.g. `22.05.png`. Changing `split` to `rsplit` fixes it.

Related https://github.com/huggingface/datasets/issues/6880
Modify add_column() to optionally accept a FeatureType as param

Fix #7142.

**Before (Add + Cast)**:

```
from datasets import load_dataset, Value
ds = load_dataset("rotten_tomatoes", split="test")
lst = [i for i in range(len(ds))]

ds = ds.add_column("new_col", lst)
# Assigns int64 to new_col by default
print(ds.features)

ds = ds.cast_column("new_col", Value(dtype="uint16", id=None))
print(ds.features)
```


**Before (Numpy Workaround)**:

```
from datasets import load_dataset
import numpy as np
ds = load_dataset("rotten_tomatoes", split="test")
lst = [i for i in range(len(ds))]

ds = ds.add_column("new_col", np.array(lst, dtype=np.uint16))
print(ds.features)
```

**After**:
```
from datasets import load_dataset, Value
ds = load_dataset("rotten_tomatoes", split="test")
lst = [i for i in range(len(ds))]
val = Value(dtype="uint16", id=None))
ds = ds.add_column("new_col", lst, feature=val)
print(ds.features)
```
Specifying datatype when adding a column to a dataset.

### Feature request

There should be a way to specify the datatype of a column in `datasets.add_column()`. 

### Motivation

To specify a custom datatype, we have to use `datasets.add_column()` followed by `datasets.cast_column()` which is slow for large datasets. Another workaround is to pass a `numpy.array()` of desired type to the `datasets.add_column()` function.

IMO this functionality should be natively supported.

https://discuss.huggingface.co/t/add-column-with-a-particular-type-in-datasets/95674



### Your contribution

I can submit a PR for this.
Older datasets throwing safety errors with 2.21.0

### Describe the bug

The dataset loading was throwing some safety errors for this popular dataset `wmt14`. 

[in]:

```
import datasets

# train_data = datasets.load_dataset("wmt14", "de-en", split="train")
train_data = datasets.load_dataset("wmt14", "de-en", split="train")
val_data = datasets.load_dataset("wmt14", "de-en", split="validation[:10%]")
```


[out]:

```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
[<ipython-input-9-445f0ecc4817>](https://localhost:8080/#) in <cell line: 4>()
      2 
      3 # train_data = datasets.load_dataset("wmt14", "de-en", split="train")
----> 4 train_data = datasets.load_dataset("wmt14", "de-en", split="train")
      5 val_data = datasets.load_dataset("wmt14", "de-en", split="validation[:10%]")

12 frames
[/usr/local/lib/python3.10/dist-packages/huggingface_hub/hf_api.py](https://localhost:8080/#) in __init__(self, **kwargs)
    636         if security is not None:
    637             security = BlobSecurityInfo(
--> 638                 safe=security["safe"], av_scan=security["avScan"], pickle_import_scan=security["pickleImportScan"]
    639             )
    640         self.security = security

KeyError: 'safe'
```


### Steps to reproduce the bug

See above.

### Expected behavior

Dataset properly loaded.

### Environment info

version: 2.21.0
Use load_dataset to load imagenet-1K But find a empty dataset

### Describe the bug

```python
def get_dataset(data_path, train_folder="train", val_folder="val"):
    traindir = os.path.join(data_path, train_folder)
    valdir = os.path.join(data_path, val_folder)

    def transform_val_examples(examples):
        transform = Compose([
            Resize(256),
            CenterCrop(224),
            ToTensor(),
        ])
        examples["image"] = [transform(image.convert("RGB")) for image in examples["image"]]
        return examples

    def transform_train_examples(examples):
        transform = Compose([
            RandomResizedCrop(224),
            RandomHorizontalFlip(),
            ToTensor(),
        ])
        examples["image"] = [transform(image.convert("RGB")) for image in examples["image"]]
        return examples

    # @fengsicheng: This way is very slow for big dataset like ImageNet-1K (but can pass the network problem using local dataset)
    # train_set = load_dataset("imagefolder", data_dir=traindir, num_proc=4)
    # test_set = load_dataset("imagefolder", data_dir=valdir, num_proc=4)

    train_set = load_dataset("imagenet-1K", split="train", trust_remote_code=True)                                                                                                                                                                                                            
    test_set = load_dataset("imagenet-1K", split="test", trust_remote_code=True)

    print(train_set["label"])

    train_set.set_transform(transform_train_examples)
    test_set.set_transform(transform_val_examples)

    return train_set, test_set
```
    above the code, but output of the print is a list of None:
    
<img width="952" alt="image" src="https://github.com/user-attachments/assets/c4e2fdd8-3b8f-481e-8f86-9bbeb49d79fb">


### Steps to reproduce the bug

1. just ran the code 
2. see the print


### Expected behavior

I do not know how to fix this, can anyone provide help or something? It is hurry for me

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-5.4.0-190-generic-x86_64-with-glibc2.31
- Python version: 3.10.14
- `huggingface_hub` version: 0.24.6
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1
Cache only changed columns?

### Feature request

Cache only the actual changes to the dataset i.e. changed columns.

### Motivation

I realized that caching actually saves the complete dataset again.
This is especially problematic for image datasets if one wants to only change another column e.g. some metadata and then has to save 5 TB again.

### Your contribution

Is this even viable in the current architecture of the package?
I quickly looked into it and it seems it would require significant changes.

I would spend some time looking into this but maybe somebody could help with the feasibility and some plan to implement before spending too much time on it?
[BUG] dataset_info sequence unexpected behavior in README.md YAML

### Describe the bug

When working on `dataset_info` yaml, I find my data column with format `list[dict[str, str]]` cannot be coded correctly.

My data looks like
```
{"answers":[{"text": "ADDRESS", "label": "abc"}]}
```

My `dataset_info` in README.md is:
```
dataset_info:
- config_name: default
  features:
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: label
      dtype: string
```

**Error log**:
```
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from list<item: struct<text: string, label: string>> to struct using function cast_struct
```

## Potential Reason
After some analysis, it turns out that my yaml config is requiring `dict[str, list[str]]` instead of `list[dict[str, str]]`. It would work if I change my data to
```
{"answers":{"text": ["ADDRESS"], "label": ["abc", "def"]}}
```

These following 2 different `dataset_info` are actually equivalent.

```
dataset_info:
- config_name: default
  features:
  - name: answers
    dtype:
    - name: text
      sequence: string
    - name: label
      sequence: string

dataset_info:
- config_name: default
  features:
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: label
      dtype: string
```



### Steps to reproduce the bug


```
# README.md
---
dataset_info:
- config_name: default
  features:
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: label
      dtype: string
configs:
- config_name: default
  default: true
  data_files:
  - split: train
    path:
    - "test.jsonl"
---



# test.jsonl

# expected but not working
{"answers":[{"text": "ADDRESS", "label": "abc"}]}

# unexpected but working
{"answers":{"text": ["ADDRESS"], "label": ["abc", "def"]}}

```

### Expected behavior

```
dataset_info:
- config_name: default
  features:
  - name: answers
    sequence:
    - name: text
      dtype: string
    - name: label
      dtype: string
```
Should work on following data format:

```
{"answers":[{"text":"ADDRESS", "label": "abc"}]}
```

### Environment info

- `datasets` version: 2.21.0
- Platform: macOS-14.6.1-arm64-arm-64bit
- Python version: 3.12.4
- `huggingface_hub` version: 0.24.5
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1
Bug: Type Mismatch in Dataset Mapping

# Issue: Type Mismatch in Dataset Mapping

## Description
There is an issue with the `map` function in the `datasets` library where the mapped output does not reflect the expected type change. After applying a mapping function to convert an integer label to a string, the resulting type remains an integer instead of a string.

## Reproduction Code
Below is a Python script that demonstrates the problem:

```python
from datasets import Dataset

# Original data
data = {
    'text': ['Hello', 'world', 'this', 'is', 'a', 'test'],
    'label': [0, 1, 0, 1, 1, 0]
}

# Creating a Dataset object
dataset = Dataset.from_dict(data)

# Mapping function to convert label to string
def add_one(example):
    example['label'] = str(example['label'])
    return example

# Applying the mapping function
dataset = dataset.map(add_one)

# Iterating over the dataset to show results
for item in dataset:
    print(item)
    print(type(item['label']))
```

## Expected Output
After applying the mapping function, the expected output should have the `label` field as strings:

```plaintext
{'text': 'Hello', 'label': '0'}
<class 'str'>
{'text': 'world', 'label': '1'}
<class 'str'>
{'text': 'this', 'label': '0'}
<class 'str'>
{'text': 'is', 'label': '1'}
<class 'str'>
{'text': 'a', 'label': '1'}
<class 'str'>
{'text': 'test', 'label': '0'}
<class 'str'>
```

## Actual Output
The actual output still shows the `label` field values as integers:

```plaintext
{'text': 'Hello', 'label': 0}
<class 'int'>
{'text': 'world', 'label': 1}
<class 'int'>
{'text': 'this', 'label': 0}
<class 'int'>
{'text': 'is', 'label': 1}
<class 'int'>
{'text': 'a', 'label': 1}
<class 'int'>
{'text': 'test', 'label': 0}
<class 'int'>
```

## Why necessary
In the case of Image process we often need to convert PIL to tensor with same column name.

Thank for every dev who review this issue. 🤗
Fix data file module inference

I saved a dataset with two splits to disk with `DatasetDict.save_to_disk`. The train is bigger and ended up in 10 shards, whereas the test split only resulted in 1 split.

Now when trying to load the dataset, an error is raised that not all splits have the same data format:

> ValueError: Couldn't infer the same data file format for all splits. Got {NamedSplit('train'): ('arrow', {}), NamedSplit('test'): ('json', {})}

This is not expected because both splits are saved as arrow files.

I did some debugging and found that this is the case because the list of data_files includes a `state.json` file. 

Now this means for train split I get 10 ".arrow" and 1 ".json" file. Since datasets picks based on the most common extension this is correctly inferred as "arrow". In the test split, there is 1 .arrow and 1 .json file. Given the function description:

> It picks the module based on the most common file extension.
    In case of a draw ".parquet" is the favorite, and then alphabetical order.

This is not quite true though, because in a tie the extensions are actually based on reverse-alphabetical order:
```
for (ext, _), _ in sorted(extensions_counter.items(), key=sort_key, *reverse=True*):
```

Which thus leads to the module wrongly inferred as "json", whereas it should be "arrow", matching the train split.

I first thought about adding "state.json" in the list of excluded files for the inference: https://github.com/huggingface/datasets/blob/main/src/datasets/load.py#L513. However, I think from digging into the code it looks like the right thing to do is to exclude it in the list of `data_files` to start with, because it is more of a metadata than a data file.
Filter Large Dataset Entry by Entry

### Feature request

I am not sure if this is a new feature, but I wanted to post this problem here, and hear if others have ways of optimizing and speeding up this process.

Let's say I have a really large dataset that I cannot load into memory. At this point, I am only aware of `streaming=True` to load the dataset. Now, the dataset consists of many tables. Ideally, I would want to have some simple filtering criterion, such that I only see the "good" tables. Here is an example of what the code might look like:

```
dataset = load_dataset(
    "really-large-dataset",
    streaming=True
)
# And let's say we process the dataset bit by bit because we want intermediate results
dataset = islice(dataset, 10000)

# Define a function to filter the data
def filter_function(table):
    if some_condition:
        return True
    else:
        return False

# Use the filter function on your dataset
filtered_dataset = (ex for ex in dataset if filter_function(ex))
```

And then I work on the processed dataset, which would be magnitudes faster than working on the original. I would love to hear if the problem setup + solution makes sense to people, and if anyone has suggestions!

### Motivation

See description above

### Your contribution

Happy to make PR if this is a new feature
Caching shuffles by np.random.Generator results in unintiutive behavior

### Describe the bug

Create a dataset. Save it to disk. Load from disk. Shuffle, usning a `np.random.Generator`. Iterate. Shuffle again. Iterate. The iterates are different since the supplied np.random.Generator has progressed between the shuffles.

Load dataset from disk again. Shuffle and Iterate. See same result as before. Shuffle and iterate, and this time it does not have the same shuffling as ion previous run.

The motivation is I have a deep learning loop with 

```
for epoch in range(10):
    for batch in dataset.shuffle(generator=generator).iter(batch_size=32):
        .... # do stuff
```
where I want a new shuffling at every epoch. Instead I get the same shuffling.

### Steps to reproduce the bug

Run the code below two times. 

```python
import datasets
import numpy as np

generator = np.random.default_rng(0)
ds = datasets.Dataset.from_dict(mapping={"X":range(1000)})
ds.save_to_disk("tmp")
print("First loop: ", end="")
for _ in range(10):
    print(next(ds.shuffle(generator=generator).iter(batch_size=1))['X'], end=", ")
print("")

print("Second loop: ", end="")
ds = datasets.Dataset.load_from_disk("tmp")
for _ in range(10):
    print(next(ds.shuffle(generator=generator).iter(batch_size=1))['X'], end=", ")
print("")
```

The output is:

```
$ python main.py 
Saving the dataset (1/1 shards): 100%|███████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 495019.95 examples/s]
First loop: 459, 739, 72, 943, 241, 181, 845, 830, 896, 334, 
Second loop: 741, 847, 944, 795, 483, 842, 717, 865, 231, 840,
$ python main.py 
Saving the dataset (1/1 shards): 100%|████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 22243.40 examples/s]
First loop: 459, 739, 72, 943, 241, 181, 845, 830, 896, 334, 
Second loop: 741, 741, 741, 741, 741, 741, 741, 741, 741, 741, 
```

The second loop, on the second run, only spits out "741, 741, 741...." which is *not* the desired output

### Expected behavior

I want the dataset to shuffle at every epoch since I provide it with a generator for shuffling. 

### Environment info

Datasets version 2.21.0
Ubuntu linux.
Test get_dataset_config_info with non-existing/gated/private dataset

Test get_dataset_config_info with non-existing/gated/private dataset.

Related to:
- #7109

See also:
- https://github.com/huggingface/dataset-viewer/pull/3037: https://github.com/huggingface/dataset-viewer/pull/3037/commits/bb1a7e00c53c242088597cab6572e4fd57797ecb
Make dataset viewer more flexible in displaying metadata alongside images

### Feature request

To display images with their associated metadata in the dataset viewer, a `metadata.csv` file is required. In the case of a dataset with multiple subsets, this would require the CSVs to be contained in the same folder as the images since they all need to be named `metadata.csv`. The request is that this be made more flexible for datasets with multiple subsets to avoid the need to put a `metadata.csv` into each image directory where they are not as easily accessed. 

### Motivation

When creating datasets with multiple subsets I can't get the images to display alongside their associated metadata (it's usually one or the other that will show up). Since this requires a file specifically named `metadata.csv`, I then have to place that file within the image directory, which makes it much more difficult to access. Additionally, it still doesn't necessarily display the images alongside their metadata correctly (see, for instance, [this discussion](https://huggingface.co/datasets/imageomics/2018-NEON-beetles/discussions/8)).

It was suggested I bring this discussion to GitHub on another dataset struggling with a similar issue ([discussion](https://huggingface.co/datasets/imageomics/fish-vista/discussions/4)). In that case, it's a mix of data subsets, where some just reference the image URLs, while others actually have the images uploaded. The ones with images uploaded are not displaying images, but renaming that file to just `metadata.csv` would diminish the clarity of the construction of the dataset itself (and I'm not entirely convinced it would solve the issue).

### Your contribution

I can make a suggestion for one approach to address the issue:

For instance, even if it could just end in `_metadata.csv` or `-metadata.csv`, that would be very helpful to allow for more flexibility of dataset structure without impacting clarity. I would think that the functionality on the backend looking for `metadata.csv` could reasonably be adapted to look for such an ending on a filename (maybe also check that it has a `file_name` column?).

Presumably, requiring the `configs` in a setup like on [this dataset](https://huggingface.co/datasets/imageomics/rare-species/blob/main/README.md) could also help in figuring out how it should work?
```
configs:
  - config_name: <image subset>
    data_files:
      - <image-metadata>.csv
      - <path/to/images>/*.jpg
```

I'd also be happy to look at whatever solution is decided upon and contribute to the ideation.

Thanks for your time and consideration! The dataset viewer really is fabulous when it works :)
don't mention the script if trust_remote_code=False

See https://huggingface.co/datasets/Omega02gdfdd/bioclip-demo-zero-shot-mistakes for example. The error is:

```
FileNotFoundError: Couldn't find a dataset script at /src/services/worker/Omega02gdfdd/bioclip-demo-zero-shot-mistakes/bioclip-demo-zero-shot-mistakes.py or any data file in the same directory. Couldn't find 'Omega02gdfdd/bioclip-demo-zero-shot-mistakes' on the Hugging Face Hub either: FileNotFoundError: Unable to find 'hf://datasets/Omega02gdfdd/bioclip-demo-zero-shot-mistakes@12b0313ba4c3189ee5a24cb76200959e9bf7492e/data.csv' with any supported extension ['.csv', '.tsv', '.json', '.jsonl', '.parquet', '.geoparquet', '.gpq', '.arrow', '.txt', '.tar', '.blp', '.bmp', '.dib', '.bufr', '.cur', '.pcx', '.dcx', '.dds', '.ps', '.eps', '.fit', '.fits', '.fli', '.flc', '.ftc', '.ftu', '.gbr', '.gif', '.grib', '.h5', '.hdf', '.png', '.apng', '.jp2', '.j2k', '.jpc', '.jpf', '.jpx', '.j2c', '.icns', '.ico', '.im', '.iim', '.tif', '.tiff', '.jfif', '.jpe', '.jpg', '.jpeg', '.mpg', '.mpeg', '.msp', '.pcd', '.pxr', '.pbm', '.pgm', '.ppm', '.pnm', '.psd', '.bw', '.rgb', '.rgba', '.sgi', '.ras', '.tga', '.icb', '.vda', '.vst', '.webp', '.wmf', '.emf', '.xbm', '.xpm', '.BLP', '.BMP', '.DIB', '.BUFR', '.CUR', '.PCX', '.DCX', '.DDS', '.PS', '.EPS', '.FIT', '.FITS', '.FLI', '.FLC', '.FTC', '.FTU', '.GBR', '.GIF', '.GRIB', '.H5', '.HDF', '.PNG', '.APNG', '.JP2', '.J2K', '.JPC', '.JPF', '.JPX', '.J2C', '.ICNS', '.ICO', '.IM', '.IIM', '.TIF', '.TIFF', '.JFIF', '.JPE', '.JPG', '.JPEG', '.MPG', '.MPEG', '.MSP', '.PCD', '.PXR', '.PBM', '.PGM', '.PPM', '.PNM', '.PSD', '.BW', '.RGB', '.RGBA', '.SGI', '.RAS', '.TGA', '.ICB', '.VDA', '.VST', '.WEBP', '.WMF', '.EMF', '.XBM', '.XPM', '.aiff', '.au', '.avr', '.caf', '.flac', '.htk', '.svx', '.mat4', '.mat5', '.mpc2k', '.ogg', '.paf', '.pvf', '.raw', '.rf64', '.sd2', '.sds', '.ircam', '.voc', '.w64', '.wav', '.nist', '.wavex', '.wve', '.xi', '.mp3', '.opus', '.AIFF', '.AU', '.AVR', '.CAF', '.FLAC', '.HTK', '.SVX', '.MAT4', '.MAT5', '.MPC2K', '.OGG', '.PAF', '.PVF', '.RAW', '.RF64', '.SD2', '.SDS', '.IRCAM', '.VOC', '.W64', '.WAV', '.NIST', '.WAVEX', '.WVE', '.XI', '.MP3', '.OPUS', '.zip']
```

The issue there is that a `configs` parameter is set in the README, while the mentioned data file (`data.csv`) does not exist.
Audio dataset load everything in RAM and is very slow

Hello, I'm working with an audio dataset. I want to transcribe the audio that the dataset contain, and for that I use whisper. My issue is that the dataset load everything in the RAM when I map the dataset, obviously, when RAM usage is too high, the program crashes.

To fix this issue, I'm using writer_batch_size that I set to 10, but in this case, the mapping of the dataset is extremely slow.
To illustrate this, on 50 examples, with `writer_batch_size` set to 10, it takes 123.24 seconds to process the dataset, but without `writer_batch_size` set to 10, it takes about ten seconds to process the dataset, but then the process remains blocked (I assume that it is writing the dataset and therefore suffers from the same problem as `writer_batch_size`)

### Steps to reproduce the bug

Hug ram usage but fast (but actually slow when saving the dataset):
```py
from datasets import load_dataset
import time

ds = load_dataset("WaveGenAI/audios2", split="train[:50]")


# map the dataset
def transcribe_audio(row):
    audio = row["audio"]  # get the audio but do nothing with it
    row["transcribed"] = True
    return row


time1 = time.time()
ds = ds.map(
    transcribe_audio
) 

for row in ds:
    pass  # do nothing, just iterate to trigger the map function

print(f"Time taken: {time.time() - time1:.2f} seconds")
```

Low ram usage but very very slow:

```py
from datasets import load_dataset
import time

ds = load_dataset("WaveGenAI/audios2", split="train[:50]")


# map the dataset
def transcribe_audio(row):
    audio = row["audio"]  # get the audio but do nothing with it
    row["transcribed"] = True
    return row


time1 = time.time()
ds = ds.map(
    transcribe_audio, writer_batch_size=10
)  # set low writer_batch_size to avoid memory issues

for row in ds:
    pass  # do nothing, just iterate to trigger the map function

print(f"Time taken: {time.time() - time1:.2f} seconds")
```

### Expected behavior

I think the processing should be much faster, on only 50 audio examples, the mapping takes several minutes while nothing is done (just loading the audio).

### Environment info

- `datasets` version: 2.21.0
- Platform: Linux-6.10.5-arch1-1-x86_64-with-glibc2.40
- Python version: 3.10.4
- `huggingface_hub` version: 0.24.5
- PyArrow version: 17.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2024.6.1

# Extra

The dataset has been generated by using audio folder, so I don't think anything specific in my code is causing this problem.
```py
import argparse

from datasets import load_dataset

parser = argparse.ArgumentParser()
parser.add_argument("--folder", help="folder path", default="/media/works/test/")
args = parser.parse_args()

dataset = load_dataset("audiofolder", data_dir=args.folder)

# push the dataset to hub
dataset.push_to_hub("WaveGenAI/audios")
```

Also, it's the combination of `audio = row["audio"]` and `row["transcribed"] = True` which causes problems, `row["transcribed"] = True `alone does nothing and `audio = row["audio"]` alone sometimes causes problems, sometimes not.
datasets cannot handle nested json if features is given.

### Describe the bug

I have a json named temp.json.
```json
{"ref1": "ABC", "ref2": "DEF", "cuts":[{"cut1": 3, "cut2": 5}]}
```
I want to load it.
```python
ds = datasets.load_dataset('json', data_files="./temp.json", features=datasets.Features({
    'ref1': datasets.Value('string'),
    'ref2': datasets.Value('string'),
    'cuts': datasets.Sequence({
        "cut1": datasets.Value("uint16"),
        "cut2": datasets.Value("uint16")
    })
}))
```
The above code does not work. However, I can load it without giving features.
```python
ds = datasets.load_dataset('json', data_files="./temp.json")
```
Is it possible to load integers as uint16 to save some memory?

### Steps to reproduce the bug

As in the bug description.

### Expected behavior

The data are loaded and integers are uint16.

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.21.0
- Platform: Linux-5.15.0-118-generic-x86_64-with-glibc2.35
- Python version: 3.11.9
- `huggingface_hub` version: 0.24.5
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0