sjwsjw/github-issues huggingface dataset:
GitHub issue: .add_faiss_index and .add_elasticsearch_index returns ImportError at Google Colab
State: open

### Describe the bug

At Google Colab
```!pip install faiss-cpu``` works
```import faiss``` no error
but
```embeddings_dataset.add_faiss_index(column='embeddings')```
returns
```
[/usr/local/lib/python3.11/dist-packages/datasets/search.py](https://localhost:8080/#) in init(self, device, string_factory, metric_type, custom_index)
247 self.faiss_index = custom_index
248 if not _has_faiss:
--> 249 raise ImportError(
250 "You must install Faiss to use FaissIndex. To do so you can run conda install -c pytorch faiss-cpu or conda install -c pytorch faiss-gpu. "
251 "A community supported package is also available on pypi: pip install faiss-cpu or pip install faiss-gpu. "
```

because
```_has_faiss = importlib.util.find_spec("faiss") is not None```    at the beginning of ```datasets/search.py``` returns ```False```
when
the same code at colab notebook returns
```ModuleSpec(name='faiss', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7b7851449f50>, origin='/usr/local/lib/python3.11/dist-packages/faiss/init.py', submodule_search_locations=['/usr/local/lib/python3.11/dist-packages/faiss'])```

But
```
import datasets
datasets.search._has_faiss
```
at ```colab notebook``` also returns ```False```

The same story with   ```_has_elasticsearch```

### Steps to reproduce the bug

1. Follow https://huggingface.co/learn/nlp-course/chapter5/6?fw=pt at Google Colab
2. till ```embeddings_dataset.add_faiss_index(column='embeddings')```
3. ```embeddings_dataset.add_elasticsearch_index(column='embeddings')```
4. https://colab.research.google.com/drive/1h2cjuiClblqzbNQgrcoLYOC8zBqTLLcv#scrollTo=3ddzRp72auOF

### Expected behavior

I've only started Tutorial and don't know exactly. But something tells me that ```embeddings_dataset.add_faiss_index(column='embeddings')```
should work without ```Import Error```

### Environment info

Google Colab notebook with default config
GitHub issue: Problems with local dataset after upgrade from 3.3.2 to 3.4.0
State: open

### Describe the bug

I was not able to open a local saved dataset anymore that was created using an older datasets version after the upgrade yesterday from datasets 3.3.2 to 3.4.0

The traceback is 

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/arrow/arrow.py", line 67, in _generate_tables
    batches = pa.ipc.open_stream(f)
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 190, in open_stream
    return RecordBatchStreamReader(source, options=options,
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 52, in __init__
    self._open(source, options=options, memory_pool=memory_pool)
  File "pyarrow/ipc.pxi", line 1006, in pyarrow.lib._RecordBatchStreamReader._open
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Expected to read 538970747 metadata bytes, but only read 2126

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1855, in _prepare_split_single
    for _, table in generator:
  File "/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/arrow/arrow.py", line 69, in _generate_tables
    reader = pa.ipc.open_file(f)
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 234, in open_file
    return RecordBatchFileReader(
  File "/usr/local/lib/python3.10/dist-packages/pyarrow/ipc.py", line 110, in __init__
    self._open(source, footer_offset=footer_offset,
  File "pyarrow/ipc.pxi", line 1090, in pyarrow.lib._RecordBatchFileReader._open
  File "pyarrow/error.pxi", line 155, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Not an Arrow file
```

### Steps to reproduce the bug

Load a dataset from a local folder with 

```
dataset = load_dataset(
                args.train_data_dir,
                cache_dir=args.cache_dir,
            )
```
as it is done for example in the training script for SD3 controlnet.

This is the minimal script to test it: 

```
from datasets import load_dataset

def main():
    dataset = load_dataset(
        "local_dataset",  
    )
    print(dataset)
    print("Sample data:", dataset["train"][0])

if __name__ == "__main__":
    main()
````

### Expected behavior

Work in 3.4.0 like in 3.3.2

### Environment info

- `datasets` version: 3.4.0
- Platform: Linux-5.15.0-75-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.29.3
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
GitHub issue: Add IterableDataset.decode with multithreading
State: closed

Useful for dataset streaming for multimodal datasets, and especially for lerobot.
It speeds up streaming up to 20 times.

When decoding is enabled (default), media types are decoded:

* audio -> dict of "array" and "sampling_rate" and "path"
* image -> PIL.Image
* video -> torchvision.io.VideoReader

You can enable multithreading using `num_threads`. This is especially useful to speed up remote
data streaming. However it can be slower than `num_threads=0` for local data on fast disks.

PS: Disabling decoding is useful if you want to iterate on the paths or bytes of the media files
without actually decoding their content.

Example: Speed up streaming with multithreading:

```py
>>> import os
>>> from datasets import load_dataset
>>> from tqdm import tqdm
>>> ds = load_dataset("sshh12/planet-textures", split="train", streaming=True)
>>> num_threads = min(32, (os.cpu_count() or 1) + 4)
>>> ds = ds.decode(num_threads=num_threads)
>>> for _ in tqdm(ds):  # 20 times faster !
...     ...
```

why not multiprocessing ? decoding is done with the GIL released in soundfile/PIL/torchvision so multiprocessing would just use more memory

TODO
- [x] test
- [x] add to docs
GitHub issue: Epochs shortened after resuming mid-epoch with Iterable dataset+StatefulDataloader(persistent_workers=True)
State: closed

### Describe the bug

When `torchdata.stateful_dataloader.StatefulDataloader(persistent_workers=True)` the epochs after resuming only iterate through the examples that were left in the epoch when the training was interrupted. For example, in the script below training is interrupted on step 124 (epoch 1) when 3 batches are left. Then after resuming, the rest of epochs (2 and 3) only iterate through these 3 batches.

### Steps to reproduce the bug

Run the following script with and with PERSISTENT_WORKERS=true.

```python
# !/usr/bin/env python3
# torch==2.5.1
# datasets==3.3.2
# torchdata>=0.9.0
import datasets
import pprint
from torchdata.stateful_dataloader import StatefulDataLoader

import os

PERSISTENT_WORKERS = (
    os.environ.get("PERSISTENT_WORKERS", "False").lower() == "true"
)

# PERSISTENT_WORKERS = True  # Incorrect resume


# ds = datasets.load_from_disk("dataset").to_iterable_dataset(num_shards=4)
def generator():
    for i in range(128):
        yield {"x": i}


ds = datasets.Dataset.from_generator(
    generator, features=datasets.Features({"x": datasets.Value("int32")})
).to_iterable_dataset(num_shards=4)

dl = StatefulDataLoader(
    ds, batch_size=2, num_workers=2, persistent_workers=PERSISTENT_WORKERS
)
global_step = 0
epoch = 0
ds_state_dict = None
state_dict = None
resumed = False
while True:
    if epoch >= 3:
        break
    if state_dict is not None:
        dl.load_state_dict(state_dict)
        state_dict = None
        ds_state_dict = None
        resumed = True
        print("resumed")
    for i, batch in enumerate(dl):
        print(f"epoch: {epoch}, global_step: {global_step}, batch: {batch}")
        global_step += 1  # consume datapoint
        # simulate error
        if global_step == 124 and not resumed:
            ds_state_dict = ds.state_dict()
            state_dict = dl.state_dict()
            print("checkpoint")
            print("ds_state_dict")
            pprint.pprint(ds_state_dict)
            print("dl_state_dict")
            pprint.pprint(state_dict)
            break

    if state_dict is None:
        ds.set_epoch(epoch)
        epoch += 1
```

The script checkpoints when there are three batches left in the second epoch. After resuming, only the last three batches are repeated in the rest of the epochs.

If it helps, following are the two state_dicts for the dataloader save at the same step with the two settings. The left one is for `PERSISTENT_WORKERS=False`

![Image](https://github.com/user-attachments/assets/c97d6502-d7bd-4ef4-ae2d-66fe1a9732b1)

### Expected behavior

All the elements in the dataset should be iterated through in the epochs following the one where we resumed. The expected behavior can be seen by setting `PERSISTENT_WORKERS=False`. 

### Environment info

torch==2.5.1
datasets==3.3.2
torchdata>=0.9.0
GitHub issue: pyarrow.lib.ArrowTypeError: Expected dict key of type str or bytes, got 'int'
State: open

### Describe the bug

A dict with its keys are all str but get following error
```python
test_data=[{'input_ids':[1,2,3],'labels':[[Counter({2:1})]]}]
dataset = datasets.Dataset.from_list(test_data)
```
```bash
pyarrow.lib.ArrowTypeError: Expected dict key of type str or bytes, got 'int'
```

### Steps to reproduce the bug

.

### Expected behavior

.

### Environment info

datasets 3.3.2
GitHub issue: Fix small bugs with async map
State: closed

helpful for the next PR to enable parallel image/audio/video decoding and make multimodal datasets go brr (e.g. for lerobot)

- fix with_indices
- fix resuming with save_state_dict() / load_state_dict() - omg that wasn't easy
- remove unnecessary decoding in map() to enable parallelism in FormattedExampleIterable later

small bonus: keeping features in batch()
GitHub issue: index error when num_shards > len(dataset)
State: open

In `ds.push_to_hub()` and `ds.save_to_disk()`, `num_shards` must be smaller than or equal to the number of rows in the dataset, but currently this is not checked anywhere inside these functions. Attempting to invoke these functions with `num_shards > len(dataset)` should raise an informative `ValueError`.

I frequently work with datasets with a small number of rows where each row is pretty large, so I often encounter this issue, where the function runs until the shard index in `ds.shard(num_shards, indx)` goes out of bounds. Ideally, a `ValueError` should be raised before reaching this point (i.e. as soon as `ds.push_to_hub()` or `ds.save_to_disk()` is invoked with `num_shards > len(dataset)`).

It seems that adding something like:
```python
if len(self) < num_shards:
   raise ValueError(f"num_shards ({num_shards}) must be smaller than or equal to the number of rows in the dataset ({len(self)}). Please either reduce num_shards or increase max_shard_size to make sure num_shards <= len(dataset).")
```
to the beginning of the definition of the `ds.shard()` function [here](https://github.com/huggingface/datasets/blob/f693f4e93aabafa878470c80fd42ddb10ec550d6/src/datasets/arrow_dataset.py#L4728) would deal with this issue for both `ds.push_to_hub()` and `ds.save_to_disk()`, but I'm not exactly sure if this is the best place to raise the `ValueError` (it seems that a more correct way to do it would be to write separate checks for `ds.push_to_hub()` and `ds.save_to_disk()`). I'd be happy to submit a PR if you think something along these lines would be acceptable.
GitHub issue: Flexible Loader
State: open

### Feature request

Can we have a utility function that will use `load_from_disk` when given the local path and `load_dataset` if given an HF dataset?

It can be something as simple as this one:

```
def load_hf_dataset(path_or_name):
    if os.path.exists(path_or_name):
        return load_from_disk(path_or_name)
    else:
        return load_dataset(path_or_name)
```

### Motivation

This can be done inside the user codebase, too, but in my experience, it becomes repetitive code.

### Your contribution

I can open a pull request.
GitHub issue: Allow dataset row indexing with np.int types (#7423)
State: open

@lhoestq 
Proposed fix for #7423. Added a couple simple tests as requested. I had some test failures related to Java and pyspark even when installing with dev but these don't seem to be related to the changes here and fail for me even on clean main.

The typeerror raised when using the wrong type is: "Wrong key type: '{key}' of type '{type(key)}'. Expected one of int, slice, range, str or Iterable." I think that is fine. But I could modify the int part to something more generic (although I'm not sure what) if wanted.
GitHub issue: Refactor `string_to_dict` to return `None` if there is no match instead of raising `ValueError`
State: closed

Making this change, as encouraged here:

* https://github.com/huggingface/datasets/pull/7434#discussion_r1979933054

instead of having the pattern of using `try`-`except` to handle when there is no match, we can instead check if the return value is `None`; we can also assert that the return value should not be `None` if we know that should be true
GitHub issue: Refactor `Dataset.map` to reuse cache files mapped with different `num_proc`
State: open

Fixes #7433

This refactor unifies `num_proc is None or num_proc == 1` and `num_proc > 1`; instead of handling them completely separately where one uses a list of kwargs and shards and the other just uses a single set of kwargs and `self`, by wrapping the `num_proc == 1` case in a list and making the difference just whether or not you use a pool, you set up either case to be able to load each other's cache files just by changing `num_shards`; `num_proc == 1` can sequentially load the shards of a dataset mapped `num_shards > 1` and map any missing shards

Other than the structural refactor, the main contribution of this PR is `existing_cache_file_map`, which uses a regex of `cache_file_name` and `suffix_template` to find existing cache files, grouped by their `num_shards`; using this data structure, we can reset `num_shards` to an existing set of cache files, and load them accordingly
GitHub issue: `Dataset.map` ignores existing caches and remaps when ran with different `num_proc`
State: open

### Describe the bug

If you `map` a dataset and save it to a specific `cache_file_name` with a specific `num_proc`, and then call map again with that same existing `cache_file_name` but a different `num_proc`, the dataset will be re-mapped.

### Steps to reproduce the bug

1. Download a dataset
```python
import datasets

dataset = datasets.load_dataset("ylecun/mnist")
```

```
Generating train split: 100%|██████████| 60000/60000 [00:00<00:00, 116429.85 examples/s]
Generating test split: 100%|██████████| 10000/10000 [00:00<00:00, 103310.27 examples/s]
```

2. `map` and cache it with a specific `num_proc`

```python
cache_file_name="./cache/train.map"
dataset["train"].map(lambda x: x, cache_file_name=cache_file_name, num_proc=2)
```

```
Map (num_proc=2): 100%|██████████| 60000/60000 [00:01<00:00, 53764.03 examples/s]
```

3. `map` it with a different `num_proc` and the same `cache_file_name` as before

```python
dataset["train"].map(lambda x: x, cache_file_name=cache_file_name, num_proc=3)
```

```
Map (num_proc=3): 100%|██████████| 60000/60000 [00:00<00:00, 65377.12 examples/s]
```

### Expected behavior

If I specify an existing `cache_file_name`, I don't expect using a different `num_proc` than the one that was used to generate it to cause the dataset to have be be re-mapped.

### Environment info

```console
$ datasets-cli env

- `datasets` version: 3.3.2
- Platform: Linux-5.15.0-131-generic-x86_64-with-glibc2.35
- Python version: 3.10.16
- `huggingface_hub` version: 0.29.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
```
GitHub issue: Issues with large Datasets
State: open

### Describe the bug

If the coco annotation file is too large the dataset will not be able to load it, not entirely sure were the issue is but I am guessing it is due to the code trying to load it all as one line into a dataframe. This was for object detections.

My current work around is the following code but would be nice to be able to do it without worrying about it also probably there is a better way of doing it:
`
            dataset_dict = json.load(open("./local_data/annotations/train.json"))
            df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
            df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'], 'categories': dataset_dict['categories']}, ignore_index=True)
            train=Dataset.from_pandas(df)

            dataset_dict = json.load(open("./local_data/annotations/validation.json"))
            df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
            df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'],
                             'categories': dataset_dict['categories']}, ignore_index=True)
            val = Dataset.from_pandas(df)
            dataset_dict = json.load(open("./local_data/annotations/test.json"))
            df = pd.DataFrame(columns=['images', 'annotations', 'categories'])
            df = df._append({'images': dataset_dict['images'], 'annotations': dataset_dict['annotations'],
                             'categories': dataset_dict['categories']}, ignore_index=True)
            test = Dataset.from_pandas(df)
            dataset = DatasetDict({'train': train, 'validation': val, 'test': test})
`

### Steps to reproduce the bug

1) step up directory in and have the json files in coco format

-local_data
|-images
|---1.jpg
|---2.jpg
|---....
|---n.jpg
|-annotations
|---test.json
|---train.json
|---validation.json

2) try to load local_data into a dataset if the file is larger than about 300kb it will cause an error.

### Expected behavior

That it loads the jsons preferably in the same format as it has done with a smaller size.

### Environment info

- `datasets` version: 3.3.3.dev0
- Platform: Linux-6.11.0-17-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.29.0
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
GitHub issue: Error in code "Time to slice and dice" from course "NLP Course"
State: closed

### Describe the bug

When we execute code 
```
frequencies = (
    train_df["condition"]
    .value_counts()
    .to_frame()
    .reset_index()
    .rename(columns={"index": "condition", "condition": "frequency"})
)
frequencies.head()
```
answer should be like this

condition      | frequency
birth control | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

but he is different

frequency      | count
birth control  | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

this is not correct, correct code
```
frequencies = (
    train_df["condition"]
    .value_counts()
    .to_frame()
    .reset_index()
    .rename(columns={"index": "condition", "count": "frequency"})
)
````

### Steps to reproduce the bug

```
frequencies = (
train_df["condition"]
.value_counts()
.to_frame()
.reset_index()
.rename(columns={"index": "condition", "condition": "frequency"})
)
frequencies.head()
```

### Expected behavior


condition      | frequency
birth control | 27655
depression    | 8023
acne              | 5209
anxiety          | 4991
pain               | 4744

### Environment info

Google Colab
GitHub issue: load_dataset("livecodebench/code_generation_lite", version_tag="release_v2")  TypeError: 'NoneType' object is not callable
State: open

### Describe the bug

from datasets import load_dataset
lcb_codegen = load_dataset("livecodebench/code_generation_lite", version_tag="release_v2")

or
configs = get_dataset_config_names("livecodebench/code_generation_lite", trust_remote_code=True)

both error:
Traceback (most recent call last):
File "", line 1, in
File "/workspace/miniconda/envs/grpo/lib/python3.10/site-packages/datasets/load.py", line 2131, in load_dataset
builder_instance = load_dataset_builder(
File "/workspace/miniconda/envs/grpo/lib/python3.10/site-packages/datasets/load.py", line 1888, in load_dataset_builder
builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable

### Steps to reproduce the bug

from datasets import get_dataset_config_names

configs = get_dataset_config_names("livecodebench/code_generation_lite", trust_remote_code=True)
OR 
lcb_codegen = load_dataset("livecodebench/code_generation_lite", version_tag="release_v2")

### Expected behavior

load  datasets  livecodebench/code_generation_lite


### Environment info

 import  datasets
version '3.3.2'
GitHub issue: Row indexing a dataset with numpy integers
State: open

### Feature request

Allow indexing datasets with a scalar numpy integer type.

### Motivation

Indexing a dataset with a scalar numpy.int* object raises a TypeError. This is due to the test in `datasets/formatting/formatting.py:key_to_query_type`

``` python
def key_to_query_type(key: Union[int, slice, range, str, Iterable]) -> str:
    if isinstance(key, int):
        return "row"
    elif isinstance(key, str):
        return "column"
    elif isinstance(key, (slice, range, Iterable)):
        return "batch"
    _raise_bad_key_type(key)
```

In the row case, it checks if key is an int, which returns false when key is integer like but not a builtin python integer type. This is counterintuitive because a numpy array of np.int64s can be used for the batch case.

For example:

``` python
import numpy as np

import datasets

dataset = datasets.Dataset.from_dict({"a": [1, 2, 3, 4], "b": [5, 6, 7, 8]})

# Regular indexing
dataset[0]
dataset[:2]

# Indexing with numpy data types (expect same results)
idx = np.asarray([0, 1])
dataset[idx]  # Succeeds when using an array of np.int64 values
dataset[idx[0]]  # Fails with TypeError when using scalar np.int64
```

For the user, this can be solved by wrapping `idx[0]` in `int` but the test could also be changed in `key_to_query_type` to accept a less strict definition of int.

``` diff
+import numbers
+
def key_to_query_type(key: Union[int, slice, range, str, Iterable]) -> str:
+   if isinstance(key, numbers.Integral):
-   if isinstance(key, int):
        return "row"
    elif isinstance(key, str):
        return "column"
    elif isinstance(key, (slice, range, Iterable)):
        return "batch"
    _raise_bad_key_type(key)
```

Looking at how others do it, pandas has an `is_integer` definition that it checks which uses `is_integer_object` defined in `pandas/_libs/utils.pxd`:

``` cython
cdef inline bint is_integer_object(object obj) noexcept:
    """
    Cython equivalent of

    `isinstance(val, (int, np.integer)) and not isinstance(val, (bool, np.timedelta64))`

    Parameters
    ----------
    val : object

    Returns
    -------
    is_integer : bool

    Notes
    -----
    This counts np.timedelta64 objects as integers.
    """
    return (not PyBool_Check(obj) and isinstance(obj, (int, cnp.integer))
            and not is_timedelta64_object(obj))
```

This would be less flexible as it explicitly checks for numpy integer, but worth noting that they had the need to ensure the key is not a bool.

### Your contribution

I can submit a pull request with the above changes after checking that indexing succeeds with the numpy integer type. Or if there is a different integer check that would be preferred I could add that.

If there is a reason not to want this behavior that is fine too.
GitHub issue: Import order crashes script execution
State: open

### Describe the bug

Hello,

I'm trying to convert an HF dataset into a TFRecord so I'm importing `tensorflow` and `datasets` to do so.
Depending in what order I'm importing those librairies, my code hangs forever and is unkillable (CTRL+C doesn't work, I need to kill my shell entirely).

Thank you for your help
🙏 

### Steps to reproduce the bug

If you run the following script, this will hang forever : 
```python
import tensorflow as tf
import datasets

dataset = datasets.load_dataset("imagenet-1k", split="validation", streaming=True)
print(next(iter(dataset)))
```

however running the following will work fine (I just changed the order of the imports) :
```python
import datasets
import tensorflow as tf

dataset = datasets.load_dataset("imagenet-1k", split="validation", streaming=True)
print(next(iter(dataset)))
```

### Expected behavior

I'm expecting the script to reach the end and my case print the content of the first item in the dataset 
```
{'image': <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=408x500 at 0x70C646A03110>, 'label': 91}
```

### Environment info

```
$ datasets-cli env
- `datasets` version: 3.3.2
- Platform: Linux-6.8.0-1017-aws-x86_64-with-glibc2.35
- Python version: 3.11.7
- `huggingface_hub` version: 0.29.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
```

I'm also using `tensorflow==2.18.0`.
GitHub issue: pyarrow.lib.arrowinvalid: cannot mix list and non-list, non-null values with map function
State: open

### Describe the bug

Encounter pyarrow.lib.arrowinvalid error with map function in some example when loading the dataset

### Steps to reproduce the bug

```
from datasets import load_dataset
from PIL import Image, PngImagePlugin

dataset = load_dataset("leonardPKU/GEOQA_R1V_Train_8K")
system_prompt="You are a helpful AI Assistant"
def make_conversation(example):
    prompt = []

    prompt.append({"role": "system", "content": system_prompt})
    prompt.append(
            {
                "role": "user", 
                "content": [
                        {"type": "image"},
                        {"type": "text", "text": example["problem"]},
                        ]
            }
        )
    return {"prompt": prompt}

def check_data_types(example):
    for key, value in example.items():
        if key == 'image':
            if not isinstance(value, PngImagePlugin.PngImageFile):
                print(value)
        if key == "problem" or key == "solution":
            if not isinstance(value, str):
                print(value)

    return example

dataset = dataset.map(check_data_types)
dataset = dataset.map(make_conversation)
```

### Expected behavior

Successfully process the dataset with map

### Environment info

datasets==3.3.1
GitHub issue: Shard Dataset at specific indices
State: open

I have a dataset of sequences, where each example in the sequence is a separate row in the dataset (similar to LeRobotDataset). When running `Dataset.save_to_disk` how can I provide indices where it's possible to shard the dataset such that no episode spans more than 1 shard. Consequently, when I run `Dataset.load_from_disk`, how can I load just a subset of the shards to save memory and time on different ranks?

I guess an alternative to this would be, given a loaded `Dataset`, how can I run `Dataset.shard` such that sharding doesn't split any episode across shards?
GitHub issue: Documentation on multiple media files of the same type with WebDataset
State: open

The [current documentation](https://huggingface.co/docs/datasets/en/video_dataset) on a creating a video dataset includes only examples with one media file and one json. It would be useful to have examples where multiple files of the same type are included. For example, in a sign language dataset, you may have a base video and a video annotation of the extracted pose. According to the WebDataset documentation, this should be able to be done with period separated filenames. For example:

```e39871fd9fd74f55.base.mp4
e39871fd9fd74f55.pose.mp4
e39871fd9fd74f55.json
f18b91585c4d3f3e.base.mp4
f18b91585c4d3f3e.pose.mp4
f18b91585c4d3f3e.json
... 
```

If you can confirm that this method of including multiple media files works with huggingface datasets and include an example in the documentation, I'd appreciate it.
GitHub issue: Index Error Invalid Ket is out of bounds for size 0 for code-search-net/code_search_net dataset
State: open

### Describe the bug

I am trying to do model pruning on sentence-transformers/all-mini-L6-v2 for the code-search-net/code_search_net dataset using INCTrainer class 
However I am getting below error

```
raise IndexError(f"Invalid Key: {key is our of bounds for size {size}")
IndexError: Invalid key: 1840208 is out of bounds for size 0
```

### Steps to reproduce the bug

Model pruning on the above dataset using the below guide 

https://huggingface.co/docs/optimum/en/intel/neural_compressor/optimization#pruning
### Expected behavior

The modsl should be successfully pruned 

### Environment info

Torch version: 2.4.1
Python version: 3.8.10
GitHub issue: Attempt to fix multiprocessing hang by closing and joining the pool before termination
State: closed

https://github.com/huggingface/datasets/issues/6393 has plagued me on and off for a very long time. I have had various workarounds (one time combining two filter calls into one filter call removed the issue, another time making rank 0 go first resolved a cache race condition, one time i think upgrading the version of something resolved it). I don't know hf datasets well enough to fully understand the root cause, but I _think_ this PR fixes it.

Evidence: I have an LLM Foundry training yaml/script (datasets version 3.2.0) that results in a hang ~1/10 times (for a baseline for this testing, it was 2/36 runs that hung). I also reran with the latest datasets version (3.3.1) and got 4/36 hung. Installing datasets from this PR, I was able to successful run the script 144 times without a hang occurring. Assuming the base probability is 1/10, this should be more than enough times to have confidence it works.

After adding some logging, I could see that the code hung during the __exit__ of the mp pool context manager, after all shards had been processed, and the tqdm context manager had exited.

My best explanation: When multiprocessing pool __exit__ is called, it calls pool.terminate, which forcefully exits all the processes (and calls code related to this that I haven't looked at closely). I'm guessing this forceful termination has a bad interaction with some multithreading/multiprocessing that hf datasets does. If we instead call pool.close and pool.join before the pool.terminate happens, perhaps whatever that bad interaction is is able to complete gracefully, and then terminate call proceeds without issue.

If this PR seems good to you, I'd be very appreciative if you were able to do a patch release including it. Thank you!

@lhoestq
GitHub issue: Lazy loading of environment variables
State: open

### Describe the bug

Loading a `.env` file after an `import datasets` call does not correctly use the environment variables. 

This is due the fact that environment variables are read at import time:
https://github.com/huggingface/datasets/blob/de062f0552a810c52077543c1169c38c1f0c53fc/src/datasets/config.py#L155C1-L155C80

### Steps to reproduce the bug

```bash
# make tmp dir
mkdir -p /tmp/debug-env
# make .env file
echo HF_HOME=/tmp/debug-env/data > /tmp/debug-env/.env
# first load dotenv, downloads to /tmp/debug-env/data
uv run --with datasets,python-dotenv python3 -c \
'import dotenv; dotenv.load_dotenv("/tmp/debug-env/.env"); import datasets; datasets.load_dataset("Anthropic/hh-rlhf")' 
# first import datasets, downloads to `~/.cache/huggingface`
uv run --with datasets,python-dotenv python3 -c \
'import datasets; import dotenv; dotenv.load_dotenv("/tmp/debug-env/.env"); datasets.load_dataset("Anthropic/hh-rlhf")' 

```

### Expected behavior

I expect that setting environment variables with something like this:

```python3
if __name__ == "__main__":
    load_dotenv()
    main()
```
works correctly. 

### Environment info

    "datasets>=3.3.0",
GitHub issue: Performance regression in `dataset.filter`
State: closed

### Describe the bug

We're filtering dataset of ~1M (small-ish) records. At some point in the code we do `dataset.filter`, before (including 3.2.0) it was taking couple of seconds, and now it takes 4 hours.

We use 16 threads/workers, and stack trace at them look as follows:
```
Traceback (most recent call last):
  File "/python/lib/python3.12/site-packages/multiprocess/process.py", line 314, in _bootstrap
    self.run()
  File "/python/lib/python3.12/site-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/python/lib/python3.12/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3511, in _map_single
    for i, batch in iter_outputs(shard_iterable):
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3461, in iter_outputs
    yield i, apply_function(example, i, offset=offset)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3390, in apply_function
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 6416, in get_indices_from_mask_function
    indices_array = indices_mapping.column(0).take(indices_array)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 1079, in pyarrow.lib.ChunkedArray.take
  File "/python/lib/python3.12/site-packages/pyarrow/compute.py", line 458, in take
    def take(data, indices, *, boundscheck=True, memory_pool=None):
```

### Steps to reproduce the bug

1. Save dataset of 1M records in arrow
2. Filter it with 16 threads
3. Watch it take too long

### Expected behavior

Filtering done fast

### Environment info

datasets 3.3.0, python 3.12
GitHub issue: Synchronize parameters for various datasets
State: open

### Describe the bug

[IterableDatasetDict](https://huggingface.co/docs/datasets/v3.2.0/en/package_reference/main_classes#datasets.IterableDatasetDict.map) map function is missing the `desc` parameter. You can see the equivalent map function for [Dataset here](https://huggingface.co/docs/datasets/v3.2.0/en/package_reference/main_classes#datasets.Dataset.map).

There might be other parameters missing - I haven't checked.

### Steps to reproduce the bug

from datasets import Dataset, IterableDataset, IterableDatasetDict

ds = IterableDatasetDict({"train": Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3), 
                          "validate": Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3)})

for d in ds["train"]:
    print(d)

ds = ds.map(lambda x: {k: v+1 for k, v in x.items()}, desc="increment")

for d in ds["train"]:
    print(d)

### Expected behavior

The description parameter should be available for all datasets (or none). 

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.11.11
- `huggingface_hub` version: 0.28.1
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.9.0
GitHub issue: Using load_dataset with data_files and split arguments yields an error
State: open

### Describe the bug

It seems the list of valid splits recorded by the package becomes incorrectly overwritten when using the `data_files` argument.

If I run
```python
from datasets import load_dataset
load_dataset("allenai/super", split="all_examples", data_files="tasks/expert.jsonl")
```
then I get the error
```
ValueError: Unknown split "all_examples". Should be one of ['train'].
```

However, if I run
```python
from datasets import load_dataset
load_dataset("allenai/super", split="train", name="Expert")
```
then I get
```
ValueError: Unknown split "train". Should be one of ['all_examples'].
```

### Steps to reproduce the bug

Run
```python
from datasets import load_dataset
load_dataset("allenai/super", split="all_examples", data_files="tasks/expert.jsonl")
```

### Expected behavior

No error.

### Environment info

Python = 3.12
datasets = 3.2.0
GitHub issue: Optimized sequence encoding for scalars
State: closed

The change in https://github.com/huggingface/datasets/pull/3197 introduced redundant list-comprehensions when `obj` is a long sequence of scalars. This becomes a noticeable overhead when loading data from an `IterableDataset` in the function `_apply_feature_types_on_example` and can be eliminated by adding a check for scalars in `encode_nested_example` proposed here. 

In the following code example
```
import time
from datasets.features import Sequence, Value
from datasets.features.features import encode_nested_example

schema = Sequence(Value("int32"))
obj = list(range(100000))

start = time.perf_counter()
result = encode_nested_example(schema, obj)
stop = time.perf_counter()

print(f"Time spent is {stop-start} sec")
```
`encode_nested_example` becomes 492x faster (from 0.0769 to 0.0002 sec), respectively 322x (from 0.00814 to 0.00003 sec) for a list of length 10000, on a GH200 system, making it unnoticeable when loading data with tokenization.

Another change is made to avoid creating arrays from scalars and afterwards re-extracting them during casting to python (`obj == obj.__array__()[()]` in that case), which avoids a regression in the array write benchmarks.
GitHub issue: push_to_hub payload too large error when using large ClassLabel feature
State: open

### Describe the bug

When using `datasets.DatasetDict.push_to_hub` an `HfHubHTTPError: 413 Client Error: Payload Too Large for url` is raised if the dataset contains a large `ClassLabel` feature. Even if the total size of the dataset is small.

### Steps to reproduce the bug

``` python
import random
import sys

import datasets

random.seed(42)


def random_str(sz):
    return "".join(chr(random.randint(ord("a"), ord("z"))) for _ in range(sz))


data = datasets.DatasetDict(
    {
        str(i): datasets.Dataset.from_dict(
            {
                "label": [list(range(3)) for _ in range(10)],
                "abstract": [random_str(10_000) for _ in range(10)],
            },
        )
        for i in range(3)
    }
)
features = data["1"].features.copy()
features["label"] = datasets.Sequence(
    datasets.ClassLabel(names=[str(i) for i in range(50_000)])
)
data = data.map(lambda examples: {}, features=features)

feat_size = sys.getsizeof(data["1"].features["label"].feature.names)
print(f"Size of ClassLabel names: {feat_size}")
# Size of ClassLabel names: 444376


data.push_to_hub("dconnell/pubtator3_test")
```

Note that this succeeds if `ClassLabel` has fewer names or if `ClassLabel` is replaced with `Value("int64")`


### Expected behavior

Should push the dataset to hub.

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 3.2.0
- Platform: Linux-5.15.0-126-generic-x86_64-with-glibc2.35
- Python version: 3.12.8
- `huggingface_hub` version: 0.28.1
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
GitHub issue: Re-add py.typed
State: open

### Feature request

The motivation for removing py.typed no longer seems to apply.  Would a solution like [this one](https://github.com/huggingface/huggingface_hub/pull/2752) work here?

### Motivation

MyPy support is broken.  As more type checkers come out, such as RedKnot, these may also be broken.  It would be good to be PEP 561 compliant as long as it's not too onerous.

### Your contribution

I can re-add py.typed, but I don't know how to make sur all of the `__all__` files are provided (although you may not need to with modern PyRight).
GitHub issue: Getting statistics about filtered examples
State: closed

@lhoestq wondering if the team has thought about this and if there are any recommendations?

Currently when processing datasets some examples are bound to get filtered out, whether it's due to bad format, or length is too long, or any other custom filters that might be getting applied. Let's just focus on the filter by length for now, since that would be something that gets applied dynamically for each training run. Say we want to show a graph in W&B with the running total of the number of filtered examples so far.

What would be a good way to go about hooking this up? Because the map/filter operations happen before the DataLoader batches are created, at training time if we're just grabbing batches from the DataLoader then we won't know how many things have been filtered already. But there's not really a good way to include a 'num_filtered' key into the dataset itself either because dataset map/filter process examples independently and don't have a way to track a running sum.

The only approach I can kind of think of is having a 'is_filtered' key in the dataset, and then creating a custom batcher/collator that reads that and tracks the metric?
GitHub issue: OSError: [Errno 22] Invalid argument forbidden character
State: closed

### Describe the bug

I'm on Windows and i'm trying to load a datasets but i'm having title error because files in the repository are named with charactere like < >which can't be in a name file. Could it be possible to load this datasets but removing those charactere ?

### Steps to reproduce the bug

load_dataset("CATMuS/medieval") on Windows

### Expected behavior

Making the function to erase the forbidden character to allow loading the datasets who have those characters. 

### Environment info

- `datasets` version: 3.2.0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.12.2
- `huggingface_hub` version: 0.28.1
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
GitHub issue: Add bookfolder Dataset Builder for Digital Book Formats
State: closed

### Feature request

This feature proposes adding a new dataset builder called bookfolder to the datasets library. This builder would allow users to easily load datasets consisting of various digital book formats, including: AZW, AZW3, CB7, CBR, CBT, CBZ, EPUB, MOBI, and PDF.

### Motivation

Currently, loading datasets of these digital book files requires manual effort. This would also lower the barrier to entry for working with these formats, enabling more diverse and interesting datasets to be used within the Hugging Face ecosystem.

### Your contribution

This feature is rather simple as it will be based on the folder-based builder, similar to imagefolder. I'm willing to contribute to this feature by submitting a PR
GitHub issue: Make IterableDataset (optionally) resumable
State: open

### What does this PR do?
This PR introduces a new `stateful` option to the `dataset.shuffle` method, which defaults to `False`. 
When enabled, this option allows for resumable shuffling of `IterableDataset` instances, albeit with some additional memory overhead.

Key points:
* All tests have passed
* Docstrings have been updated to reflect the new functionality

I'm very looking forward to receiving feedback on this implementation! @lhoestq
GitHub issue: Support async functions in map()
State: closed

e.g. to download images or call an inference API like HF or vLLM

```python
import asyncio
import random

from datasets import Dataset


async def f(x):
    await asyncio.sleep(random.random())

ds = Dataset.from_dict({"data": range(100)})
ds.map(f)
# Map: 100%|█████████████████████████████| 100/100 [00:01<00:00, 99.81 examples/s]
```

TODO
- [x] clean code (right now it's a big copy paste)
- [x] batched
- [x] Dataset.map()
- [x] IterableDataset.map()
- [x] Dataset.filter()
- [x] IterableDataset.filter()
- [x] test
- [x] docs
GitHub issue: Iterating over values of a column in the IterableDataset
State: open

### Feature request

I would like to be able to iterate (and re-iterate if needed) over a column of an `IterableDataset` instance. The following example shows the supposed API:
```
def gen():
    yield {"text": "Good", "label": 0}
    yield {"text": "Bad", "label": 1}

ds = IterableDataset.from_generator(gen)
texts = ds["text"]

for v in texts:
    print(v)  # Prints "Good" and "Bad"

for v in texts:
    print(v)  # Prints "Good" and "Bad" again
```

### Motivation

In the real world problems, huge NNs like Transformer are not always the best option, so there is a need to conduct experiments with different methods. While 🤗Datasets is perfectly adapted to 🤗Transformers, it may be inconvenient when being used with other libraries. The ability to retrieve a particular column is the case (e.g., gensim's FastText [requires](https://radimrehurek.com/gensim/models/fasttext.html#gensim.models.fasttext.FastText.train) only lists of strings, not dictionaries).
While there are ways to achieve the desired functionality, they are not good ([forum](https://discuss.huggingface.co/t/how-to-iterate-over-values-of-a-column-in-the-iterabledataset/135649)). It would be great if there was a built-in solution.

### Your contribution

Theoretically, I can submit a PR, but I have very little knowledge of the internal structure of 🤗Datasets, so some help may be needed.
Moreover, I can only work on weekends, since I have a full-time job. However, the feature does not seem to be popular, so there is no need to implement it as fast as possible.
GitHub issue: Allow pushing config version to hub
State: open

### Feature request

Currently, when datasets are created, they can be versioned by passing the `version` argument to `load_dataset(...)`. For example creating `outcomes.csv` on the command line

```
echo "id,value\n1,0\n2,0\n3,1\n4,1\n" > outcomes.csv
```
and creating it
```
import datasets
dataset = datasets.load_dataset(
    "csv",
    data_files ="outcomes.csv",
    keep_in_memory = True,
    version = '1.0.0')
```

The version info is stored in the `info` and can be accessed e.g. by `next(iter(dataset.values())).info.version`

This dataset can be uploaded to the hub with `dataset.push_to_hub(repo_id = "maomlab/example_dataset")`. This will create a dataset  on the hub with the following in the `README.md`, but it doesn't upload the version information:

```
---
dataset_info:
  features:
  - name: id
    dtype: int64
  - name: value
    dtype: int64
  splits:
  - name: train
    num_bytes: 64
    num_examples: 4
  download_size: 1332
  dataset_size: 64
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---
```

However, when I download from the hub, the version information is missing:

```
dataset_from_hub_no_version = datasets.load_dataset("maomlab/example_dataset")
next(iter(dataset.values())).info.version
```

I can add the version information manually to the hub, by appending it to the end of config section:

```
...
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  version: 1.0.0
---
```

And then when I download it, the version information is correct.





### Motivation

### Why adding version information for each config makes sense
  1. The version information is already recorded in the dataset config info data structure and is able to parse it correctly, so it makes sense to sync it with `push_to_hub`. 
  2. Keeping the version info in at the config level is different from version info at the branch level. As the former relates to the version of the specific dataset the config refers to rather than the version of the dataset curation itself.

## A explanation for the current behavior:
In [datasets/src/datasets/info.py:159](https://github.com/huggingface/datasets/blob/fb91fd3c9ea91a818681a777faf8d0c46f14c680/src/datasets/info.py#L159C1-L160C1
), the `_INCLUDED_INFO_IN_YAML` variable doesn't include `"version"`.

If my reading of the code is right, adding `"version"` to `_INCLUDED_INFO_IN_YAML`, would allow the version information to be uploaded to the hub.

### Your contribution

Request: add `"version"` to `_INCLUDE_INFO_IN_YAML` in [datasets/src/datasets/info.py:159](https://github.com/huggingface/datasets/blob/fb91fd3c9ea91a818681a777faf8d0c46f14c680/src/datasets/info.py#L159C1-L160C1
)