sjwsjw/github-issues huggingface dataset:
GitHub issue: fix: Image Feature in Datasets Library Fails to Handle bytearray Objects from Spark DataFrames (#7517)
State: open

## Task
Support bytes-like objects (bytes and bytearray) in Features classes

### Description
The `Features` classes only accept `bytes` objects for binary data, but not `bytearray`. This leads to errors when using `IterableDataset.from_spark()` with Spark DataFrames as they contain `bytearray` objects, even though both `bytes` and `bytearray` are valid [*bytes-like objects* in Python](https://docs.python.org/3/glossary.html#term-bytes-like-object).

### Changes
- Updated `Features` classes to accept both `bytes` and `bytearray` types for binary data fields.

### Reasoning
- `bytes` and `bytearray` serve the same purpose for binary data, with the only difference being mutability.
- Modifying the Spark iterator to convert `bytearray` to `bytes` would be a workaround, not a true fix. I think the correct solution is to accept all bytes-like objects as input.
- This approach is more robust and future-proof since Python 3.12+ provides a [standard way to check for buffer protocol](https://docs.python.org/3/c-api/buffer.html#bufferobjects).

### Testing
- Added tests to cover `bytearray` inputs for image features.

### Related Issues
- Fixes: #7517
GitHub issue: Update items in the dataset without `map`
State: open

### Feature request

I would like to be able to update items in my dataset without affecting all rows. At least if there was a range option, I would be able to process those items, save the dataset, and then continue.

If I am supposed to split the dataset first, that is not clear, since the docs suggest that any of those functions returns a new object, so I don't think I can do that.

### Motivation

I am applying an extremely time-consuming function to each item in my `Dataset`. Unfortunately, datasets only supports updating values via `map`, so if my computer dies in the middle of this long-running process, I lose all progress. This is far from ideal. I would like to use `datasets` throughout this processing, but this limitation is now forcing me to write my own dataset format just to do this intermediary operation.

It would be less intuitive but I suppose I could split and then concatenate the dataset before saving? But this feels very inefficient.

### Your contribution

I can test the feature.
GitHub issue: num_proc parallelization works only for first ~10s.
State: open

### Describe the bug

When I try to load an already downloaded dataset with num_proc=64, the speed is very high for the first 10-20 seconds acheiving 30-40K samples / s, and 100% utilization for all cores but it soon drops to <= 1000 with almost 0% utilization for most cores. 

### Steps to reproduce the bug

```
// download dataset with cli
!huggingface-cli download --repo-type dataset timm/imagenet-1k-wds --max-workers 32

from datasets import load_dataset
ds = load_dataset("timm/imagenet-1k-wds", num_proc=64)
```


### Expected behavior

100% core utilization throughout. 

### Environment info

Azure A100-80GB, 16 cores VM

![Image](https://github.com/user-attachments/assets/69d00fe3-d720-4474-9439-21e046d85034)
GitHub issue: Image Feature in Datasets Library Fails to Handle bytearray Objects from Spark DataFrames
State: open

### Describe the bug

When using `IterableDataset.from_spark()` with a Spark DataFrame containing image data, the `Image` feature class fails to properly process this data type, causing an `AttributeError: 'bytearray' object has no attribute 'get'`

### Steps to reproduce the bug

1. Create a Spark DataFrame with a column containing image data as bytearray objects
2. Define a Feature schema with an Image feature
3. Create an IterableDataset using `IterableDataset.from_spark()`
4. Attempt to iterate through the dataset

```
from pyspark.sql import SparkSession
from datasets import Dataset, IterableDataset, Features, Image, Value

# initialize spark
spark = SparkSession.builder.appName("MinimalRepro").getOrCreate()

# create spark dataframe
data = [(0, open("image.png", "rb").read())]
df = spark.createDataFrame(data, "idx: int, image: binary")

# convert to dataset
features = Features({"idx": Value("int64"), "image": Image()})
ds = Dataset.from_spark(df, features=features)
ds_iter = IterableDataset.from_spark(df, features=features)

# iterate
print(next(iter(ds)))
print(next(iter(ds_iter)))
```

### Expected behavior

The features should work on `IterableDataset` the same way they work on `Dataset`

### Environment info

- `datasets` version: 3.5.0
- Platform: macOS-15.3.2-arm64-arm-64bit
- Python version: 3.12.7
- `huggingface_hub` version: 0.30.2
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
GitHub issue: unsloth/DeepSeek-R1-Distill-Qwen-32B server error
State: closed

### Describe the bug

hfhubhttperror: 500 server error: internal server error for url: https://huggingface.co/api/models/unsloth/deepseek-r1-distill-qwen-32b-bnb-4bit/commits/main (request id: root=1-67fe23fa-3a2150eb444c2a823c388579;de3aed68-c397-4da5-94d4-6565efd3b919) internal error - we're working hard to fix this as soon as possible!

### Steps to reproduce the bug

unsloth/DeepSeek-R1-Distill-Qwen-32B server error

### Expected behavior

Network repair

### Environment info

The web side is also unavailable
GitHub issue: `concatenate_datasets` does not preserve Pytorch format for IterableDataset
State: open

### Describe the bug

When concatenating datasets with `concatenate_datasets`, I would expect the resulting combined dataset to be in the same format as the inputs (assuming it's consistent). This is indeed the behavior when combining `Dataset`, but not when combining `IterableDataset`. Specifically, when applying `concatenate_datasets` to a list of `IterableDataset` in Pytorch format (i.e. using `.with_format(Pytorch)`), the output `IterableDataset` is not in Pytorch format.

### Steps to reproduce the bug

```
import datasets
ds = datasets.Dataset.from_dict({"a": [1,2,3]})
iterable_ds = ds.to_iterable_dataset()
datasets.concatenate_datasets([ds.with_format("torch")]) # <- this preserves Pytorch format
datasets.concatenate_datasets([iterable_ds.with_format("torch")]) # <- this does NOT preserves Pytorch format
```

### Expected behavior

Pytorch format should be preserved when combining IterableDataset in Pytorch format.

### Environment info

datasets==3.5.0, Python 3.11.11, torch==2.2.2
GitHub issue: Do not hash `generator` in `BuilderConfig.create_config_id`
State: closed

`Dataset.from_generator` function passes all of its arguments to `BuilderConfig.create_config_id`, including generator function itself. `BuilderConfig.create_config_id` function tries to hash all the args, and hashing a `generator` can take a large amount of time or even cause MemoryError if the dataset processed in a generator function is large enough.

Maybe we should pop generator from `config_kwargs_to_add_to_suffix` before hashing to avoid it.

There is a more detailed description of the problem this PR solves in #7513
GitHub issue: .map() fails if function uses pyvista
State: open

### Describe the bug

Using PyVista inside a .map() produces a crash with `objc[78796]: +[NSResponder initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.`

### Steps to reproduce the bug

Run 

```python
import numpy as np
import pyvista as pv
import datasets

data = [{"coords": np.random.rand(5, 3)} for _ in range(3)]


def render_point(example):
  plotter = pv.Plotter(off_screen=True)
  cloud = pv.PolyData(example["coords"])
  plotter.add_mesh(cloud)
  img = plotter.screenshot(return_img=True)
  return {"image": img}


# breaks if num_proc>1
ds = datasets.Dataset.from_list(data).map(render_point, num_proc=2)

```

### Expected behavior

It should work. Just like when I use a process pool to make it work.

```python
import numpy as np
import pyvista as pv
import multiprocessing

data = [{"coords": np.random.rand(5, 3)} for _ in range(3)]


def render_point(example):
  plotter = pv.Plotter(off_screen=True)
  cloud = pv.PolyData(example["coords"])
  plotter.add_mesh(cloud)
  img = plotter.screenshot(return_img=True)
  return {"image": img}


if __name__ == "__main__":
  with multiprocessing.Pool(processes=2) as pool:
    results = pool.map(render_point, data)
  print(results[0]["image"].shape)
```

### Environment info


- `datasets` version: 3.3.2
- Platform: macOS-15.3.2-arm64-arm-64bit
- Python version: 3.11.10
- `huggingface_hub` version: 0.28.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.10.0

And then I suppose pyvista info is good to have.

```python
import pyvista as pv
print(pv.Report())
```
gives 
--------------------------------------------------------------------------------
  Date: Mon Apr 14 21:42:08 2025 CEST

                OS : Darwin (macOS 15.3.2)
            CPU(s) : 10
           Machine : arm64
      Architecture : 64bit
               RAM : 32.0 GiB
       Environment : IPython
       File system : apfs
        GPU Vendor : Apple
      GPU Renderer : Apple M1 Max
       GPU Version : 4.1 Metal - 89.3
  MathText Support : True

  Python 3.11.10 (main, Oct  7 2024, 23:25:27) [Clang 18.1.8 ]

           pyvista : 0.44.2
               vtk : 9.4.0
             numpy : 2.2.2
        matplotlib : 3.10.0
            scooby : 0.10.0
             pooch : 1.8.2
            pillow : 11.1.0
           imageio : 2.36.1
             PyQt5 : 5.15.11
           IPython : 8.30.0
             scipy : 1.14.1
              tqdm : 4.67.1
        jupyterlab : 4.3.5
      nest_asyncio : 1.6.0
--------------------------------------------------------------------------------
GitHub issue: Incompatibile dill version (0.3.9) in datasets 2.18.0 - 3.5.0
State: open

### Describe the bug

Datasets 2.18.0 - 3.5.0 has a dependency on dill < 0.3.9. This causes errors with dill >= 0.3.9.

Could you please take a look into it and make it compatible?

### Steps to reproduce the bug

1. Install setuptools >= 2.18.0
2. Install dill >=0.3.9
3. Run pip check
4. Output:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 2.18.0 requires dill<0.3.9,>=0.3.0, but you have dill 0.3.9 which is incompatible.

### Expected behavior

Pip install both libraries without any errors

### Environment info

Datasets version: 2.18 - 3.5
Python: 3.11
GitHub issue: Dataset uses excessive memory when loading files
State: open

### Describe the bug

Hi
I am having an issue when loading a dataset.
I have about 200 json files each about 1GB (total about 215GB). each row has a few features which are a list of ints.
I am trying to load the dataset using `load_dataset`.
The dataset is about 1.5M samples
I use `num_proc=32` and a node with 378GB of memory. 
About a third of the way there I get an OOM.
I also saw an old bug with a similar issue, which says to set `writer_batch_size`. I tried to lower it to 10, but it still crashed.
I also tried to lower the `num_proc` to 16 and even 8, but still the same issue.

### Steps to reproduce the bug

`dataset = load_dataset("json", data_dir=data_config.train_path, num_proc=data_config.num_proc, writer_batch_size=50)["train"]`


### Expected behavior

Loading a dataset with more than 100GB to spare should not cause an OOM error.
maybe i am missing something but I would love some help.

### Environment info

- `datasets` version: 3.5.0
- Platform: Linux-6.6.20-aufs-1-x86_64-with-glibc2.36
- Python version: 3.11.2
- `huggingface_hub` version: 0.29.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
GitHub issue: Iterating over Image feature columns is extremely slow
State: open

We are trying to load datasets where the image column stores `PIL.PngImagePlugin.PngImageFile` images. However, iterating over these datasets is extremely slow. 
What I have found:
1. It is the presence of the image column that causes the slowdown. Removing the column from the dataset results in blazingly fast (as expected) times
2. It is ~2x faster to iterate when the column contains a single image as opposed to a list of images i.e., the feature is a Sequence of Image objects. We often need multiple images per sample, so we need to work with a list of images
3. It is ~17x faster to store paths to PNG files and load them using `PIL.Image.open`, as opposed to iterating over a `Dataset` with an Image column, and ~30x faster compared to `Sequence` of `Image`s. See a simple script below with an openly available dataset.


It would be great to understand the standard practices for storing and loading multimodal datasets (image + text). 

https://huggingface.co/docs/datasets/en/image_load seems a bit underdeveloped? (e.g., `dataset.decode` only works with `IterableDataset`, but it's not clear from the doc)

Thanks!

```python
from datasets import load_dataset, load_from_disk
from PIL import Image
from pathlib import Path

ds = load_dataset("getomni-ai/ocr-benchmark")

for idx, sample in enumerate(ds["test"]):
    image = sample["image"]
    image.save(f"/tmp/ds_files/images/image_{idx}.png")

ds.save_to_disk("/tmp/ds_columns")

# Remove the 'image' column
ds["test"] = ds["test"].remove_columns(["image"])

# Create image paths for each sample
image_paths = [f"images/image_{idx}.png" for idx in range(len(ds["test"]))]

# Add the 'image_path' column to the dataset
ds["test"] = ds["test"].add_column("image_path", image_paths)

# Save the updated dataset
ds.save_to_disk("/tmp/ds_files")
files_path = Path("/tmp/ds_files")
column_path = Path("/tmp/ds_columns")

# load and benchmark
ds_file = load_from_disk(files_path)
ds_column = load_from_disk(column_path)
import time


images_files = []
start = time.time()
for idx in range(len(ds_file["test"])):
    image_path = files_path / ds_file["test"][idx]["image_path"]
    image = Image.open(image_path)
    images_files.append(image)
end = time.time()
print(f"Time taken to load images from files: {end - start} seconds")

# Time taken to load images from files: 1.2364635467529297 seconds


images_column = []
start = time.time()
for idx in range(len(ds_column["test"])):
    images_column.append(ds_column["test"][idx]["image"])
end = time.time()
print(f"Time taken to load images from columns: {end - start} seconds")

# Time taken to load images from columns: 20.49347186088562 seconds
```
GitHub issue: Front-end statistical data quantity deviation
State: open

### Describe the bug

While browsing the dataset at https://huggingface.co/datasets/NeuML/wikipedia-20250123, I noticed that a dataset with nearly 7M entries was estimated to be only 4M in size—almost half the actual amount. According to the post-download loading and the dataset_info (https://huggingface.co/datasets/NeuML/wikipedia-20250123/blob/main/train/dataset_info.json), the true data volume is indeed close to 7M. This significant discrepancy could mislead users when sorting datasets by row count. Why not directly retrieve this information from dataset_info?

Not sure if this is the right place to report this bug, but leaving it here for the team's awareness.
GitHub issue: BuilderConfig ParquetConfig(...) doesn't have a 'use_auth_token' key.
State: open

### Describe the bug

Trying to run the following fine-tuning script (based on this page [here](https://github.com/huggingface/instruction-tuned-sd)):

```
! accelerate launch /content/instruction-tuned-sd/finetune_instruct_pix2pix.py \
    --pretrained_model_name_or_path=${MODEL_ID} \
    --dataset_name=${DATASET_NAME} \
    --use_ema \
    --enable_xformers_memory_efficient_attention \
    --resolution=512 --random_flip \
    --train_batch_size=2 --gradient_accumulation_steps=4 --gradient_checkpointing \
    --max_train_steps=500 \
    --checkpointing_steps=25 --checkpoints_total_limit=1 \
    --learning_rate=5e-05 --max_grad_norm=1 --lr_warmup_steps=20 \
    --conditioning_dropout_prob=0.1 \
    --mixed_precision=fp16 \
    --seed=42 \
    --output_dir=${OUTPUT_DIR} \
    --original_image_column=before \
    --edit_prompt=prompt \
    --edited_image=after
```

but I keep getting the following error:

```
Traceback (most recent call last):
  File "/content/instruction-tuned-sd/finetune_instruct_pix2pix.py", line 1137, in <module>
    main()
  File "/content/instruction-tuned-sd/finetune_instruct_pix2pix.py", line 652, in main
    dataset = load_dataset(
              ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/load.py", line 2129, in load_dataset
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/load.py", line 1886, in load_dataset_builder
    builder_instance: DatasetBuilder = builder_cls(
                                       ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/builder.py", line 342, in __init__
    self.config, self.config_id = self._create_builder_config(
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/datasets/builder.py", line 590, in _create_builder_config
    raise ValueError(f"BuilderConfig {builder_config} doesn't have a '{key}' key.")
ValueError: BuilderConfig ParquetConfig(name='default', version=0.0.0, data_dir=None, data_files={'train': ['data/train-*']}, description=None, batch_size=None, columns=None, features=None, filters=None) doesn't have a 'use_auth_token' key.
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 10, in <module>
    sys.exit(main())
             ^^^^^^
```

Any ideas? `datasets` version should be `3.2.0`.

### Steps to reproduce the bug

Just running the script above.

### Expected behavior

No errors

### Environment info

Python 3.11.11

datasets==3.2.0
GitHub issue: `load_dataset` of size 40GB creates a cache of >720GB
State: closed

Hi there,

I am trying to load a dataset from the Hugging Face Hub and split it into train and validation splits. Somehow, when I try to do it with `load_dataset`, it exhausts my disk quota. So, I tried manually downloading the parquet files from the hub and loading them as follows:

```python
 ds = DatasetDict(
        {
            "train": load_dataset(
                "parquet", 
                data_dir=f"{local_dir}/{tok}", 
                cache_dir=cache_dir, 
                num_proc=min(12, os.cpu_count()),   # type: ignore
                split=ReadInstruction("train", from_=0, to=NUM_TRAIN, unit="abs"),  # type: ignore
            ),
            "validation": load_dataset(
                "parquet", 
                data_dir=f"{local_dir}/{tok}", 
                cache_dir=cache_dir, 
                num_proc=min(12, os.cpu_count()),   # type: ignore
                split=ReadInstruction("train", from_=NUM_TRAIN, unit="abs"),  # type: ignore
            )
        }
    )

```

which still strangely creates 720GB of cache. In addition, if I remove the raw parquet file folder (`f"{local_dir}/{tok}"` in this example), I am not able to load anything. So, I am left wondering what this cache is doing. Am I missing something? Is there a solution to this problem?

Thanks a lot in advance for your help!

A related issue: https://github.com/huggingface/transformers/issues/10204#issue-809007443.

---

Python: 3.11.11
datasets: 3.5.0
GitHub issue: Nested Feature raises ArrowNotImplementedError: Unsupported cast using function cast_struct
State: closed

### Describe the bug

`datasets.Features` seems to be unable to handle json file that contains fields of `list[dict]`.

### Steps to reproduce the bug

```json
// test.json
{"a": 1, "b": [{"c": 2, "d": 3}, {"c": 4, "d": 5}]}
{"a": 5, "b": [{"c": 7, "d": 8}, {"c": 9, "d": 10}]}
```

```python
import json
from datasets import Dataset, Features, Value, Sequence, load_dataset

annotation_feature = Features({
    "a": Value("int32"),
    "b": Sequence({
        "c": Value("int32"),
        "d": Value("int32"),
    }),
})
annotation_dataset = load_dataset(
    "json", 
    data_files="test.json",
    features=annotation_feature
)
```

```
ArrowNotImplementedError: Unsupported cast from list<item: struct<c: int32, d: int32>> to struct using function cast_struct

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
Cell In[46], line 11
      2 from datasets import Dataset, Features, Value, Sequence, load_dataset
      4 annotation_feature = Features({
      5     "a": Value("int32"),
      6     "b": Sequence({
   (...)      9     }),
     10 })
---> 11 annotation_dataset = load_dataset(
     12     "json", 
     13     data_files="test.json",
     14     features=annotation_feature
     15 )
```

### Expected behavior

A `datasets.Datasets` instance should be initialized.

### Environment info

- `datasets` version: 3.5.0
- Platform: Linux-6.11.0-21-generic-x86_64-with-glibc2.39
- Python version: 3.11.11
- `huggingface_hub` version: 0.30.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
GitHub issue: Make `with_format` correctly indicate that a `Dataset` is compatible with PyTorch's `Dataset` class
State: open

### Feature request

Currently `datasets` does not correctly indicate to the Python type-checker (e.g. `pyright` / `Pylance`) that the output of `with_format` is compatible with PyTorch's `Dataloader` since it does not indicate that the HuggingFace `Dataset` is compatible with the PyTorch `Dataset` class. It would be great if we could get the typing to work nicely.

### Motivation

To avoid casting types in our Python code.

### Your contribution

I would be happy to contribute a PR if this is something that may be accepted and could work with the current approach.
This doesn't have to be for just PyTorch, but I imagine that the same thing would be useful for `tensorflow` and such, but we only have a need for PyTorch at this stage.
GitHub issue: Extreme memory bandwidth.
State: open

### Describe the bug

When I use hf datasets on 4 GPU with 40 workers I get some extreme memory bandwidth of constant ~3GB/s. 

However, if I wrap the dataset in `IterableDataset`, this issue is gone and the data also loads way faster (4x faster training on 1 worker).

It seems like the workers don't share memory and basically duplicate the data 4x40.



### Steps to reproduce the bug

Trainer arguments:
```
    dataloader_pin_memory=True,
    dataloader_num_workers=40,
    dataloader_prefetch_factor=2,
    dataloader_persistent_workers=True,
```
Call trainer:
```
trainer = Trainer(
    model=model,
    args=train_args,
    train_dataset=load_from_disk('..').with_fromat('torch'),
)
```
The dataset has 600GB and consists of 1225 files.

### Expected behavior

The optimal bandwidth should be 100MB/s to keep up with GPU.

### Environment info

Linux
Python 3.11
datasets==3.2.0
GitHub issue: How to convert videos to images?
State: open

### Feature request

Does someone know how to return the images from videos?

### Motivation

I am trying to use openpi(https://github.com/Physical-Intelligence/openpi) to finetune my Lerobot dataset(V2.0 and V2.1). I find that although the codedaset is v2.0, they are different. It seems like Lerobot V2.0 has two version, one is data include images infos and another one is separate to data and videos.

Does someone know how to return the images from videos?
GitHub issue: Json builder: Allow features to override problematic Arrow types
State: open

### Feature request

In the JSON builder, use explicitly requested feature types before or while converting to Arrow.

### Motivation

Working with JSON datasets is really hard because of Arrow.  At the very least, it seems like it should be possible to work-around these problems by explicitly setting problematic columns's types.  But it seems like this is not possible because the features are only used *after* converting to arrow.

Here's a simple example where the Arrow error could potentially be avoided by converting the column to a string: https://colab.research.google.com/drive/16QHRdbUwKSrpwVfGwu8V8AHr8v2dv0dt?usp=sharing

### Your contribution

Maybe with some guidance.  I'm not very familiar with arrow or pandas.
GitHub issue: Columns in the dataset obtained though load_dataset do not correspond to the one in the dataset viewer since 3.4.0
State: open

### Describe the bug

I have noticed that on my dataset named [BrunoHays/Accueil_UBS](https://huggingface.co/datasets/BrunoHays/Accueil_UBS), since the version 3.4.0, every column except audio is missing when I load the dataset.

Interestingly, the dataset viewer still shows the correct columns

### Steps to reproduce the bug

```python
from datasets import load_dataset
ds = load_dataset("BrunoHays/Accueil_UBS", streaming=True)
print(next(iter(ds["test"])).keys())
```
With datasets >= 3.4.0: 
-> dict_keys(['audio'])
With datasets == 3.3.2:
-> dict_keys(['audio', 'id', 'speaker', 'sentence', 'raw_sentence', 'start_timestamp', 'end_timestamp', 'overlap'])


### Expected behavior

All the columns should be present

### Environment info

- `datasets` version: 3.3.2
- Platform: macOS-14.6.1-x86_64-i386-64bit
- Python version: 3.10.15
- `huggingface_hub` version: 0.30.1
- PyArrow version: 16.1.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0
GitHub issue: Broken links in pdf loading documentation
State: closed

### Describe the bug

Hi, just a couple of small issues I ran into while reading the docs for [loading pdf data](https://huggingface.co/docs/datasets/main/en/document_load):

1. The link for the [`Create a pdf dataset`](https://huggingface.co/docs/datasets/main/en/document_load#pdffolder) points to https://huggingface.co/docs/datasets/main/en/pdf_dataset instead of https://huggingface.co/docs/datasets/main/en/document_dataset and hence gives a 404 error.

2. At the top of the page, it's mentioned that to work with pdf datasets we need to have the `pdfplumber` package installed but the link to its installation guide points to `pytorch/vision` [installation instructions](https://github.com/pytorch/vision#installation) instead of `pdfplumber`'s [guide](https://github.com/jsvine/pdfplumber#installation)

I love the work on enabling pdf dataset support and these small tweaks would help everyone navigate the docs better. Thanks!

### Steps to reproduce the bug

The issue is on the [Load Document Data](https://huggingface.co/docs/datasets/main/en/document_load) page of the datasets docs.

### Expected behavior

1. For solving the first issue, I went through the [source .mdx code](https://github.com/huggingface/datasets/blob/main/docs/source/document_load.mdx?plain=1#L188) of the datasets docs and found that the link is pointing to `./pdf_dataset` instead of `./document_dataset`

2. For the second issue, I went through the [source .mdx code](https://github.com/huggingface/datasets/blob/main/docs/source/document_load.mdx?plain=1#L13) of the datasets docs and found that the link is `pytorch/vision` [installation instructions](https://github.com/pytorch/vision#installation) instead of `pdfplumber`'s [guide](https://github.com/jsvine/pdfplumber#installation)

Just replacing these two links should fix the bugs

### Environment info

datasets v3.5.0 (main at the time of writing)
GitHub issue: push_to_hub does not upload videos
State: open

### Describe the bug

Hello,

I would like to upload a video dataset (some .mp4 files and some segments within them), i.e. rows correspond to subsequences from videos. Videos might be referenced by several rows.

I created a dataset locally and it references the videos and the video readers can read them correctly. I use push_to_hub() to upload the dataset to the hub.

Expectation: A user uses `load_dataset` and can load the videos.

However, the videos seem to be just referenced via paths on the computer and not uploaded to the hub. Therefore a target user cannot load the videos in the dataset.

### Steps to reproduce the bug

1. create a video dataset with paths e.g. { ["videos"]: [path1, path2, ...]}
2. dataset.push_to_hub
3. on a different computer (or same pc if relative paths are used in a different folder):
```
dataset = load_dataset("siplab/egosim", split="train")
video = dataset[0]["video_head"]
```

3. will fail

### Expected behavior

Expectation: A user uses `load_dataset` and can load the videos.

### Environment info

datasets                       3.1.0
Python 3.8.18
GitHub issue: (refactor) remove redundant logic in _check_valid_index_key
State: open

This PR contributes a minor refactor, in a small function in `src/datasets/formatting/formatting.py`. No change in logic. 

In the original code, there are separate if-conditionals for `isinstance(key, range)` and `isinstance(key, Iterable)`, with essentially the same logic. 

This PR combines these two using a single if statement.

**Considerations**

1. Although range in python is guaranteed to have integers, internally calling `int()` on an object that is already an int is negligible. (In python it returns the original object. It doesn't create a new integer object or perform any actual conversion)

2. Technically a range is already an Iterable, and we could just do `isinstance(key, Iterable)` but I explicitly did `isinstance(key, (range, Iterable))` just to be super obvious and consistent that both cases are handled because I see `slice, range, Iterable` everywhere in this `formatting.py`

3. This PR removes the `if len(key)>0` conditional. I think it is cleaner to have it this way for three reasons.
- There was originally no else statement and the code would have failed silently anyway.  
- The if len(key)>0 should be caught much earlier, rather than in `formatting.py`. 
- There are actually multiple cases where this would fail, if len(key)>0, if key is non numeric or float, or if key is a list of lists. It's clunky to state all this and the error be thrown during max or indexing. 


**Previous PR and Issues Checks**

1. No known PR or Issues (both closed or open) in hf datasets repository 

**Tests**

1. Tested using Dataset (load_dataset("wikitext", "wikitext-103-raw-v1")), Pytorch DataLoader, with a Pytorch BatchSampler (list of indexes returned instead of single index).
GitHub issue: Implement capability to restore non-nullability in Features
State: open

This PR attempts to keep track of non_nullable pyarrow fields when converting a `pa.Schema` to `Features`. At the same time, when outputting the `arrow_schema`, the original non-nullable fields are restored. This allows for more consistent behavior and avoids breaking behavior as illustrated in #7479.

I am by no means a pyarrow expert so some logic in `find_non_nullable_fields` may not perfect. Not sure if more logic (type checks) are needed for deep-checking a given schema. Maybe there are other pyarrow structures that need to be covered?

Tests are added, but again, these may not have sufficient coverage in terms of pyarrow structure types.

closes #7479
GitHub issue: deal with python `10_000` legal number in slice syntax
State: closed

### Feature request

```
In [6]: ds = datasets.load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:1000]")

In [7]: ds = datasets.load_dataset("HuggingFaceH4/ultrachat_200k", split="train_sft[:1_000]")
[dozens of frames skipped]
File /usr/local/lib/python3.10/dist-packages/datasets/arrow_reader.py:444, in _str_to_read_instruction(spec)
    442 res = _SUB_SPEC_RE.match(spec)
    443 if not res:
--> 444     raise ValueError(f"Unrecognized instruction format: {spec}")
ValueError: Unrecognized instruction format: train_sft[:1_000]
```

It took me a while to understand what the problem was. But apparently `pyarrow` doesn't allow python numbers that may include `_` as in `1_000`. The `_` aids readability since `10_000_000` vs `10000000` is obviously easier to grasp of what the actual number is.

Feature request:

ideally `datasets` being a python module will do the right thing and convert python numbers into whatever pyarrow supports - in this case stripping `_`s.

Second best it'd err and tell the user that using numbers with `_` in split slices is not acceptible, so that the user won't have to deal with a huge pyarrow assert they know nothing about.

Thank you!
GitHub issue: HF_DATASETS_CACHE ignored?
State: open

### Describe the bug

I'm struggling to get things to respect HF_DATASETS_CACHE.

Rationale: I'm on a system that uses NFS for homedir, so downloading to NFS is expensive, slow, and wastes valuable quota compared to local disk. Instead, it seems to rely mostly on HF_HUB_CACHE.

Current version: 3.2.1dev. In the process of testing 3.4.0

### Steps to reproduce the bug

[Currently writing using datasets 3.2.1dev. Will follow up with 3.4.0 results]

dump.py:
```python
from datasets import load_dataset
dataset = load_dataset("HuggingFaceFW/fineweb", name="sample-100BT", split="train")
```

Repro steps
```bash
# ensure no cache
$ mv ~/.cache/huggingface ~/.cache/huggingface.bak

$ export HF_DATASETS_CACHE=/tmp/roller/datasets
$ rm -rf ${HF_DATASETS_CACHE}
$ env | grep HF | grep -v TOKEN
HF_DATASETS_CACHE=/tmp/roller/datasets

$ python dump.py
# (omitted for brevity)

# (while downloading) 
$ du -hcs ~/.cache/huggingface/hub
18G     hub
18G     total

# (after downloading)
$ du -hcs ~/.cache/huggingface/hub
```

It's a shame because datasets supports s3 (which I could really use right now) but hub does not.

### Expected behavior

* ~/.cache/huggingface/hub stays empty
* /tmp/roller/datasets becomes full of stuff

### Environment info

[Currently writing using datasets 3.2.1dev. Will follow up with 3.4.0 results]
GitHub issue: Features.from_arrow_schema is destructive
State: open

### Describe the bug

I came across this, perhaps niche, bug where `Features` does not/cannot account for pyarrow's `nullable=False` option in Fields. Interestingly, I found that in regular "flat" fields this does not necessarily lead to conflicts, but when a non-nullable field is in a struct, an incompatibility arises.

It's not easy to explain in words, so the minimal example below should help I hope.

Note that I suggest a solution in the comments in the code, simply allowing `Dataset.to_parquet` to allow for a `schema` argument which, when provided, will override the default ds.features.arrow_schema.

### Steps to reproduce the bug

```python
import os
from datasets import Dataset, Features

import pyarrow as pa
import pyarrow.parquet as pq

# HF datasets is destructive when you call Features.from_arrow_schema(schema) on a schema 
# because it will not account for nullable and non-nullable fields in structs (it will always allow nullable)
# Reloading the same dataset with the original schema will raise an error because the schema is not the same anymore
non_nullable_schema = pa.schema(
    [
        pa.field("text", pa.string(), nullable=False),
        pa.field("meta",
            pa.struct(
                [
                    pa.field("date", pa.list_(pa.string()), nullable=False),
                ],
            ),
        ),

    ]
)
print("ORIGINAL SCHEMA")
print(non_nullable_schema)
print()

feats = Features.from_arrow_schema(non_nullable_schema)

print("FEATUR-IZED SCHEMA (nullable-restrictions are gone)")
print(feats.arrow_schema)
print()

ds = Dataset.from_dict(
    {
        "text": ["a", "b", "c"],
        "meta": [{"date": ["2021-01-01"]}, {"date": ["2021-01-02"]}, {"date": ["2021-01-03"]}],
    },
    features=feats,
)

fname = "tmp.parquet"

# This is not possible: TypeError: pyarrow.parquet.core.ParquetWriter() got multiple values for keyword argument 'schema'
# Though I believe this would be the easiest fix: allow schema to be passed to to_parquet and overwrite the schema in the dataset
# ds.to_parquet(fname, schema=non_nullable_schema)

ds.to_parquet(fname)

try:
    _ = pq.read_table(fname, schema=non_nullable_schema)
finally:
    os.unlink(fname)
```


### Expected behavior

- Non-destructive behavior when converting an arrow schema to Features; or
- the ability to override the default arrow schema with a custom one

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-427.20.1.el9_4.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.10
- `huggingface_hub` version: 0.27.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
GitHub issue: What is the canonical way to compress a Dataset?
State: open

Given that Arrow is the preferred backend for a Dataset, what is a user supposed to do if they want concurrent reads, concurrent writes AND on-disk compression for a larger dataset?

Parquet would be the obvious answer except that there is no native support for writing sharded, parquet datasets concurrently [[1](https://github.com/huggingface/datasets/issues/7047)].

Am I missing something?  

And if so, why is this not the standard/default way that `Dataset`'s work as they do in Xarray, Ray Data, Composer, etc.?
GitHub issue: IterableDataset's state_dict shard_example_idx is always equal to the number of samples in a shard
State: open

### Describe the bug

I've noticed a strange behaviour with Iterable state_dict: the value of shard_example_idx is always equal to the amount of samples in a shard.


### Steps to reproduce the bug

I am reusing the example from the doc
```python
from datasets import Dataset

ds = Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=1)
state_dict = None
# Iterate through the dataset and print examples
for idx, example in enumerate(ds):
    print(example)
    if idx == 2:
        state_dict = ds.state_dict()
        print("checkpoint")
        break
print(state_dict)
```
Returns:
```
{'a': 0}
{'a': 1}
checkpoint
{'examples_iterable': {'shard_idx': 0, 'shard_example_idx': 6, 'type': 'ArrowExamplesIterable'}, 'epoch': 0}
```


### Expected behavior

shard_example_idx should be 2 instead of 6
If we run with num_shards=2, then shard_example_idx is 3 instead of 2 and so on.

### Environment info

- `datasets` version: 3.4.1
- Platform: macOS-14.6.1-arm64-arm-64bit
- Python version: 3.12.9
- `huggingface_hub` version: 0.29.3
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
GitHub issue: Webdataset data format problem
State: closed

### Describe the bug

Please see https://huggingface.co/datasets/ejschwartz/idioms/discussions/1

Error code: FileFormatMismatchBetweenSplitsError

All three splits, train, test, and validation, use webdataset. But only the train split has more than one file. How can I force the other two splits to also be interpreted as being the webdataset format?  (I don't think there is currently a way, but happy to be told that I am wrong.)

### Steps to reproduce the bug

```
import datasets
datasets.load_dataset("ejschwartz/idioms")

### Expected behavior

The dataset loads.  Alternatively, there is a YAML syntax for manually specifying the format.

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-6.8.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.28.1
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
GitHub issue: Adding argument to `_get_data_files_patterns`
State: closed

### Feature request

How about adding if the user already know about the pattern?

https://github.com/huggingface/datasets/blob/a256b85cbc67aa3f0e75d32d6586afc507cf535b/src/datasets/data_files.py#L252

### Motivation

While using this load_dataset people might use 10M of images for the local files.
However, due to searching all the appropriate file pattern in fsspec, purely searching this pattern takes more than 10 hours (real use-case).

### Your contribution

Yeah I can make this happen if this seems valid. @lhoestq WDYT?
such like
```
def _get_data_files_patterns(pattern_resolver: Callable[[str], list[str]], patterns: PATTERNS) -> dict[str, list[str]]:
```
GitHub issue: Is it possible to shard a single-sharded IterableDataset?
State: closed

I thought https://github.com/huggingface/datasets/pull/7252 might be applicable but looking at it maybe not.

Say we have a process, eg. a database query, that can return data in slightly different order each time. So, the initial query needs to be run by a single thread (not to mention running multiple times incurs more cost too). But the results are also big enough that we don't want to materialize it entirely and instead stream it with an IterableDataset.

But after we have the results we want to split it up across workers to parallelize processing.

Is something like this possible to do?

Here's a failed attempt. The end result should be that each of the shards has unique data, but unfortunately with this attempt the generator gets run once in each shard and the results end up with duplicates...

```
import random
import datasets


def gen():
  print('RUNNING GENERATOR!')
  items = list(range(10))
  random.shuffle(items)
  yield from items


ds = datasets.IterableDataset.from_generator(gen)

print('dataset contents:')
for item in ds:
  print(item)
print()

print('dataset contents (2):')
for item in ds:
  print(item)
print()


num_shards = 3


def sharded(shard_id):
  for i, example in enumerate(ds):
    if i % num_shards in shard_id:
      yield example


ds1 = datasets.IterableDataset.from_generator(
  sharded, gen_kwargs={'shard_id': list(range(num_shards))}
)

for shard in range(num_shards):
  print('shard', shard)
  for item in ds1.shard(num_shards, shard):
    print(item)
```
GitHub issue: Custom split name with the web interface
State: closed

### Describe the bug

According the doc here: https://huggingface.co/docs/hub/datasets-file-names-and-splits#custom-split-name
it should infer the split name from the subdir of data or the beg of the name of the files in data.
When doing this manually through web upload it does not work. it uses "train" as a unique split.
example: https://huggingface.co/datasets/eole-nlp/estimator_chatml

### Steps to reproduce the bug

follow the link above

### Expected behavior

there should be two splits "mlqe" and "1720_da"

### Environment info

website
GitHub issue: function `load_dataset`  can't solve folder path with regex characters like "[]"
State: open

### Describe the bug

When using the `load_dataset` function with a folder path containing regex special characters (such as "[]"), the issue occurs due to how the path is handled in the `resolve_pattern` function. This function passes the unprocessed path directly to `AbstractFileSystem.glob`, which supports regular expressions. As a result, the globbing mechanism interprets these characters as regex patterns, leading to a traversal of the entire disk partition instead of confining the search to the intended directory.

### Steps to reproduce the bug

just create a folder like `E:\[D_DATA]\koch_test`, then `load_dataset("parquet", data_dir="E:\[D_DATA]\\test", split="train")`
it will keep searching the whole disk.

I add two `print` in `glob` and `resolve_pattern` to see the path

### Expected behavior

it should load the dataset as in normal folders

### Environment info

- `datasets` version: 3.3.2
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.10.16
- `huggingface_hub` version: 0.29.1
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2024.12.0
GitHub issue: load_dataset with streaming hangs on parquet datasets
State: open

### Describe the bug

When I try to load a dataset with parquet files (e.g. "bigcode/the-stack") the dataset loads, but python interpreter can't exit and hangs

### Steps to reproduce the bug

```python3
import datasets

print('Start')
dataset = datasets.load_dataset("bigcode/the-stack", data_dir="data/yaml", streaming=True, split="train")
it = iter(dataset)
next(it)
print('Finish')
```

The program prints finish but doesn't exit and hangs indefinitely.
I tried this on two different machines and several datasets.

### Expected behavior

The program exits successfully

### Environment info

datasets==3.4.1
Python 3.12.9.
MacOS and Ubuntu Linux
GitHub issue: Fix local pdf loading
State: closed

fir this error when accessing a local pdf

```
File ~/.pyenv/versions/3.12.2/envs/hf-datasets/lib/python3.12/site-packages/pdfminer/psparser.py:220, in PSBaseParser.seek(self, pos)
    218 """Seeks the parser to the given position."""
    219 log.debug("seek: %r", pos)
--> 220 self.fp.seek(pos)
    221 # reset the status for nextline()
    222 self.bufpos = pos

ValueError: seek of closed file
```
GitHub issue: List of images behave differently on IterableDataset and Dataset
State: closed

### Describe the bug

This code:

```python
def train_iterable_gen():
        images = np.array(load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg").resize((128, 128)))
        yield {
            "images": np.expand_dims(images, axis=0),
            "messages": [
                {
                    "role": "user",
                    "content": [{"type": "image", "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" }]
                },
                {
                    "role": "assistant",
                    "content": [{"type": "text", "text": "duck" }]
                }
            ]
        }
    train_ds = Dataset.from_generator(train_iterable_gen,
                                      features=Features({
                                           'images': [datasets.Image(mode=None, decode=True, id=None)],
                                           'messages': [{'content': [{'text': datasets.Value(dtype='string', id=None), 'type': datasets.Value(dtype='string', id=None) }], 'role': datasets.Value(dtype='string', id=None)}]
                                           } )
                                           )
```

works as I'd expect; if I iterate the dataset then the `images` column returns a `List[PIL.Image.Image]`, i.e. `'images': [<PIL.PngImagePlugin.PngImageFile image mode=RGB size=128x128 at 0x77EFB7EF4680>]`.

But if I change `Dataset` to `IterableDataset`, the `images` column changes into `'images': [{'path': None, 'bytes': ..]`

### Steps to reproduce the bug

The code above +

```python
def load_image(url):
        response = requests.get(url)
        image = Image.open(io.BytesIO(response.content))
        return image
```

I'm feeding it to SFTTrainer

### Expected behavior

Dataset and IterableDataset would behave the same

### Environment info

```yaml
requires-python = ">=3.12"
dependencies = [
    "av>=14.1.0",
    "boto3>=1.36.7",
    "datasets>=3.3.2",
    "docker>=7.1.0",
    "google-cloud-storage>=2.19.0",
    "grpcio>=1.70.0",
    "grpcio-tools>=1.70.0",
    "moviepy>=2.1.2",
    "open-clip-torch>=2.31.0",
    "opencv-python>=4.11.0.86; sys_platform == 'darwin'",
    "opencv-python-headless>=4.11.0.86; sys_platform == 'linux'",
    "pandas>=2.2.3",
    "pillow>=10.4.0",
    "plotly>=6.0.0",
    "py-spy>=0.4.0",
    "pydantic>=2.10.6",
    "pydantic-settings>=2.7.1",
    "pymysql>=1.1.1",
    "ray[data,default,serve,train,tune]>=2.43.0",
    "torch>=2.6.0",
    "torchmetrics>=1.6.1",
    "torchvision>=0.21.0",
    "transformers[torch]@git+https://github.com/huggingface/transformers",
    "wandb>=0.19.4",
    # https://github.com/Dao-AILab/flash-attention/issues/833
    "flash-attn @ https://github.com/Dao-AILab/flash-attention/releases/download/v2.7.3/flash_attn-2.7.3+cu12torch2.6cxx11abiFALSE-cp312-cp312-linux_x86_64.whl; sys_platform == 'linux'",
    "trl@https://github.com/huggingface/trl.git",
    "peft>=0.14.0",
]
```
GitHub issue: Document the HF_DATASETS_CACHE env variable
State: open

### Feature request

Hello,

I have a use case where my team is sharing models and dataset in shared directory to avoid duplication.
I noticed that the [cache documentation for datasets](https://huggingface.co/docs/datasets/main/en/cache) only mention the `HF_HOME` environment variable but never the `HF_DATASETS_CACHE`.

It should be nice to add `HF_DATASETS_CACHE` to datasets documentation if it's an intended feature.
If it's not, I think a depreciation warning would be appreciated.

### Motivation

This variable is fully working and similar to what `HF_HUB_CACHE` does for models, so it's nice to know that this exists. This seems to be a quick change to implement.

### Your contribution

I could contribute since this is only affecting a small portion of the documentation