rIsHu009/github-issues-updated huggingface dataset:
Title: loading mozilla-foundation--common_voice_11_0 fails
Author: pavel-esir (NONE)
State: open
Created: 2025-06-26T12:23:48Z

### Describe the bug

Hello everyone,

i am trying to load `mozilla-foundation--common_voice_11_0` and it fails. Reproducer 

```
import datasets
datasets.load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True, trust_remote_code=True)
```
and it fails with
```
File ~/opt/envs/.../lib/python3.10/site-packages/datasets/utils/file_utils.py:827, in _add_retries_to_file_obj_read_method.<locals>.read_with_retries(*args, **kwargs)
    825 for retry in range(1, max_retries + 1):
    826     try:
--> 827         out = read(*args, **kwargs)
    828         break
    829     except (
    830         _AiohttpClientError,
    831         asyncio.TimeoutError,
    832         requests.exceptions.ConnectionError,
    833         requests.exceptions.Timeout,
    834     ) as err:

File /usr/lib/python3.10/codecs.py:322, in BufferedIncrementalDecoder.decode(self, input, final)
    319 def decode(self, input, final=False):
    320     # decode input (taking the buffer into account)
    321     data = self.buffer + input
--> 322     (result, consumed) = self._buffer_decode(data, self.errors, final)
    323     # keep undecoded input until the next call
    324     self.buffer = data[consumed:]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
```
When i remove streaming then everything is good but i need `streaming=True`

### Steps to reproduce the bug

```
import datasets
datasets.load_dataset("mozilla-foundation/common_voice_11_0", "en", split="test", streaming=True, trust_remote_code=True)
```

### Expected behavior

Expected that it will download dataset

### Environment info

datasets==3.6.0 
python3.10
on all platforms linux/win/mac

Comments:
Title: Update data_files.py #7066
Author: ArjunJagdale (CONTRIBUTOR)
State: open
Created: 2025-06-26T07:01:37Z

Fixes #7066
This PR introduces automatic **subset-level grouping** for folder-based dataset builders by:

1. Adding a utility function `group_files_by_subset()` that clusters files by root name (ignoring digits and shard suffixes).
2. Integrating this logic into `FolderBasedBuilder._split_generators()` to yield one split per subset.
3. Adding unit tests for the grouping function.
4. Updating the documentation to describe this new behavior under `docs/source/repository_structure.mdx`.

---

### Motivation

Datasets with files like:
```

train0.jsonl
train1.jsonl
animals.jsonl
metadata.jsonl

```

will now be **automatically grouped** as:
- `"train"` subset → `train0.jsonl`, `train1.jsonl`
- `"animals"` subset → `animals.jsonl`
- `"metadata"` subset → `metadata.jsonl`

This enables structured multi-subset loading even when the dataset doesn't follow traditional `train/validation/test` split conventions.

---

### Files Changed

- `src/datasets/data_files.py`: added `group_files_by_subset()` utility
- `src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py`: grouped files before yielding splits
- `tests/test_data_files.py`: added unit test `test_group_files_by_subset`
- `docs/source/repository_structure.mdx`: documented subset grouping for maintainers and users

---

### Benefits

- More flexible and robust dataset split logic
- Enables logical grouping of user-uploaded files without nested folder structure
- Backward-compatible with all existing folder-based configs

---

Ready for review ✅

Comments:
It adds automatic grouping of files into subsets based on their root name (e.g., `train0.jsonl`, `train1.jsonl` → `"train"`), as discussed above. The logic is integrated into `FolderBasedBuilder` and is fully tested + documented.

Let me know if any changes are needed — happy to iterate!

Hi ! I believe the subsets need to be instantiated here as `configs` - not `splits` (which are meant for train/validation/test):

https://github.com/huggingface/datasets/blob/ef762e664a2a1675368ed7a203b0ac8cecca6e19/src/datasets/load.py#L647-L662

Also the subset names should probably be inferred only from the parquet/csv/json files and not from png/jpeg/wav/mp4 etc. WDYT ?
Title: `ClassLabel` docs: Correct value for unknown labels
Author: l-uuz (NONE)
State: open
Created: 2025-06-25T20:01:35Z

This small change fixes the documentation to to be compliant with what happens in `encode_example`.
https://github.com/huggingface/datasets/blob/e71b0b19d79c7531f9b9bea7c09916b5f6157f42/src/datasets/features/features.py#L1126-L1129

Comments:
Title: fix sequence ci
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-25T17:07:55Z

fix error from https://github.com/huggingface/datasets/pull/7643

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7644). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Backward compat sequence instance
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-25T17:05:09Z

useful to still get `isinstance(Sequence(Value("int64")), Sequence)`for downstream libs like evaluate

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7643). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: update docs and docstrings
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-25T14:48:58Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7641). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: fix save_infos
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-25T13:16:26Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7639). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Add ignore_decode_errors option to Image feature for robust decoding #7612
Author: ArjunJagdale (CONTRIBUTOR)
State: open
Created: 2025-06-24T16:47:51Z

This PR implements support for robust image decoding in the `Image` feature, as discussed in issue #7612.

## 🔧 What was added
- A new boolean field: `ignore_decode_errors` (default: `False`)
- If set to `True`, any exceptions during decoding will be caught, and `None` will be returned instead of raising an error

```python
features = Features({
    "image": Image(decode=True, ignore_decode_errors=True),
})
````

This enables robust iteration over potentially corrupted datasets — especially useful when streaming datasets like WebDataset or image-heavy public sets where sample corruption is common.

## 🧪 Behavior

* If `ignore_decode_errors=False` (default), decoding behaves exactly as before
* If `True`, decoding errors are caught, and a warning is emitted:

  ```
  [Image.decode_example] Skipped corrupted image: ...
  ```

## 🧵 Linked issue

Closes #7612

Let me know if you'd like a follow-up test PR. Happy to write one!

Comments:
cc @lhoestq
Title: Introduce subset_name as an alias of config_name
Author: albertvillanova (MEMBER)
State: open
Created: 2025-06-24T12:49:01Z
Labels: enhancement

### Feature request

Add support for `subset_name` as an alias for `config_name` in the datasets library and related tools (such as loading scripts, documentation, and metadata).

### Motivation

The Hugging Face Hub dataset viewer displays a column named **"Subset"**, which refers to what is currently technically called config_name in the datasets library. This inconsistency has caused confusion for many users, especially those unfamiliar with the internal terminology.

I have repeatedly received questions from users trying to understand what "config" means, and why it doesn’t match what they see as "subset" on the Hub. Renaming everything to `subset_name` might be too disruptive, but introducing subset_name as a clear alias for config_name could significantly improve user experience without breaking backward compatibility.

This change would:
- Align terminology across the Hub UI and datasets codebase
- Reduce user confusion, especially for newcomers
- Make documentation and examples more intuitive

Comments:
I second this! When you come from the Hub, the intuitive question is "how do I set the subset name", and it's not easily answered from the docs: `subset_name` would answer this directly.
Title: "open" in globals()["__builtins__"], an error occurs: "TypeError: argument of type 'module' is not iterable"
Author: kuanyan9527 (NONE)
State: open
Created: 2025-06-24T08:09:39Z

When I run the following code, an error occurs: "TypeError: argument of type 'module' is not iterable"
```python
print("open" in globals()["__builtins__"])
```
Traceback (most recent call last):
  File "./main.py", line 2, in <module>
    print("open" in globals()["__builtins__"])
          ^^^^^^^^^^^^^^^^^^^^^^
TypeError: argument of type 'module' is not iterable

But this code runs fine in datasets, I don't understand why
[src/datasets/utils/patching.py#L96](https://github.com/huggingface/datasets/blob/3.6.0/src/datasets/utils/patching.py#L96)

Comments:
Title: Fix: Preserve float columns in JSON loader when values are integer-like (e.g. 0.0, 1.0)
Author: ArjunJagdale (CONTRIBUTOR)
State: open
Created: 2025-06-24T06:16:48Z

This PR fixes a bug in the JSON loader where columns containing float values like `[0.0, 1.0, 2.0]` were being implicitly coerced to `int`, due to pandas or Arrow type inference.

This caused issues downstream in statistics computation (e.g., dataset-viewer) where such columns were incorrectly labeled as `"int"` instead of `"float"`.

### 🔍 What was happening:
When the JSON loader falls back to `pandas_read_json()` (after `pa.read_json()` fails), pandas/Arrow can coerce float values to integers if all values are integer-like (e.g., `0.0 == 0`).

### ✅ What this PR does:
- Adds a check in the fallback path of `_generate_tables()`
- Ensures that columns made entirely of floats are preserved as `"float64"` even if they are integer-like (e.g. `0.0`, `1.0`)
- This prevents loss of float semantics when creating the Arrow table

### 🧪 Reproducible Example:
```json
[{"col": 0.0}, {"col": 1.0}, {"col": 2.0}]
````

Previously loaded as:

* `int`

Now correctly loaded as:

* `float`

Fixes #6937

Comments:
Title: Replace Sequence by List
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-23T20:35:48Z

Sequence is just a utility that we need to keep for backward compatibility. And `[ ]` was used instead but doesn't allow passing the length of the list.

This PR removes most mentions of Sequence and usage of `[ ]` and defines a proper List type instead.

before: `Sequence(Value("int64"))` or `[Value("int64")]`
now: `List(Value("int64"))`

This PR conserves full backward compatibility. And it's a good occasion with the release of 4.0.0

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7634). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Proposal: Small Tamil Discourse Coherence Dataset.
Author: bikkiNitSrinagar (NONE)
State: open
Created: 2025-06-23T14:24:40Z

I’m a beginner from NIT Srinagar proposing a dataset of 50 Tamil text pairs for discourse coherence (coherent/incoherent labels) to support NLP research in low-resource languages.
- Size: 50 samples
- Format: CSV with columns (text1, text2, label)
- Use case: Training NLP models for coherence
I’ll use GitHub’s web editor and Google Colab. Please confirm if this fits.

Comments:
Title: Graceful Error Handling for cast_column("image", Image(decode=True)) in Hugging Face Datasets
Author: ganiket19 (NONE)
State: open
Created: 2025-06-23T13:49:24Z
Labels: enhancement

### Feature request

Currently, when using dataset.cast_column("image", Image(decode=True)), the pipeline throws an error and halts if any image in the dataset is invalid or corrupted (e.g., truncated files, incorrect formats, unreachable URLs). This behavior disrupts large-scale processing where a few faulty samples are common.
reference : https://discuss.huggingface.co/t/handle-errors-when-loading-images-404-corrupted-etc/50318/5
                  https://discuss.huggingface.co/t/handling-non-existing-url-in-image-dataset-while-cast-column/69185
                  
Proposed Feature

Introduce a mechanism (e.g., a continue_on_error=True flag or global error handling mode) in Image(decode=True) that:

Skips invalid images and sets them as None, or

Logs the error but allows the rest of the dataset to be processed without interruption.


Example Usage

from datasets import load_dataset, Image

dataset = load_dataset("my_dataset")
dataset = dataset.cast_column("image", Image(decode=True, continue_on_error=True))

Benefits

Ensures robust large-scale image dataset processing.

Improves developer productivity by avoiding custom retry/error-handling code.

Aligns with best practices in dataset preprocessing pipelines that tolerate minor data corruption.


Potential Implementation Options

Internally wrap the decoding in a try/except block.

Return None or a placeholder on failure.

Optionally allow custom error callbacks or logging.

### Motivation

Robustness: Large-scale image datasets often contain a small fraction of corrupt files or unreachable URLs. Halting on the first error forces users to write custom workarounds or preprocess externally.

Simplicity: A built-in flag removes boilerplate try/except logic around every decode step.

Performance: Skipping invalid samples inline is more efficient than a two-pass approach (filter then decode).

### Your contribution


1. API Change
Extend datasets.features.Image(decode=True) to accept continue_on_error: bool = False.

2. Behavior
If continue_on_error=False (default), maintain current behavior: any decode error raises an exception.
If continue_on_error=True, wrap decode logic in try/except:
On success: store the decoded image.
On failure: log a warning (e.g., via logging.warning) and set the field to None (or a sentinel value).


3. Optional Enhancements
Allow a callback hook:
Image(decode=True, continue_on_error=True, on_error=lambda idx, url, exc: ...)
Emit metrics or counts of skipped images.

Comments:
Title: Pass user-agent from DownloadConfig into fsspec storage_options
Author: ArjunJagdale (CONTRIBUTOR)
State: open
Created: 2025-06-21T14:22:25Z

Fixes part of issue #6046

### Problem
The `user-agent` defined in `DownloadConfig` was not passed down to fsspec-based filesystems like `HfFileSystem`, which prevents proper identification/tracking of client requests.

### Solution
Added support for injecting the `user-agent` into `storage_options["headers"]` within `_prepare_single_hop_path_and_storage_options()` based on the `protocol`.

Now, when using `hf://`, `http://`, or `https://`, the custom user-agent is passed automatically.

### Code Location
Modified:
- `src/datasets/utils/file_utils.py`

Used `get_datasets_user_agent(...)` to ensure proper formatting and fallback logic.

Comments:
- This PR assumes that `HfFileSystem` in `huggingface_hub` supports receiving `headers` in `storage_options`. If not, a follow-up PR can be opened to add this support to `HfFileSystem.__init__`.
- No test was added for this since it’s a config passthrough. If needed, I’d be happy to add one.
Title: [bug] resume from ckpt skips samples if .map is applied
Author: felipemello1 (NONE)
State: open
Created: 2025-06-21T01:50:03Z

### Describe the bug

resume from ckpt skips samples if .map is applied

Maybe related: https://github.com/huggingface/datasets/issues/7538

### Steps to reproduce the bug

```python
from datasets import Dataset
from datasets.distributed import split_dataset_by_node


# Create dataset with map transformation
def create_dataset():
    ds = Dataset.from_dict({"id": list(range(100))})
    ds = ds.to_iterable_dataset(num_shards=4)
    ds = ds.map(lambda x: x) #comment it out to get desired behavior
    ds = split_dataset_by_node(ds, rank=0, world_size=2)
    return ds


ds = create_dataset()

# Iterate and save checkpoint after 10 samples
it = iter(ds)
for idx, sample in enumerate(it):
    if idx == 9:  # Checkpoint after 10 samples
        checkpoint = ds.state_dict()
        print(f"Checkpoint saved at sample: {sample['id']}")
        break

# Continue with original iterator
original_next_samples = []
for idx, sample in enumerate(it):
    original_next_samples.append(sample["id"])
    if idx >= 4:
        break

# Resume from checkpoint
ds_new = create_dataset()
ds_new.load_state_dict(checkpoint)

# Get samples from resumed iterator
it_new = iter(ds_new)
resumed_next_samples = []
for idx, sample in enumerate(it_new):
    resumed_next_samples.append(sample["id"])
    if idx >= 4:
        break

print(f"\nExpected next samples: {original_next_samples}")
print(f"Actual next samples:   {resumed_next_samples}")
print(
    f"\n❌ BUG: {resumed_next_samples[0] - original_next_samples[0]} samples were skipped!"
)

```

With map
```
Checkpoint saved at sample: 9

Expected next samples: [10, 11, 12, 13, 14]
Actual next samples:   [50, 51, 52, 53, 54]

❌ BUG: 40 samples were skipped!
```

### Expected behavior

without map
```
Expected next samples: [10, 11, 12, 13, 14]
Actual next samples:   [10, 11, 12, 13, 14]

❌ BUG: 0 samples were skipped!
```

### Environment info

datasets == 3.6.0

Comments:
#selfassign

Hi! I'd like to work on this issue.
Title: Add test for `as_iterable_dataset()` method in DatasetBuilder
Author: ArjunJagdale (CONTRIBUTOR)
State: open
Created: 2025-06-19T19:23:55Z

This PR adds a test for the new `as_iterable_dataset()` method introduced in PR #7628.

The test:
- Loads a builder using `load_dataset_builder("c4", "en")`
- Runs `download_and_prepare()`
- Streams examples using `builder.as_iterable_dataset(split="train[:100]")`
- Verifies streamed examples contain the "text" field

This ensures that the builder correctly streams data from cached Arrow files.

Comments:
Title: Add `as_iterable_dataset()` method to DatasetBuilder for streaming from cached Arrow files
Author: ArjunJagdale (CONTRIBUTOR)
State: open
Created: 2025-06-19T19:15:41Z

This PR implements `builder.as_iterable_dataset(split=...)` as discussed in #5481.

It allows users to load an `IterableDataset` directly from cached Arrow files (using ArrowReader and ArrowExamplesIterable), without loading the full dataset into memory.

This is useful for large-scale training scenarios where memory is constrained. A test has also been added in `test_builder.py`.

Related to: #5481

Comments:
Title: Creating a HF Dataset from lakeFS with S3 storage takes too much time!
Author: Thunderhead-exe (NONE)
State: closed
Created: 2025-06-19T14:28:41Z

Hi,

I’m new to HF dataset and I tried to create datasets based on data versioned in **lakeFS** _(**MinIO** S3 bucket as storage backend)_

Here I’m using ±30000 PIL image from MNIST data however it is taking around 12min to execute, which is a lot!

From what I understand, it is loading the images into cache then building the dataset.
– Please find bellow the execution screenshot –

Is there a way to optimize this or am I doing something wrong?

Thanks!

![Image](https://github.com/user-attachments/assets/c79257c8-f023-42a9-9e6f-0898b3ea93fe)

Comments:
### > Update

The bottleneck, from what I understand, was making one network request per file

For 30k images, this meant 30k separate GET requests to the MinIO server through the S3 API, and that was killing the performance

Using webDataset to transform the large number of files to few .tar files and passing “webdataset” instead of “imagefolder” to the load_dataset function worked perfectly (took only ~11s)
Title: feat(map): reuse unchanged columns when input_columns specified to reduce disk usage (#6013)
Author: ArjunJagdale (CONTRIBUTOR)
State: open
Created: 2025-06-19T07:41:45Z

## Summary

This PR addresses [#6013](https://github.com/huggingface/datasets/issues/6013) by reusing unchanged columns from the original dataset in the `map()` method when `input_columns` is specified.

## What’s Implemented

- Injected logic at the end of `Dataset.map()` to:
  - Identify untouched columns not in `input_columns` or `remove_columns`
  - Select those columns from the original dataset
  - Concatenate them with the transformed result using `pyarrow.concat_tables`

## Example Behavior

```python
ds = Dataset.from_dict({"a": [1, 2], "b": [3, 4]})
ds2 = ds.map(lambda x: {"c": x["a"] + 10}, input_columns=["a"], remove_columns=["a"])
print(ds2.column_names)  # Output: ['b', 'c']
````

Column `b` is reused from the original dataset.

## Notes

* This keeps disk usage and caching minimal by avoiding full dataset duplication.
* Only triggered when `input_columns` is set.

---

cc @lhoestq @mariosasko for review 🙂

Comments:
Title: feat: Add h5folder dataset loader for HDF5 support
Author: ArjunJagdale (CONTRIBUTOR)
State: open
Created: 2025-06-19T05:39:10Z

### Related Issue
Closes #3113

### What does this PR do?

This PR introduces a new dataset loader module called **`h5folder`** to support loading datasets stored in **HDF5 (.h5)** format.

It allows users to do:

```python
from datasets import load_dataset
dataset = load_dataset("h5folder", data_dir="path/to/")
````

### 🧩 Design Overview

* Implemented inside `datasets/packaged_modules/h5folder/h5folder.py`
* Based on the `GeneratorBasedBuilder` API
* Uses `h5py` to read HDF5 files and yield examples
* Expects datasets such as `id`, `data`, and `label` inside `data.h5`
* Converts numpy arrays to Python types before yielding

### 🧪 Example `.h5` Structure (for local testing)

```python
import h5py
import numpy as np

with h5py.File("data.h5", "w") as f:
    f.create_dataset("id", data=np.arange(100))
    f.create_dataset("data", data=np.random.randn(100, 10))
    f.create_dataset("label", data=np.random.randint(0, 2, size=100))
```

### ✅ Testing

- The loader logic follows the structure of existing modules like `imagefolder`
- Will rely on Hugging Face CI to validate integration
- Manually testing planned once merged or during feedback

### 📁 Files Added

* `datasets/src/datasets/packaged_modules/h5folder/h5folder.py`

### 📌 Component(s) Affected

* `area/datasets`
* `area/load`

### 📦 Release Note Classification

* `rn/feature` – Adds support for loading `.h5` datasets via `load_dataset("h5folder", ...)`

---

Let me know if any changes or improvements are needed — happy to iterate. Thanks for reviewing!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7625). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I guess test failed cause import os, import h5py, and import datasets lines are not alphabetically sorted, or not grouped properly.

![image](https://github.com/user-attachments/assets/ab73f8f9-da50-4ba8-9b2d-b7c30fce94f5)


This commit was accidental - `[Merge branch 'main' into patch-4]`. The 
`[chore: fix import order in h5folder.py to satisfy linter]` should solve the import order issue.
Title: #Dataset Make "image" column appear first in dataset preview UI
Author: jcerveto (NONE)
State: closed
Created: 2025-06-18T09:25:19Z

Hi!

#Dataset

I’m currently uploading a dataset that includes an `"image"` column (PNG files), along with some metadata columns. The dataset is loaded from a .jsonl file. My goal is to have the "image" column appear as the first column in the dataset card preview UI on the :hugs: Hub.

However, at the moment, the `"image"` column is not the first—in fact, it appears last, which is not ideal for the presentation I’d like to achieve.

I have a couple of questions:

Is there a way to force the dataset card to display the `"image"` column first?
Is there currently any way to control or influence the column order in the dataset preview UI?
Does the order of keys in the .jsonl file or the features argument affect the display order?
Thanks again for your time and help! :blush:

Comments:
Hi ! It should follow the same order as the order of the keys in the metadata file

Hi! Thank you for your answer. 

As you said it, I I forced every key in every JSON to have an order using `collections. OrderedDict` in Python. Now, it works!

TY
Title: fix: raise error in FolderBasedBuilder when data_dir and data_files are missing
Author: ArjunJagdale (CONTRIBUTOR)
State: closed
Created: 2025-06-17T19:16:34Z

### Related Issues/PRs

Fixes #6152

---

### What changes are proposed in this pull request?

This PR adds a dedicated validation check in the `_info()` method of the `FolderBasedBuilder` class to ensure that users provide either `data_dir` or `data_files` when loading folder-based datasets (such as `audiofolder`, `imagefolder`, etc.).

---

### Why this change?

Previously, when calling:

```python
load_dataset("audiofolder")
````

without specifying `data_dir` or `data_files`, the loader would silently fallback to the **current working directory**, leading to:

* Long loading times
* Unexpected behavior (e.g., scanning unrelated files)

This behavior was discussed in issue #6152. As suggested by maintainers, the fix has now been implemented directly inside the `FolderBasedBuilder._info()` method — keeping the logic localized to the specific builder instead of a generic loader function.

---

### How is this PR tested?

* ✅ Manually tested by calling `load_dataset("audiofolder")` with no `data_dir` or `data_files` → a `ValueError` is now raised early.
* ✅ Existing functionality (with valid input) remains unaffected.

---

### Does this PR require documentation update?

* [x] No

---

### Release Notes

#### Is this a user-facing change?

* [x] Yes

> Folder-based datasets now raise an explicit error if neither `data_dir` nor `data_files` are specified, preventing unintended fallback to the current working directory.

---

#### What component(s) does this PR affect?

* [x] `area/datasets`
* [x] `area/load`

---

<a name="release-note-category"></a>

#### How should the PR be classified?

* [x] `rn/bug-fix` - A user-facing bug fix

---

#### Should this be included in the next patch release?

* [x] Yes

Comments:
@lhoestq Moved the logic to FolderBasedBuilder._info() as discussed in previous PR (#7618). Let me know if anything else is needed — happy to update!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7623). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Guard against duplicate builder_kwargs/config_kwargs in load_dataset_…
Author: Shohail-Ismail (NONE)
State: open
Created: 2025-06-17T18:28:35Z

…builder (#4910 )

### What does this PR do?

Fixes edge case in `load_dataset_builder` by raising a `TypeError` if the same key exists in both `builder_kwargs` and `config_kwargs`.

### Implementation details

- Added a guard clause in `load_dataset_builder` to detect duplicate keys between `builder_kwargs` and `config_kwargs`
 
- Wrote a unit test in `tests/test_load_duplicate_keys.py` to verify the exception is raised correctly 

### Fixes

Closes #4910 

### Reviewers

@zach-huggingface
@SunMarc 

Would appreciate your review if you have time - thanks!

Comments:
Title: minor docs data aug
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-17T14:46:57Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7621). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Fixes in docs
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-17T13:41:54Z

before release 4.0

(I also did minor improvements to `features` to not show their `id=None` in their `__repr__()`)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7620). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: `from_list` fails while `from_generator` works for large datasets
Author: abdulfatir (NONE)
State: open
Created: 2025-06-17T10:58:55Z

### Describe the bug

I am constructing a large time series dataset and observed that first constructing a list of entries and then using `Dataset.from_list` led to a crash as the number of items became large. However, this is not a problem when using `Dataset.from_generator`.

### Steps to reproduce the bug

#### Snippet A (crashes)

```py
from tqdm.auto import tqdm
import numpy as np
import datasets


def data_generator():
    for i in tqdm(range(10_000_000)):
        length = np.random.randint(2048)
        series = np.random.rand(length)
        yield {"target": series, "item_id": str(i), "start": np.datetime64("2000", "ms")}

data_list = list(data_generator())
ds = datasets.Dataset.from_list(data_list)
```
The last line crashes with
```
ArrowInvalid: Value 2147483761 too large to fit in C integer type
```

#### Snippet B (works)

```py
from tqdm.auto import tqdm
import numpy as np
import datasets


def data_generator():
    for i in tqdm(range(10_000_000)):
        length = np.random.randint(2048)
        series = np.random.rand(length)
        yield {"target": series, "item_id": str(i), "start": np.datetime64("2000", "ms")}

ds = datasets.Dataset.from_generator(data_generator)
```

### Expected behavior

I expected both the approaches to work or to fail similarly. 

### Environment info

```
- `datasets` version: 3.6.0
- Platform: Linux-6.8.0-1029-aws-x86_64-with-glibc2.35
- Python version: 3.11.11
- `huggingface_hub` version: 0.32.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.3
- `fsspec` version: 2025.3.0
```

Comments:
@lhoestq any thoughts on this?
Title: fix: raise error when folder-based datasets are loaded without data_dir or data_files
Author: ArjunJagdale (CONTRIBUTOR)
State: open
Created: 2025-06-16T07:43:59Z

### Related Issues/PRs

<!-- Uncomment 'Resolve' if this PR can close the linked items. -->
<!-- Resolve --> #6152

---

### What changes are proposed in this pull request?

This PR adds an early validation step for folder-based datasets (like `audiofolder`) to prevent silent fallback behavior.

**Before this fix**:
- When `data_dir` or `data_files` were not provided, the loader defaulted to the current working directory.
- This caused unexpected behavior like:
  - Long loading times
  - Scanning unintended local files

**Now**:
- If both `data_dir` and `data_files` are missing, a `ValueError` is raised early with a helpful message.

---

### How is this PR tested?

- [x] Manual test via `load_dataset("audiofolder")` with missing `data_dir`
- [ ] Existing unit tests (should not break any)
- [ ] New tests (if needed, maintainers can guide)

---

### Does this PR require documentation update?

- [x] No. You can skip the rest of this section.

---

### Release Notes

#### Is this a user-facing change?

- [x] Yes. Give a description of this change to be included in the release notes for users.

> Adds early error handling for folder-based datasets when neither `data_dir` nor `data_files` is specified, avoiding unintended resolution to the current directory.

#### What component(s), interfaces, languages, and integrations does this PR affect?

Components:
- [x] `area/datasets`
- [x] `area/load`

---

<a name="release-note-category"></a>

#### How should the PR be classified in the release notes? Choose one:

- [x] `rn/bug-fix` - A user-facing bug fix worth mentioning in the release notes

---

#### Should this PR be included in the next patch release?

- [x] Yes (this PR will be cherry-picked and included in the next patch release)

Comments:
Great ! Since this logic is specific to one builder class maybe this check can be in the class definition ? I think you can put it in FolderBasedBuilder's `_info()` method.
Title: Unwanted column padding in nested lists of dicts
Author: qgallouedec (MEMBER)
State: closed
Created: 2025-06-15T22:06:17Z

```python
from datasets import Dataset

dataset = Dataset.from_dict({
    "messages": [
        [
            {"a": "...",},
            {"b": "...",},
        ],
    ]
})
print(dataset[0])
```

What I get:
```
{'messages': [{'a': '...', 'b': None}, {'a': None, 'b': '...'}]}
```

What I want:

```
{'messages': [{'a': '...'}, {'b': '...'}]}
```

Is there an easy way to automatically remove these auto-filled null/none values?

If not, I probably need a recursive none exclusion function, don't I?

Datasets 3.6.0

Comments:
Answer from @lhoestq:

> No
> This is because Arrow and Parquet a columnar format: they require a fixed type for each column. So if you have nested dicts, each item should have the same subfields

The way around I found is the handle it after sampling with this function:

```python
def remove_padding(example):
    if isinstance(example, list):
        return [remove_padding(value) if isinstance(value, (dict, list)) else value for value in example]
    elif isinstance(example, Mapping):
        return {
            key: remove_padding(value) if isinstance(value, (dict, list)) else value
            for key, value in example.items()
            if value is not None
        }
    else:
        raise TypeError("Input must be a list or a dictionary.")

# Example:
example = next(iter(dataset))
example = remove_padding(example)
```
Title: Lazy column
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-13T12:12:57Z

Same as https://github.com/huggingface/datasets/pull/7564 but for `Dataset`, cc @TopCoder2K FYI

e.g. `ds[col]` now returns a lazy Column instead of a list

This way calling `ds[col][idx]` only loads the required data in memory

(bonus: also supports subfields access with `ds[col][subcol][idx]`)

the breaking change will be for the next major release, which also includes removal of dataset scripts support

close https://github.com/huggingface/datasets/issues/4180

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7614). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: fix parallel push_to_hub in dataset_dict
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-13T09:02:24Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7613). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Code example for dataset.add_column() does not reflect correct way to use function
Author: shaily99 (NONE)
State: open
Created: 2025-06-12T19:42:29Z

https://github.com/huggingface/datasets/blame/38d4d0e11e22fdbc4acf373d2421d25abeb43439/src/datasets/arrow_dataset.py#L5925C10-L5925C10

The example seems to suggest that dataset.add_column() can add column inplace, however, this is wrong -- it cannot. It returns a new dataset with the column added to it.

Comments:
Title: i cant confirm email
Author: lykamspam (NONE)
State: open
Created: 2025-06-12T18:58:49Z

### Describe the bug

This is dificult, I cant confirm email because I'm not get any email!
I cant post forum because I cant confirm email!
I can send help desk because... no exist on web page.

paragraph 44

### Steps to reproduce the bug

rthjrtrt

### Expected behavior

ewtgfwetgf

### Environment info

sdgfswdegfwe

Comments:
Title: Update `_dill.py` to use `co_linetable` for Python 3.10+ in place of `co_lnotab`
Author: qgallouedec (MEMBER)
State: closed
Created: 2025-06-12T13:47:01Z

Not 100% about this one, but it seems to be recommended.

```
/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/datasets/utils/_dill.py:385: DeprecationWarning: co_lnotab is deprecated, use co_lines instead.
```
Tests pass locally. And the warning is gone with this change.

https://peps.python.org/pep-0626/#backwards-compatibility

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7609). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

not 100% sure either, I tried removing unnecessary checks - let me know if they sound good to you otherwise I'll revert

I can't reproduce the warning anymore... 🤦🏻‍♂️


Ah now I can reproduce!, and I can confirm that the warning is gone when you apply the change in this PR
Title: Tests typing and fixes for push_to_hub
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-11T17:13:52Z

todo:
- [x] fix TestPushToHub.test_push_dataset_dict_to_hub_iterable_num_proc

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7608). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Add `num_proc=` to `.push_to_hub()` (Dataset and IterableDataset)
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-10T14:35:10Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7606). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Make `push_to_hub` atomic (#7600)
Author: sharvil (NONE)
State: closed
Created: 2025-06-09T22:29:38Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7605). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi ! unfortunately we can't allow atomic commits for commits with hundreds of files additions (HF would time out)

Maybe an alternative would be to retry if there was a commit in between ? this could be the default behavior as well

Thanks for taking a look – much appreciated!

I've verified that commits with up to 20,000 files don't time out and the commit time scales linearly with the number of operations enqueued. It took just under 2 minutes to complete (successfully) the 20k file commit.

The fundamental issue I'm trying to tackle here is dataset corruption: getting into a state where a dataset on the hub cannot be used when downloaded. Non-atomic commits won't get us there, I think. If, for example, 3 of 5 commits complete and the machine/process calling `push_to_hub` has a network, hardware, or other failure that prevents it from completing the rest of the commits (even with retries) we'll now have some pointer files pointing to the new data and others pointing to the old data => corrupted. While this may seem like an unlikely scenario, it's a regular occurrence at scale.

If you still feel strongly that atomic commits are not the right way to go, I can either set it to not be the default or remove it entirely from this PR.

As for retries, it's a good idea. In a non-atomic world, the logic gets more complicated:
- keep an explicit queue of pending add/delete operations
- chunkwise pop from queue and commit with `parent_commit` set to previous chunked commit hash
- if `create_commit` fails:
  - re-fetch README and set `parent_commit` to latest hash for `revision`
  - re-generate dataset card content
  - swap old `CommitOperationAdd` with new one for README in the pending queue
- resume chunkwise committing from the queue as above

Entirely doable, but more involved than I signed up for with this PR.

Just to clarify – setting the `parent_commit` can be separated from making the commit atomic (which is what I'm suggesting by either atomic commits not the default or removing it from this PR). It's crucial to set the parent commit to avoid the read-modify-write race condition on the README schema.
Title: Docs and more methods for IterableDataset: push_to_hub, to_parquet...
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-09T16:44:40Z

to_csv, to_json, to_sql, to_pandas, to_polars, to_dict, to_list

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7604). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: No TF in win tests
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-09T13:56:34Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7603). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Enhance error handling and input validation across multiple modules
Author: mohiuddin-khan-shiam (NONE)
State: open
Created: 2025-06-08T23:01:06Z

This PR improves the robustness and user experience by:

1. **Audio Module**:
   - Added clear error messages when required fields ('path' or 'bytes') are missing in audio encoding

2. **DatasetDict**:
   - Enhanced key access error messages to show available splits when an invalid key is accessed

3. **NonMutableDict**:
   - Added input validation for the update() method to ensure proper mapping types

4. **Arrow Reader**:
   - Improved error messages for small dataset percentage splits with suggestions for alternatives

5. **FaissIndex**:
   - Strengthened input validation with descriptive error messages
   - Added proper type checking and shape validation for search queries

These changes make the code more maintainable and user-friendly by providing actionable feedback when issues arise.

Comments:
Title: `push_to_hub` is not concurrency safe (dataset schema corruption)
Author: sharvil (NONE)
State: closed
Created: 2025-06-07T17:28:56Z

### Describe the bug

Concurrent processes modifying and pushing a dataset can overwrite each others' dataset card, leaving the dataset unusable.

Consider this scenario:
- we have an Arrow dataset
- there are `N` configs of the dataset
- there are `N` independent processes operating on each of the individual configs (e.g. adding a column, `new_col`)
- each process calls `push_to_hub` on their particular config when they're done processing
- all calls to `push_to_hub` succeed
- the `README.md` now has some configs with `new_col` added and some with `new_col` missing

Any attempt to load a config (using `load_dataset`) where `new_col` is missing will fail because of a schema mismatch between `README.md` and the Arrow files. Fixing the dataset requires updating `README.md` by hand with the correct schema for the affected config. In effect, `push_to_hub` is doing a `git push --force` (I found this behavior quite surprising).

We have hit this issue every time we run processing jobs over our datasets and have to fix corrupted schemas by hand.

Reading through the code, it seems that specifying a [`parent_commit`](https://github.com/huggingface/huggingface_hub/blob/v0.32.4/src/huggingface_hub/hf_api.py#L4587) hash around here https://github.com/huggingface/datasets/blob/main/src/datasets/arrow_dataset.py#L5794 would get us to a normal, non-forced git push, and avoid schema corruption. I'm not familiar enough with the code to know how to determine the commit hash from which the in-memory dataset card was loaded.

### Steps to reproduce the bug

See above.

### Expected behavior

Concurrent edits to disjoint configs of a dataset should never corrupt the dataset schema.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.15.0-118-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- `huggingface_hub` version: 0.30.2
- PyArrow version: 19.0.1
- Pandas version: 2.2.2
- `fsspec` version: 2023.9.0

Comments:
@lhoestq can you please take a look? I've submitted a PR that fixes this issue. Thanks.

Thanks for the ping ! As I said in https://github.com/huggingface/datasets/pull/7605 there is maybe a more general approach using retries :)

Dropping this due to inactivity; we've implemented push_to_hub outside of HF datasets that's concurrency safe. Feel free to use the code I provided as a starting point if there's still interest in addressing this issue.
Title: My already working dataset (when uploaded few months ago) now is ignoring metadata.jsonl
Author: JuanCarlosMartinezSevilla (NONE)
State: closed
Created: 2025-06-06T18:59:00Z

### Describe the bug

Hi everyone, I uploaded my dataset https://huggingface.co/datasets/PRAIG/SMB a few months ago while I was waiting for a conference acceptance response. Without modifying anything in the dataset repository now the Dataset viewer is not rendering the metadata.jsonl annotations, neither it is being downloaded when using load_dataset. Can you please help? Thank you in advance.

### Steps to reproduce the bug

from datasets import load_dataset
ds = load_dataset("PRAIG/SMB")
ds = ds["train"]

### Expected behavior

 It is expected to have all the metadata available in the jsonl file. Fields like: "score_id", "original_width", "original_height", "regions"... among others.

### Environment info

datasets==3.6.0, python 3.13.3 (but he problem is already in the huggingface dataset page)

Comments:
Maybe its been a recent update, but i can manage to load the metadata.jsonl separately from the images with:

```
metadata = load_dataset("PRAIG/SMB", split="train", data_files=["*.jsonl"])
images = load_dataset("PRAIG/SMB", split="train")
```
Do you know it this is an expected behaviour? This makes my dataset viewer to only load the images without the labeling of metadata.jsonl.

Thanks

Hi ! this is because we now expect the metadata file to be inside the directory named after the split "train" (this way each split can have its own metadata and can be loaded independently)

You can fix that by configuring it explicitly in the dataset's README.md header:

```yaml
configs:
- config_name: default
  data_files:
  - split: train
    path:
    - "train/**/*.png"
    - "metadata.jsonl"
```

(or by moving the metadata.jsonl in train/ but in this case you also have to modify the content of the JSONL to fix the relative paths to the images)

Thank you very much, dataset viewer is already working as expected!!
Title: fix string_to_dict usage for windows
Author: lhoestq (MEMBER)
State: closed
Created: 2025-06-06T15:54:29Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7598). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Download datasets from a private hub in 2025
Author: DanielSchuhmacher (NONE)
State: closed
Created: 2025-06-06T07:55:19Z
Labels: enhancement

### Feature request

In the context of a private hub deployment, customers would like to use load_dataset() to load datasets from their hub, not from the public hub. This doesn't seem to be configurable at the moment and it would be nice to add this feature.

The obvious workaround is to clone the repo first and then load it from local storage, but this adds an extra step. It'd be great to have the same experience regardless of where the hub is hosted.

This issue was raised before here: https://github.com/huggingface/datasets/issues/3679
@juliensimon

### Motivation

none

### Your contribution

none

Comments:
Hi ! First, and in the general case, Hugging Face does offer to host private datasets, and with a subscription you can even choose the region in which the repositories are hosted (US, EU)

Then if you happen to have a private deployment, you can set the HF_ENDPOINT environment variable (same as in https://github.com/huggingface/transformers/issues/38634)

Thank you @lhoestq. Works as described!
Title: Add albumentations to use dataset
Author: ternaus (CONTRIBUTOR)
State: closed
Created: 2025-06-05T20:39:46Z

1. Fixed broken link to the list of transforms in torchvison.
2. Extended section about video image augmentations with an example from Albumentations.

Comments:
@lhoestq ping

@lhoestq ping

@lhoestq Thanks. Cleaned up torchvision.