yepengsun/github-issues huggingface dataset:
Adapt and test huggingface_hub v1.0.0.rc0

Test as part of https://github.com/huggingface/huggingface_hub/issues/3340

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7783). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
BIGPATENT dataset inaccessible (deprecated script loader)

dataset: https://huggingface.co/datasets/NortheasternUniversity/big_patent

When I try to load it with the datasets library, it fails with:

RuntimeError: Dataset scripts are no longer supported, but found big_patent.py


Could you please publish a Parquet/Arrow export of BIGPATENT on the Hugging Face so that it can be accessed with datasets>=4.x.



Hi ! I opened https://huggingface.co/datasets/NortheasternUniversity/big_patent/discussions/7 to update the dataset, hopefully it's merged soon !
[FIX] force spawning pool for MacOS

This PR gets multiprocessing to work on mac os:

```python
from datasets import load_dataset

ds = load_dataset("fka/awesome-chatgpt-prompts", split="train").take(100)

ds = ds.map(lambda x: x, num_proc=4)

ds.push_to_hub("burtenshaw/dataset-test", num_proc=4)
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7778). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

After more discussions on slack, we can switch the default to spawn.

Let's use multiprocess instead of multiprocessing and maybe add a check to apply this only on Macos ?
push_to_hub not overwriting but stuck in a loop when there are existing commits

### Describe the bug

`get_deletions_and_dataset_card` stuck at error a commit has happened error since push to hub for http error 412 for tag 4.1.0. The error does not exists in 4.0.0. 

### Steps to reproduce the bug

Create code to use push_to_hub, ran twice each time with different content for datasets.Dataset.
The code will stuck in time.sleep loop for `get_deletions_and_dataset_card`. If error is explicitly printed, the error is HTTP 412.

### Expected behavior

New datasets overwrite existing one on repo.

### Environment info

datasets 4.1.0

HTTP 412 means a commit happened in the meantime, so `get_deletions_and_dataset_card` has to retry to get the latest version of the dataset card and what files to delete based on the latest version of the dataset repository

Are you running other operations in the dataset repo for your push_to_hub ?

There was only a map() followed by a push_to_hub(). The repo had one prior commit also by using push_to_hub(). The error disappeared when I downgraded datasets to 4.0.0.

It is reproducible if you use finegrained token with Read+Write (Open pull request) access to only that repo.

Ah it was due to the use of requests_cache with POST methods, closing this.
Error processing scalar columns using tensorflow.

`datasets==4.0.0`
```
columns_to_return = ['input_ids','attention_mask', 'start_positions', 'end_positions']
train_ds.set_format(type='tf', columns=columns_to_return)
```
`train_ds`:
```
train_ds type: <class 'datasets.arrow_dataset.Dataset'>, shape: (1000, 9)
columns: ['question', 'sentences', 'answer', 'str_idx', 'end_idx', 'input_ids', 'attention_mask', 'start_positions', 'end_positions']
features:{'question': Value('string'), 'sentences': Value('string'), 'answer': Value('string'), 'str_idx': Value('int64'), 'end_idx': Value('int64'), 'input_ids': List(Value('int32')), 'attention_mask': List(Value('int8')), 'start_positions': Value('int64'), 'end_positions': Value('int64')}
```
`train_ds_tensor = train_ds['start_positions'].to_tensor(shape=(-1,1))` hits the following error:
```
AttributeError: 'Column' object has no attribute 'to_tensor'
```
`tf.reshape(train_ds['start_positions'], shape=[-1,1])` hits the following error:
```
TypeError: Scalar tensor has no `len()`
```
Add support for arrow iterable when concatenating or interleaving

Fixes a case when concatenating or interleaving datasets with `with_format(...)` call was slower.
Details here: https://github.com/huggingface/datasets/issues/6637

@lhoestq I tried to minimize the duplication between iter and iter_arrow methods, not sure if this is against the design, can separate those if needed.

Seeing the following numbers on the script shared in the original issue. (MacBook Pro M4)

```
1000it [00:00, 4074.63it/s] # ds_a.with_format("torch")
1000it [00:01, 593.39it/s] # ds_a.shuffle()
1999it [00:03, 594.09it/s] # datasets.interleave_datasets([ds_a, ds_b])
1000it [00:00, 5382.45it/s] # ds_a.shuffle().with_format("torch") <--- Was slow <2it/s
1999it [00:00, 4743.45it/s] # datasets.interleave_datasets([ds_a, ds_b]).with_format("torch") <--- Was slow <2it/s
1999it [00:20, 98.94it/s]  # torch.tensor(example["tensor"])
```


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7771). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq I've implemented the iteration on arrow as separate methods, can you take another look/trigger ci? 

@lhoestq Any idea why the integration tests are failing, is this expected? Anything I can do on my side?

They seem unrelated to your changes. Merging :)
Fix: Correct float feature generation in `generate_examples` 

This PR fixes a bug in the `generate_examples` function where `datasets.Value` features with a `float` dtype were incorrectly generated using `np.random.randint`. This resulted in integer values being cast to float, which is not representative of true floating-point data.


**Key changes include:**

* Added explicit handling for `float` features using `np.random.rand` to generate continuous values.
* Introduced fail-fast type checks for unsupported dtypes to improve robustness.
* Added validation for sequence features to ensure `seq_shapes` is provided.

### Before Fix

Float features were generated incorrectly as integers cast to float:

```text
- Example 0:
- int_feature: 0
- float_feature: 9.0  <-- Incorrect: An integer disguised as a float
- string_feature: The small grey turtle was surprisingly fast...
- seq_feature: [0.3048 0.4291 0.4283]
```

### After Fix

Float features are now correctly generated as continuous numbers in the range [0, 1):
```text
+ Example 0:
+ int_feature: 0
+ float_feature: 0.0183  <-- Correct: A true random float
+ string_feature: The small grey turtle was surprisingly fast...
+ seq_feature: [0.9237 0.7972 0.8526]
```
#### Note: This PR is a follow-up/fix of the previously closed PR #7769 for clarity and context.


Hey @lhoestq, just following up on this — it fixes float feature generation in `generate_examples`. Thanks!
Custom `dl_manager` in `load_dataset`

### Feature request
https://github.com/huggingface/datasets/blob/4.0.0/src/datasets/load.py#L1411-L1418

```

def load_dataset(
    ...
    dl_manager: Optional[DownloadManager] = None,  # add this new argument
    **config_kwargs,
) -> Union[DatasetDict, Dataset, IterableDatasetDict, IterableDataset]:

    ...

    # Create a dataset builder
    builder_instance = load_dataset_builder(
        path=path,
        name=name,
        data_dir=data_dir,
        data_files=data_files,
        cache_dir=cache_dir,
        features=features,
        download_config=download_config,
        download_mode=download_mode,
        revision=revision,
        token=token,
        storage_options=storage_options,
        **config_kwargs,
    )

    # Return iterable dataset in case of streaming
    if streaming:
        return builder_instance.as_streaming_dataset(split=split)

    # Note: This is the revised part
    if dl_manager is None:
        if download_config is None:
            download_config = DownloadConfig(
                cache_dir=builder_instance._cache_downloaded_dir,
                force_download=download_mode == DownloadMode.FORCE_REDOWNLOAD,
                force_extract=download_mode == DownloadMode.FORCE_REDOWNLOAD,
                use_etag=False,
                num_proc=num_proc,
                token=builder_instance.token,
                storage_options=builder_instance.storage_options,
            )  # We don't use etag for data files to speed up the process

        dl_manager = DownloadManager(
            dataset_name=builder_instance.dataset_name,
            download_config=download_config,
            data_dir=builder_instance.config.data_dir,
            record_checksums=(
                builder_instance._record_infos or verification_mode == VerificationMode.ALL_CHECKS
            ),
        )

    # Download and prepare data
    builder_instance.download_and_prepare(
        download_config=download_config,
        download_mode=download_mode,
        verification_mode=verification_mode,
        dl_manager=dl_manager,  # pass the new argument
        num_proc=num_proc,
        storage_options=storage_options,
    )

    ...

```

### Motivation

In my case, I'm hoping to deal with the cache files downloading manually (not using hash filenames and save to another location, or using potential existing local files).

### Your contribution

It's already implemented above. If maintainers think this should be considered, I'll open a PR.
cast columns to Image/Audio/Video with `storage_options`

### Feature request

Allow `storage_options` to be passed in
1. `cast` related operations (e.g., `cast_columns, cast`)
2. `info` related reading (e.g., `from_dict, from_pandas, from_polars`) together with `info.features`
```python3
import datasets

image_path = "s3://bucket/sample.png"
dataset = datasets.Dataset.from_dict({"image_path": [image_path]})

# dataset = dataset.cast_column("image_path", datasets.Image())  # now works without `storage_options`

# expected behavior
dataset = dataset.cast_column("image_path", datasets.Image(), storage_options={"anon": True})
```

### Motivation

I'm using my own registered fsspec filesystem (s3 with customized local cache support). I need to pass cache folder paths `cache_dirs: list[str]` to the filesystem when I read the remote images (cast from file_paths).

### Your contribution

Could help with a PR at weekends
polars dataset cannot cast column to Image/Audio/Video

### Describe the bug

`from_polars` dataset cannot cast column to Image/Audio/Video, while it works on `from_pandas` and `from_dict`

### Steps to reproduce the bug

```python3
import datasets
import pandas as pd
import polars as pl

image_path = "./sample.png"

# polars
df = pl.DataFrame({"image_path": [image_path]})
dataset = datasets.Dataset.from_polars(df)
dataset = dataset.cast_column("image_path", datasets.Image())

# # raises Error
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from large_string to struct using function cast_struct


# pandas 
df = pd.DataFrame({"image_path": [image_path]})
dataset = datasets.Dataset.from_pandas(df)
dataset = dataset.cast_column("image_path", datasets.Image())

# # pass
{'image_path': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=338x277 at 0x7FBA719D4050>}


# dict
dataset = datasets.Dataset.from_dict({"image_path": [image_path]})
dataset = dataset.cast_column("image_path", datasets.Image())

# # pass
{'image_path': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=338x277 at 0x7FBA719D4050>}
```


### Expected behavior

`from_polars` case shouldn't raise error and have the same outputs as `from_pandas` and `from_dict`

### Environment info

```
# Name                    Version                   Build  Channel
datasets                  4.0.0                    pypi_0    pypi
pandas                    2.3.1                    pypi_0    pypi
polars                    1.32.3                   pypi_0    pypi
```

I fixed this with a combination of `to_dict` and `from_dict`:

```py
datasets.Dataset.from_dict(df.to_dict(as_series=False))
```

@samuelstevens Yeah, I'm using similar workaround as well. But it would be ideal if we can avoid the copy.
Bump dill to 0.4.0

This bumps `dill` to 0.3.9 and closes #7510 

It turns out the only thing required to make the tests pass was to extend the version checks to include 0.3.9.

Have you tried to run `pytest tests/test_fingerprint.py` ? It seems dill 0.3.9 breaks a lot of tests

```
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_regex - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer_with_cache - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ignores_line_definition_of_function - AssertionError: 'c48ebfacf8768f50' != '27e49d047c02c83b'
FAILED tests/test_fingerprint.py::RecurseHashTest::test_hash_ipython_function - AssertionError: '65edc6b6d425a8e9' != '9f364fe298fb286a'
FAILED tests/test_fingerprint.py::HashingTest::test_hash_tiktoken_encoding - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_compiled_module - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_generator - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_hash_torch_tensor - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_doesnt_depend_on_order - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::HashingTest::test_set_stable - NameError: name 'log' is not defined
FAILED tests/test_fingerprint.py::test_move_script_doesnt_change_hash - AssertionError: assert b'93072ca404a697db\n' == b'cf89a7e497a97e32\n'
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7763). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @lhoestq! Yes, I did. It's not really `dill` that breaks things. Rather the shims that `datasets` has in place did not include the next version. 

FYI: I also tested it with `dill-0.4.0` and the changes would need to be analogous, but I wanted to be conservative in this PR. 

The NameError is fixed in your PR since it defines the right `log()` function for 0.3.9.

But I'm less sure about the AssertionError that may be related to deterministic hashing or ipython/shell function hashing. We would need to solve these

EDIT: ah actually it does ! cool ! let me update the branch and re-run the CI
Parquet: use data page v2 for efficient page pruning

This is needed to enable page pruning with DataFusion, which will be useful for the Dataset Viewer.

Indeed page pruning with DataFusion allows to download only certain pages of a row group, reducing the I/O required to read just a few rows.

But while data page v1 generally works, it's not easy with DataFusion to do page pruning on datasets with nested data. This is because rows can span multiple pages in v1, contrary to v2.

cc @severo for viz

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7762). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing this since it looks like the page offset index is enough :)
Option for Anonymous Dataset link

### Feature request

Allow for anonymized viewing of datasets. For instance, something similar to [Anonymous GitHub](https://anonymous.4open.science/).

### Motivation

We generally publish our data through Hugging Face. This has worked out very well as it's both our repository and archive (thanks to the DOI feature!). However, we have an increasing challenge when it comes to sharing our datasets for paper (both conference and journal) submissions. Due to the need to share data anonymously, we can't use the Hugging Face URLs, but datasets tend to be too large for inclusion as a zip. Being able to have an anonymous link would be great since we can't be double-publishing the data.

### Your contribution

Sorry, I don't have a contribution to make to the implementation of this. Perhaps it would be possible to work off the [Anonymous GitHub](https://github.com/tdurieux/anonymous_github) code to generate something analogous with pointers to the data still on Hugging Face's servers (instead of the duplication of data required for the GitHub version)?
Add support for `.conll` file format in datasets

### Feature request

I’d like to request native support in the Hugging Face datasets library for reading .conll files (CoNLL format). This format is widely used in NLP tasks, especially for Named Entity Recognition (NER), POS tagging, and other token classification problems.

Right now `.conll` datasets need to be manually parsed or preprocessed before being loaded into datasets. Having built in support would save time and make workflows smoother for researchers and practitioners.

I propose - 
Add a conll dataset builder or file parser to datasets that can:
- Read `.conll` files with customizable delimiters (space, tab).
- Handle sentence/document boundaries (typically indicated by empty lines).
- Support common CoNLL variants (e.g., CoNLL-2000 chunking, CoNLL-2003 NER).
- Output a dataset where each example contains:
- tokens: list of strings
- tags (or similar): list of labels aligned with tokens

Given a .conll snippet like:
```
EU NNP B-ORG
rejects VBZ O
German JJ B-MISC
call NN O
. . O
```


The dataset should load as:
```
{
  "tokens": ["EU", "rejects", "German", "call", "."],
  "tags": ["B-ORG", "O", "B-MISC", "O", "O"]
}
```




### Motivation

- CoNLL files are a standard benchmark format in NLP (e.g., CoNLL-2003, CoNLL-2000).
- Many users train NER or sequence labeling models (like BERT for token classification) directly on `.conll` 
- Right now you have to write your own parsing scripts. Built in support would unify this process and would be much more convenient 

### Your contribution

I’d be happy to contribute by implementing this feature. My plan is to- 
- Add a new dataset script (conll.py) to handle .conll files.
- Implement parsing logic that supports sentence/document boundaries and token-label alignment.
- Write unit tests with small `.conll` examples to ensure correctness.
- Add documentation and usage examples so new users can easily load `.conll` datasets.

This would be my first open source contribution, so I’ll follow the `CONTRIBUTING.md` guidelines closely and adjust based on feedback from the maintainers.

That would be cool ! feel free to ping me if I can help reviewing a PR
datasets.map(f, num_proc=N) hangs with N>1 when run on import

### Describe the bug

If you `import` a module that runs `datasets.map(f, num_proc=N)` at the top-level, Python hangs.


### Steps to reproduce the bug

1. Create a file that runs datasets.map at the top-level:

```bash
cat <<EOF > import_me.py
import datasets

the_dataset = datasets.load_dataset("openai/openai_humaneval")
the_dataset = the_dataset.map(lambda item: item, num_proc=2)
EOF
```

2. Start Python REPL:

```bash
uv run --python 3.12.3 --with "datasets==4.0.0"  python3
Python 3.12.3 (main, Aug 14 2025, 17:47:21) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
```

3. Import the file:

```python
import import_me
````

Observe hang.

### Expected behavior

Ideally would not hang, or would fallback to num_proc=1 with a warning.

### Environment info


- `datasets` version: 4.0.0
- Platform: Linux-6.14.0-29-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 0.34.4
- PyArrow version: 21.0.0
- Pandas version: 2.3.2
- `fsspec` version: 2025.3.0
Fix: Update Dill Version in Setup py

Currently the DIll version is less than 3.9 and now major libraries like Multiprocess, gepa requires Dill version as 0.4.0 and this is making a conflict in installation. So added this small PR to update the DIll. 

https://github.com/huggingface/datasets/issues/7751

same as https://github.com/huggingface/datasets/pull/7763: some tests need to be fixed to support 0.4.0
Dill version update

### Describe the bug

Why the datasets is not updating the dill ? 
Just want to know if I update the dill version in dill what will be the repucssion. 
For now in multiplaces I have to update the library like process requirequire dill 0.4.0 so why not datasets. 
Adding a pr too. 

### Steps to reproduce the bug

.

### Expected behavior

.

### Environment info

.

#7752 

related: #7510
Add wikipedia-2023-redirects dataset

Title: Add wikipedia-2023-redirects dataset (redirect resolution + pageviews)

Summary
- New dataset loader: wikipedia_2023_redirects
- Canonical Wikipedia pages enriched with:
  - redirects (aliases pointing to the page)
  - 2023 pageviews (aggregated)
- Streaming support; robust parsing; license notes included
- Tests with tiny dummy data (XML + TSVs); covers streaming

Motivation
RAG/retrieval often benefits from:
- Query expansion via redirect aliases
- Popularity prior via pageviews
This loader offers a practical, maintenance-light way to access canonical pages alongside their redirect aliases and 2023 pageview totals.

Features
- id: string
- title: string
- url: string
- text: string
- redirects: list[string]
- pageviews_2023: int32
- timestamp: string

Licensing
- Wikipedia text: CC BY-SA 3.0 (attribution and share-alike apply)
- Pageviews: public domain
The PR docs mention both, and the module docstring cites sources.

Notes
- The URLs in _get_urls_for_config are wired to dummy files for tests. In production, these would point to Wikimedia dumps:
  - XML page dumps: https://dumps.wikimedia.org/
  - Pageviews: https://dumps.wikimedia.org/other/pageviews/
- The schema is intentionally simple and stable. Pageview aggregation is per-title sum across 2023.

Testing
- make style && make quality
- pytest -q tests/test_dataset_wikipedia_2023_redirects.py

Example
```python
from datasets import load_dataset
ds = load_dataset("wikipedia_2023_redirects", split="train")
print(ds[0]["title"], ds[0]["redirects"][:5], ds[0]["pageviews_2023"])
```

Acknowledgements
- Wikipedia/Wikimedia Foundation for the source data
- Hugging Face Datasets for the dataset infrastructure


you should host this dataset on HF with `ds.push_to_hub()` ! we stopped using dataset scripts some time ago
Fix: Canonical 'multi_news' dataset is broken and should be updated to a Parquet version

Hi, 

The canonical `multi_news` dataset is currently broken and fails to load. This is because it points to the [alexfabri/multi_news](https://huggingface.co/datasets/alexfabbri/multi_news) repository, which contains a legacy loading script (`multi_news.py`) that requires the now-removed `trust_remote_code` parameter.

The original maintainer's GitHub and Hugging Face repositories appear to be inactive, so a community-led fix is needed.

I have created a working fix by converting the dataset to the modern Parquet format, which does not require a loading script. The fixed version is available here and loads correctly:

**[Awesome075/multi_news_parquet](https://huggingface.co/datasets/Awesome075/multi_news_parquet)**

Could the maintainers please guide me or themselves update the official `multi_news` dataset to use this working Parquet version? This would involve updating the canonical pointer for "multi_news" to resolve to the new repository.

This action would fix the dataset for all users and ensure its continued availability.

Thank you!

@sayakpaul @a-r-r-o-w could you verify this issue then i can contribute to solve this issue!😊
Audio mono argument no longer supported, despite class documentation

### Describe the bug

Either update the documentation, or re-introduce the flag (and corresponding logic to convert the audio to mono)

### Steps to reproduce the bug

Audio(sampling_rate=16000, mono=True) raises the error 
TypeError: Audio.__init__() got an unexpected keyword argument 'mono'

However, in the class documentation, is says:
Args:
        sampling_rate (`int`, *optional*):
            Target sampling rate. If `None`, the native sampling rate is used.
        mono (`bool`, defaults to `True`):
            Whether to convert the audio signal to mono by averaging samples across
            channels.
        [...]

### Expected behavior

The above call should either work, or the documentation within the Audio class should be updated

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-5.15.0-124-generic-x86_64-with-glibc2.35
- Python version: 3.12.11
- `huggingface_hub` version: 0.34.4
- PyArrow version: 21.0.0
- Pandas version: 2.3.2
- `fsspec` version: 2025.3.0

I want to solve this problem can you please assign it to me
and also can you please guide whether the mono parameter is required to be re-added or the documentation needs an update?
Refactor HDF5 and preserve tree structure

Closes #7741. Followup to #7690

- Recursive parsing and feature inference, to preserve the tree structure of the file. Note this means we now visit all links in the file. It also means we have to call` combine_chunks` on any large non-root datasets.
- Support for `complex64` (two `float32`s, used to be converted to two `float64`s)
- Support for ndim complex, compound, more field types for compound (due to reusing the main parser, compound types are treated like groups)
- Cleaned up varlen support
- Always do feature inference and always cast to features (used to cast to schema)
- Updated tests to use `load_dataset` instead of internal APIs
- Removed `columns` in config. Have to give Features (i.e., must specify types) if filtering

@lhoestq this is ready for you now!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7743). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
module 'pyarrow' has no attribute 'PyExtensionType'

### Describe the bug

When importing certain libraries, users will encounter the following error which can be traced back to the datasets library.

module 'pyarrow' has no attribute 'PyExtensionType'.

Example issue: https://github.com/explodinggradients/ragas/issues/2170

The issue occurs due to the following. I will proceed to submit a PR with the below fix:

**Issue Reason**
The issue is that PyArrow version 21.0.0 doesn’t have PyExtensionType. This was changed in newer versions of PyArrow. The

PyExtensionType class was renamed to ExtensionType in PyArrow 13.0.0 and later versions.


** Issue Solution**
Making the following changes to the following lib files should temporarily resolve the issue.

I will submit a PR to the dataets library in the meantime.

env_name/lib/python3.10/site-packages/datasets/features/features.py:

```
> 521            self.shape = tuple(shape)
       522            self.value_type = dtype
       523            self.storage_dtype = self._generate_dtype(self.value_type)
       524 -          pa.PyExtensionType.__init__(self, self.storage_dtype)
       524 +          pa.ExtensionType.__init__(self, self.storage_dtype)
       525
       526        def __reduce__(self):
       527            return self.__class__, (
```

Updated venv_name/lib/python3.10/site-packages/datasets/features/features.py:
```      
 510        _type: str = field(default=“Array5D”, init=False, repr=False)
       511
       512
       513 -  class _ArrayXDExtensionType(pa.PyExtensionType):
       513 +  class _ArrayXDExtensionType(pa.ExtensionType):
       514        ndims: Optional[int] = None
       515
       516        def __init__(self, shape: tuple, dtype: str):
```

### Steps to reproduce the bug

Ragas version: 0.3.1
Python version: 3.11

**Code to Reproduce**

_**In notebook:**_

!pip install ragas
from ragas import evaluate


### Expected behavior

The required package installs without issue.

### Environment info

In Jupyter Notebook.
venv

Just checked out the files and thishad already been addressed

For others who find this issue: 

`pip install --upgrade "datasets>=2.20.0"` 

from https://github.com/explodinggradients/ragas/issues/2170#issuecomment-3204393672 can fix it.
Preserve tree structure when loading HDF5

### Feature request

https://github.com/huggingface/datasets/pull/7740#discussion_r2285605374

### Motivation

`datasets` has the `Features` class for representing nested features. HDF5 files have groups of datasets which are nested, though in #7690 the keys are flattened. We should preserve that structure for the user.

### Your contribution

I'll open a PR (#7743)
Replacement of "Sequence" feature with "List" breaks backward compatibility

PR #7634 replaced the Sequence feature with List in 4.0.0, so datasets saved with version 4.0.0 with that feature cannot be loaded by earlier versions. There is no clear option in 4.0.0 to use the legacy feature type to preserve backward compatibility.

Why is this a problem? I have a complex preprocessing and training pipeline dependent on 3.6.0; we manage a very large number of separate datasets that get concatenated during training. If just one of those datasets is saved with 4.0.0, they become unusable, and we have no way of "fixing" them. I can load them in 4.0.0 but I can't re-save with the legacy feature type, and I can't load it in 3.6.0 for obvious reasons.

Perhaps I'm missing something here, since the PR says that backward compatibility is preserved; if so, it's not obvious to me how.

Backward compatibility here means 4.0.0 can load datasets saved with older versions.

You will need 4.0.0 to load datasets saved with 4.0.0
Allow saving multi-dimensional ndarray with dynamic shapes

### Feature request

I propose adding a dedicated feature to the datasets library that allows for the efficient storage and retrieval of multi-dimensional ndarray with dynamic shapes. Similar to how Image columns handle variable-sized images, this feature would provide a structured way to store array data where the dimensions are not fixed.

A possible implementation could be a new Array or Tensor feature type that stores the data in a structured format, for example, 

```python
{
  "shape": (5, 224, 224), 
  "dtype": "uint8", 
  "data": [...]
}
```

This would allow the datasets library to handle heterogeneous array sizes within a single column without requiring a fixed shape definition in the feature schema.

### Motivation

I am currently trying to upload data from astronomical telescopes, specifically FITS files, to the Hugging Face Hub. This type of data is very similar to images but often has more than three dimensions. For example, data from the SDSS project contains five channels (u, g, r, i, z), and the pixel values can exceed 255, making the Pillow based Image feature unsuitable.

The current datasets library requires a fixed shape to be defined in the feature schema for multi-dimensional arrays, which is a major roadblock. This prevents me from saving my data, as the dimensions of the arrays can vary across different FITS files.

https://github.com/huggingface/datasets/blob/985c9bee6bfc345787a8b9dd316e1d4f3b930503/src/datasets/features/features.py#L613-L614

A feature that supports dynamic shapes would be incredibly beneficial for the astronomy community and other fields dealing with similar high-dimensional, variable-sized data (e.g., medical imaging, scientific simulations).

### Your contribution

I am willing to create a PR to help implement this feature if the proposal is accepted.

I agree this would be super valuable.

It looks like this was discussed a few years ago in https://github.com/huggingface/datasets/issues/5272#issuecomment-1550200824 but there were some issues. Those PRs are merged now and it looks like Arrow [officially supports](https://arrow.apache.org/docs/format/CanonicalExtensions.html#variable-shape-tensor) this so it's a good time to re-evaluate!

Happy to help with this, maybe we can think of adding a new type `Tensor` (instead of Array2D, 3D etc. which imply a fixed number of dims - we can keep them for backward compat anyways) that uses VariableShapeTensor (or FixedShapeTensor if the shape is provided maybe ? happy to discuss this)
docs: Add column overwrite example to batch mapping guide

This PR adds a complementary example showing the **column-overwriting** pattern, which is both more direct and more flexible for many transformations.

### Proposed Change

The original `remove_columns` example remains untouched. Below it, this PR introduces an alternative approach that overwrites an existing column during batch mapping.

This teaches users a core `.map()` capability for in-place transformations without extra intermediate steps.

**New Example:**

> ```python
> >>> from datasets import Dataset
> >>> dataset = Dataset.from_dict({"a": [0, 1, 2]})
> # Overwrite "a" directly to duplicate each value
> >>> duplicated_dataset = dataset.map(
> ...     lambda batch: {"a": [x for x in batch["a"] for _ in range(2)]},
> ...     batched=True
> ... )
> >>> duplicated_dataset
> Dataset({
>     features: ['a'],
>     num_rows: 6
> })
> >>> duplicated_dataset["a"]
> [0, 0, 1, 1, 2, 2]
> ```


Hi @lhoestq, just a gentle follow-up on this PR.
Fixing __getitem__ of datasets which behaves inconsistent to documentation when setting _format_type to None

Setting _format_type to None, should return plain python object but as of 4.0.0 returns Column. This fails in libs such as sentencetransformers (such as in generation of hard negatives) where plain python is expected.

this breaking change is actually expected, happy to help with a fix in sentencetransformers to account for this

Thank you for the context. I thought this was a mismatch do the documentation. Good to know it was intentional. No worries, can add a PR to sentence transformers.
Add the possibility of a backend for audio decoding

### Feature request

Add the possibility of a backend for audio decoding. Before version 4.0.0, soundfile was used, and now torchcodec is used, but the problem is that torchcodec requires ffmpeg, which is problematic to install on the same colab. Therefore, I suggest adding a decoder selection when loading the dataset.

### Motivation

I use a service for training models in which ffmpeg cannot be installed.

### Your contribution

I use a service for training models in which ffmpeg cannot be installed.

is there a work around im stuck

never mind just downgraded
OSError: libcudart.so.11.0: cannot open shared object file: No such file or directory

> Hi  is there any solution for that eror i try to install this one 
pip install torch==1.12.1+cpu torchaudio==0.12.1+cpu -f https://download.pytorch.org/whl/torch_stable.html  
this is working fine but tell me how to install pytorch version that is fit for gpu
NonMatchingSplitsSizesError and ExpectedMoreSplitsError

### Describe the bug

When loading dataset, the info specified by `data_files` did not overwrite the original info.


### Steps to reproduce the bug

```python
from datasets import load_dataset

traindata = load_dataset(
            "allenai/c4",
            "en",
            data_files={"train": "en/c4-train.00000-of-01024.json.gz", 
                        "validation": "en/c4-validation.00000-of-00008.json.gz"},
        )
```


```log
NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=828589180707, num_examples=364868892, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='train', num_bytes=809262831, num_examples=356317, shard_lengths=[223006, 133311], dataset_name='c4')}, {'expected': SplitInfo(name='validation', num_bytes=825767266, num_examples=364608, shard_lengths=None, dataset_name=None), 'recorded': SplitInfo(name='validation', num_bytes=102199431, num_examples=45576, shard_lengths=None, dataset_name='c4')}]
```

```python
from datasets import load_dataset

traindata = load_dataset(
            "allenai/c4",
            "en",
            data_files={"train": "en/c4-train.00000-of-01024.json.gz"},
            split="train"
        )
```

```log
ExpectedMoreSplitsError: {'validation'}
```

### Expected behavior

No error

### Environment info

datasets 4.0.0
config paths that start with ./ are not valid as hf:// accessed repos, but are valid when accessed locally

### Describe the bug

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - images/xyz/*.jpg
```
will correctly download but

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - ./images/xyz/*.jpg
```
will error with `FileNotFoundError` due to improper url joining. `load_dataset` on the same directory locally works fine.

### Steps to reproduce the bug

1. create a README.md with the front matter of the form

```
- config_name: some_config
  data_files:
  - split: train
    path:
    - ./images/xyz/*.jpg
```

2. `touch ./images/xyz/1.jpg`
3. Observe this directory loads with `load_dataset("filesystem_path", "some_config")` correctly.
4. Observe exceptions when you load this with `load_dataset("repoid/filesystem_path", "some_config")`

### Expected behavior

`./` prefix should be interpreted correctly

### Environment info

datasets 4.0.0
datasets 3.4.0

reproduce
fix(webdataset): don't .lower() field_name

This fixes cases where keys have upper case identifiers

fixes: https://github.com/huggingface/datasets/issues/7732

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7726). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

CI failures are unrelated, merging :)
Don't remove `trust_remote_code` arg!!!

### Feature request

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!

Add `trust_remote_code` arg back please!

### Motivation

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!


### Your contribution

defaulting it to False is nice balance. we need manully setting it to True in certain scenarios!
Bad split error message when using percentages

### Describe the bug

Hi, I'm trying to download a dataset. To not load the entire dataset in memory, I split it as described [here](https://huggingface.co/docs/datasets/v4.0.0/loading#slice-splits) in 10% steps.

When doing so, the library returns this error:
    raise ValueError(f"Bad split: {split}. Available splits: {list(splits_generators)}")
ValueError: Bad split: train[0%:10%]. Available splits: ['train']

Edit: Same happens with a split like _train[:90000]_
### Steps to reproduce the bug

```
  for split in range(10):
    split_str = f"train[{split*10}%:{(split+1)*10}%]"
    print(f"Processing split {split_str}...")
    ds = load_dataset("user/dataset", split=split_str, streaming=True)
```


### Expected behavior

I'd expect the library to split my dataset in 10% steps.

### Environment info

python 3.12.11
ubuntu 24
dataset 4.0.0

I'd like to work on this: add clearer validation/messages for percent-based splits + tests

The most basic example is this code:
`load_dataset("openslr/librispeech_asr", split="train[10%:20%]")`

This results in this ValueError:
```
    raise ValueError(f'Unknown split "{split}". Should be one of {list(name2len)}.')
ValueError: Unknown split "train". Should be one of ['test.clean', 'test.other', 'train.clean.100', 'train.clean.360', 'train.other.500', 'validation.clean', 'validation.other'].
```
Datasets 4.0 map function causing column not found

### Describe the bug

Column returned after mapping is not found in new instance of the dataset.

### Steps to reproduce the bug

Code for reproduction. After running get_total_audio_length, it is errored out due to `data` not having `duration`
```
def compute_duration(x):
    return {"duration": len(x["audio"]["array"]) / x["audio"]["sampling_rate"]}


def get_total_audio_length(dataset):
    data = dataset.map(compute_duration, num_proc=NUM_PROC)
    print(data)
    durations=data["duration"]
    total_seconds = sum(durations)
    return total_seconds
```

### Expected behavior

New datasets.Dataset instance should have new columns attached.

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-5.4.0-124-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.33.2
- PyArrow version: 20.0.0
- Pandas version: 2.3.0
- `fsspec` version: 2023.12.2

Hi, I tried to reproduce this issue on the latest `main` branch but it seems to be working correctly now. My test script (which creates a dummy dataset and applies the `.map()` function) successfully creates and accesses the new column without a `KeyError`.

It's possible this was fixed by a recent commit. The maintainers might want to consider closing this issue.

Hi, have you tried on a large dataset (200GB+) perhaps? I will try my best to do a rerun with main branch when I have the time.

I ran it on a small dataset, maybe that’s why I didn’t hit the issue. If it still shows up on your side with the latest main, let me know. I can try it on a bigger set too.
Specify dataset columns types in typehint

### Feature request

Make dataset optionaly generic to datasets usage with type annotations like it was done in `torch.Dataloader` https://github.com/pytorch/pytorch/blob/134179474539648ba7dee1317959529fbd0e7f89/torch/utils/data/dataloader.py#L131

### Motivation

In MTEB we're using a lot of datasets objects, but they're a bit poor in typehints. E.g. we can specify this for dataloder

```python
from typing import TypedDict
from torch.utils.data import DataLoader

class CorpusInput(TypedDict):
    title: list[str]
    body: list[str]


class QueryInput(TypedDict):
    query: list[str]
    instruction: list[str]

def queries_loader() -> DataLoader[QueryInput]:
    ...

def corpus_loader() -> DataLoader[CorpusInput]:
   ...
```
But for datasets we can only specify columns in type in comments 
```python
from datasets import Dataset

QueryDataset = Dataset
"""Query dataset should have `query` and `instructions` columns as `str` """
```

### Your contribution

I can create draft implementation
add support for pyarrow string view in features


@lhoestq who do you think would be the best to have a look at this? Any pointers would be appreciated, thanks!

Hi ! what's the rationale for supporting string view ? I'm afraid it can complexify the typing logic without much value

Hi @lhoestq ! I mainly want to be able to create features by using `Features.from_arrow_schema(dataset_schema)` on an arrow dataset with string view columns, currently there is no easy way to do this, and string_view is becoming an increasingly common data type for string columns in arrow. Thanks for having a look!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7718). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Cached dataset is not used when explicitly passing the cache_dir parameter

### Describe the bug

Hi, we are pre-downloading a dataset using snapshot_download(). When loading this exact dataset with load_dataset() the cached snapshot is not used. In both calls, I provide the cache_dir parameter.

### Steps to reproduce the bug

```
from datasets import load_dataset, concatenate_datasets
from huggingface_hub import snapshot_download
    
def download_ds(name: str):
    snapshot_download(repo_id=name, repo_type="dataset", cache_dir="G:/Datasets/cache")

def prepare_ds():
    audio_ds = load_dataset("openslr/librispeech_asr", num_proc=4, cache_dir="G:/Datasets/cache")
    print(sfw_ds.features)

if __name__ == '__main__':
    download_ds("openslr/librispeech_asr")
    prepare_ds()
```

### Expected behavior

I'd expect that the cached version of the dataset is used. Instead, the same dataset is downloaded again to the default cache directory.

### Environment info

Windows 11
datasets==4.0.0
Python 3.12.11

Hi, I've investigated this issue and can confirm the bug. Here are my findings:

**1. Reproduction:**
I was able to reproduce the issue on the latest `main` branch. Using the provided code snippet, `snapshot_download` correctly populates the custom `cache_dir`, but `load_dataset` with the same `cache_dir` triggers a full re-download and re-processing of the dataset, ignoring the existing cache.

**2. Investigation:**
I traced the `cache_dir` parameter from `load_dataset` down to the `DatasetBuilder` class in `src/datasets/builder.py`. The root cause seems to be a mismatch between the cache path structure created by `snapshot_download` and the path structure expected by the `DatasetBuilder`.

Specifically, the `_relative_data_dir` method in `DatasetBuilder` constructs a path using `namespace___dataset_name` (with three underscores), while the cache from `snapshot_download` appears to use a `repo_id` based format like `datasets--namespace--dataset_name` (with double hyphens).

**3. Attempted Fix & Result:**
I attempted a fix by modifying the `_relative_data_dir` method to replace the path separator "/" in `self.repo_id` with "--", to align it with the `snapshot_download` structure.

This partially worked: `load_dataset` no longer re-downloads the files. However, it still re-processes them every time (triggering "Generating train split...", etc.) instead of loading the already processed Arrow files from the cache.

This suggests the issue is deeper than just the directory name and might be related to how the builder verifies the integrity or presence of the processed cache files.

I hope these findings are helpful for whoever picks up this issue.
Concurrent IterableDataset push_to_hub

Same as https://github.com/huggingface/datasets/pull/7708 but for `IterableDataset`

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7710). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Release 4.0.0 breaks usage patterns of with_format

### Describe the bug

Previously it was possible to access a whole column that was e.g. in numpy format via `with_format` by indexing the column. Now this possibility seems to be gone with the new Column() class. As far as I see, this makes working on a whole column (in-memory) more complex, i.e. normalizing an in-memory dataset for which iterating would be too slow. Is this intended behaviour? I couldn't find much documentation on the intended usage of the new Column class yet.

### Steps to reproduce the bug

Steps to reproduce:
```
from datasets import load_dataset
dataset = load_dataset("lhoestq/demo1")
dataset = dataset.with_format("numpy")
print(dataset["star"].ndim)
```

### Expected behavior

Working on whole columns should be possible.

### Environment info

- `datasets` version: 4.0.0
- Platform: Linux-6.8.0-63-generic-x86_64-with-glibc2.36
- Python version: 3.12.11
- `huggingface_hub` version: 0.34.3
- PyArrow version: 21.0.0
- Pandas version: 2.3.1
- `fsspec` version: 2025.3.0

This is a breaking change with 4.0 which introduced `Column` objects. To get the numpy array from a `Column` you can `col[i]`, `col[i:j]` or even `col[:]` if you want the full column as a numpy array:

```python
from datasets import load_dataset
dataset = load_dataset(...)
dataset = dataset.with_format("numpy")
print(dataset["star"][:].ndim)
```

Ah perfect, thanks for clearing this up. I would close this ticket then.