rIsHu009/github-issues-updated huggingface dataset:
Title: pyarrow.lib.arrowinvalid: cannot mix list and non-list, non-null values with map function
Author: alexxchen (NONE)
State: open
Created: 2025-02-21T10:58:06Z

### Describe the bug

Encounter pyarrow.lib.arrowinvalid error with map function in some example when loading the dataset

### Steps to reproduce the bug

```
from datasets import load_dataset
from PIL import Image, PngImagePlugin

dataset = load_dataset("leonardPKU/GEOQA_R1V_Train_8K")
system_prompt="You are a helpful AI Assistant"
def make_conversation(example):
    prompt = []

    prompt.append({"role": "system", "content": system_prompt})
    prompt.append(
            {
                "role": "user", 
                "content": [
                        {"type": "image"},
                        {"type": "text", "text": example["problem"]},
                        ]
            }
        )
    return {"prompt": prompt}

def check_data_types(example):
    for key, value in example.items():
        if key == 'image':
            if not isinstance(value, PngImagePlugin.PngImageFile):
                print(value)
        if key == "problem" or key == "solution":
            if not isinstance(value, str):
                print(value)

    return example

dataset = dataset.map(check_data_types)
dataset = dataset.map(make_conversation)
```

### Expected behavior

Successfully process the dataset with map

### Environment info

datasets==3.3.1

Comments:
@lhoestq 

Can you try passing text: None for the image object ? Pyarrow expects all the objects to have the exact same type, in particular the dicttionaries in "content" should all have the keys "type" and "text"

The following modification on system prompt works, but it is different from the usual way to use it.
```
def make_conversation(example):
    prompt = []

    prompt.append({"role": "system", "content": [{"type": "text", "text": system_prompt}]})
    prompt.append(
            {
                "role": "user", 
                "content": [
                        {"type": "image"},
                        {"type": "text", "text": example["problem"]},
                        ]
            }
        )
    return {"prompt": prompt}
```

Good to know ! But yes Arrow / Parquet have this typing limitation (which is great to ensure data integrity, but constraining at the same time). It's is really blocking you, feel free to ping the arrow team / community if they plan to have a Union type or a JSON type
Title: Release: 3.3.2
Author: lhoestq (MEMBER)
State: closed
Created: 2025-02-20T17:42:11Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7416). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Shard Dataset at specific indices
Author: nikonikolov (NONE)
State: open
Created: 2025-02-20T10:43:10Z

I have a dataset of sequences, where each example in the sequence is a separate row in the dataset (similar to LeRobotDataset). When running `Dataset.save_to_disk` how can I provide indices where it's possible to shard the dataset such that no episode spans more than 1 shard. Consequently, when I run `Dataset.load_from_disk`, how can I load just a subset of the shards to save memory and time on different ranks?

I guess an alternative to this would be, given a loaded `Dataset`, how can I run `Dataset.shard` such that sharding doesn't split any episode across shards?

Comments:
Hi ! if it's an option I'd suggest to have one sequence per row instead.

Otherwise you'd have to make your own save/load mechanism

Saving one sequence per row is very difficult and heavy and makes all the optimizations pointless. How would a custom save/load mechanism look like?

You can use `pyarrow` for example to save/load individual arrow or parquet files and control what they contain
Title: Gracefully cancel async tasks
Author: lhoestq (MEMBER)
State: closed
Created: 2025-02-19T16:10:58Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7414). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Documentation on multiple media files of the same type with WebDataset
Author: DCNemesis (NONE)
State: open
Created: 2025-02-18T16:13:20Z

The [current documentation](https://huggingface.co/docs/datasets/en/video_dataset) on a creating a video dataset includes only examples with one media file and one json. It would be useful to have examples where multiple files of the same type are included. For example, in a sign language dataset, you may have a base video and a video annotation of the extracted pose. According to the WebDataset documentation, this should be able to be done with period separated filenames. For example:

```e39871fd9fd74f55.base.mp4
e39871fd9fd74f55.pose.mp4
e39871fd9fd74f55.json
f18b91585c4d3f3e.base.mp4
f18b91585c4d3f3e.pose.mp4
f18b91585c4d3f3e.json
... 
```

If you can confirm that this method of including multiple media files works with huggingface datasets and include an example in the documentation, I'd appreciate it.

Comments:
Yes this is correct and it works with huggingface datasets as well ! Feel free to include an example here: https://github.com/huggingface/datasets/blob/main/docs/source/video_dataset.mdx
Title: Index Error Invalid Ket is out of bounds for size 0 for code-search-net/code_search_net dataset
Author: harshakhmk (NONE)
State: open
Created: 2025-02-18T05:58:33Z

### Describe the bug

I am trying to do model pruning on sentence-transformers/all-mini-L6-v2 for the code-search-net/code_search_net dataset using INCTrainer class 
However I am getting below error

```
raise IndexError(f"Invalid Key: {key is our of bounds for size {size}")
IndexError: Invalid key: 1840208 is out of bounds for size 0
```

### Steps to reproduce the bug

Model pruning on the above dataset using the below guide 

https://huggingface.co/docs/optimum/en/intel/neural_compressor/optimization#pruning
### Expected behavior

The modsl should be successfully pruned 

### Environment info

Torch version: 2.4.1
Python version: 3.8.10

Comments:
Title: Attempt to fix multiprocessing hang by closing and joining the pool before termination
Author: dakinggg (CONTRIBUTOR)
State: closed
Created: 2025-02-17T23:58:03Z

https://github.com/huggingface/datasets/issues/6393 has plagued me on and off for a very long time. I have had various workarounds (one time combining two filter calls into one filter call removed the issue, another time making rank 0 go first resolved a cache race condition, one time i think upgrading the version of something resolved it). I don't know hf datasets well enough to fully understand the root cause, but I _think_ this PR fixes it.

Evidence: I have an LLM Foundry training yaml/script (datasets version 3.2.0) that results in a hang ~1/10 times (for a baseline for this testing, it was 2/36 runs that hung). I also reran with the latest datasets version (3.3.1) and got 4/36 hung. Installing datasets from this PR, I was able to successful run the script 144 times without a hang occurring. Assuming the base probability is 1/10, this should be more than enough times to have confidence it works.

After adding some logging, I could see that the code hung during the __exit__ of the mp pool context manager, after all shards had been processed, and the tqdm context manager had exited.

My best explanation: When multiprocessing pool __exit__ is called, it calls pool.terminate, which forcefully exits all the processes (and calls code related to this that I haven't looked at closely). I'm guessing this forceful termination has a bad interaction with some multithreading/multiprocessing that hf datasets does. If we instead call pool.close and pool.join before the pool.terminate happens, perhaps whatever that bad interaction is is able to complete gracefully, and then terminate call proceeds without issue.

If this PR seems good to you, I'd be very appreciative if you were able to do a patch release including it. Thank you!

@lhoestq

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7411). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the fix! We have been affected by this a lot when we try to use LLM Foundry with larger multimodal ICL datasets. 

@lorabit110 are you able to test it out for your case as well? Would be great to get a second validation that it actually fixes the issue. Thanks!
Title: Release: 3.3.1
Author: lhoestq (MEMBER)
State: closed
Created: 2025-02-17T14:52:12Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7409). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Fix filter speed regression
Author: lhoestq (MEMBER)
State: closed
Created: 2025-02-17T14:25:32Z

close https://github.com/huggingface/datasets/issues/7404

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7408). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Adding Core Maintainer List to CONTRIBUTING.md
Author: jp1924 (CONTRIBUTOR)
State: closed
Created: 2025-02-17T00:32:40Z
Labels: enhancement

### Feature request

I propose adding a core maintainer list to the `CONTRIBUTING.md` file.

### Motivation

The Transformers and Liger-Kernel projects maintain lists of core maintainers for each module. 
However, the Datasets project doesn't have such a list. 

### Your contribution

I have nothing to add here.

Comments:
@lhoestq

there is no per-module maintainer and the list is me alone nowadays ^^'

@lhoestq 
Oh... I feel for you. 
What are your criteria for choosing a core maintainer? 
It seems like it's too much work for you to manage all this code by yourself.

Also, if you don't mind, can you check this PR for me?
#7368 I'd like this to be added as soon as possible because I need it.
Title: Lazy loading of environment variables
Author: nikvaessen (NONE)
State: open
Created: 2025-02-16T22:31:41Z

### Describe the bug

Loading a `.env` file after an `import datasets` call does not correctly use the environment variables. 

This is due the fact that environment variables are read at import time:
https://github.com/huggingface/datasets/blob/de062f0552a810c52077543c1169c38c1f0c53fc/src/datasets/config.py#L155C1-L155C80

### Steps to reproduce the bug

```bash
# make tmp dir
mkdir -p /tmp/debug-env
# make .env file
echo HF_HOME=/tmp/debug-env/data > /tmp/debug-env/.env
# first load dotenv, downloads to /tmp/debug-env/data
uv run --with datasets,python-dotenv python3 -c \
'import dotenv; dotenv.load_dotenv("/tmp/debug-env/.env"); import datasets; datasets.load_dataset("Anthropic/hh-rlhf")' 
# first import datasets, downloads to `~/.cache/huggingface`
uv run --with datasets,python-dotenv python3 -c \
'import datasets; import dotenv; dotenv.load_dotenv("/tmp/debug-env/.env"); datasets.load_dataset("Anthropic/hh-rlhf")' 

```

### Expected behavior

I expect that setting environment variables with something like this:

```python3
if __name__ == "__main__":
    load_dotenv()
    main()
```
works correctly. 

### Environment info

    "datasets>=3.3.0",

Comments:
Many python packages out there, including `huggingface_hub`, do load the environment variables on import.
You should `load_dotenv()` before importing the libraries.

For example you can move all you imports inside your `main()` function
Title: Performance regression in `dataset.filter`
Author: ttim (NONE)
State: closed
Created: 2025-02-16T22:19:14Z

### Describe the bug

We're filtering dataset of ~1M (small-ish) records. At some point in the code we do `dataset.filter`, before (including 3.2.0) it was taking couple of seconds, and now it takes 4 hours.

We use 16 threads/workers, and stack trace at them look as follows:
```
Traceback (most recent call last):
  File "/python/lib/python3.12/site-packages/multiprocess/process.py", line 314, in _bootstrap
    self.run()
  File "/python/lib/python3.12/site-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/python/lib/python3.12/site-packages/multiprocess/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/utils/py_utils.py", line 678, in _write_generator_to_queue
    for i, result in enumerate(func(**kwargs)):
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3511, in _map_single
    for i, batch in iter_outputs(shard_iterable):
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3461, in iter_outputs
    yield i, apply_function(example, i, offset=offset)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 3390, in apply_function
    processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/python/lib/python3.12/site-packages/datasets/arrow_dataset.py", line 6416, in get_indices_from_mask_function
    indices_array = indices_mapping.column(0).take(indices_array)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/table.pxi", line 1079, in pyarrow.lib.ChunkedArray.take
  File "/python/lib/python3.12/site-packages/pyarrow/compute.py", line 458, in take
    def take(data, indices, *, boundscheck=True, memory_pool=None):
```

### Steps to reproduce the bug

1. Save dataset of 1M records in arrow
2. Filter it with 16 threads
3. Watch it take too long

### Expected behavior

Filtering done fast

### Environment info

datasets 3.3.0, python 3.12

Comments:
Thanks for reporting, I'll fix the regression today

I just released `datasets` 3.3.1 with a fix, let me know if it's good now :)

@lhoestq it fixed the issue.

This was (very) fast, thank you very much!
Title: Synchronize parameters for various datasets
Author: grofte (NONE)
State: open
Created: 2025-02-14T09:15:11Z

### Describe the bug

[IterableDatasetDict](https://huggingface.co/docs/datasets/v3.2.0/en/package_reference/main_classes#datasets.IterableDatasetDict.map) map function is missing the `desc` parameter. You can see the equivalent map function for [Dataset here](https://huggingface.co/docs/datasets/v3.2.0/en/package_reference/main_classes#datasets.Dataset.map).

There might be other parameters missing - I haven't checked.

### Steps to reproduce the bug

from datasets import Dataset, IterableDataset, IterableDatasetDict

ds = IterableDatasetDict({"train": Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3), 
                          "validate": Dataset.from_dict({"a": range(6)}).to_iterable_dataset(num_shards=3)})

for d in ds["train"]:
    print(d)

ds = ds.map(lambda x: {k: v+1 for k, v in x.items()}, desc="increment")

for d in ds["train"]:
    print(d)

### Expected behavior

The description parameter should be available for all datasets (or none). 

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.11.11
- `huggingface_hub` version: 0.28.1
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.9.0

Comments:
Hi ! the `desc` parameter is only available for Dataset / DatasetDict for the progress bar of `map()``

Since IterableDataset only runs the map functions when you iterate over the dataset, there is no progress bar and `desc` is useless. We could still add the argument for parity but it wouldn't be used for anything

I think you should add it. It doesn't hurt. The reason I ran into it was because I re-wrote a pipeline to use either a stream or a fully loaded dataset. Of course I can simply remove it but it is nice to have on the memory loaded dataset.
Title: Release: 3.3.0
Author: lhoestq (MEMBER)
State: closed
Created: 2025-02-14T09:15:03Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7398). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Update README.md
Author: lhoestq (MEMBER)
State: closed
Created: 2025-02-13T17:44:36Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7396). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Update docs
Author: lhoestq (MEMBER)
State: closed
Created: 2025-02-13T16:43:15Z

- update min python version
- replace canonical dataset names with new names
- avoid examples with trust_remote_code

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7395). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Using load_dataset with data_files and split arguments yields an error
Author: devon-research (NONE)
State: open
Created: 2025-02-12T04:50:11Z

### Describe the bug

It seems the list of valid splits recorded by the package becomes incorrectly overwritten when using the `data_files` argument.

If I run
```python
from datasets import load_dataset
load_dataset("allenai/super", split="all_examples", data_files="tasks/expert.jsonl")
```
then I get the error
```
ValueError: Unknown split "all_examples". Should be one of ['train'].
```

However, if I run
```python
from datasets import load_dataset
load_dataset("allenai/super", split="train", name="Expert")
```
then I get
```
ValueError: Unknown split "train". Should be one of ['all_examples'].
```

### Steps to reproduce the bug

Run
```python
from datasets import load_dataset
load_dataset("allenai/super", split="all_examples", data_files="tasks/expert.jsonl")
```

### Expected behavior

No error.

### Environment info

Python = 3.12
datasets = 3.2.0

Comments:
Title: Optimized sequence encoding for scalars
Author: lukasgd (CONTRIBUTOR)
State: closed
Created: 2025-02-11T20:30:44Z

The change in https://github.com/huggingface/datasets/pull/3197 introduced redundant list-comprehensions when `obj` is a long sequence of scalars. This becomes a noticeable overhead when loading data from an `IterableDataset` in the function `_apply_feature_types_on_example` and can be eliminated by adding a check for scalars in `encode_nested_example` proposed here. 

In the following code example
```
import time
from datasets.features import Sequence, Value
from datasets.features.features import encode_nested_example

schema = Sequence(Value("int32"))
obj = list(range(100000))

start = time.perf_counter()
result = encode_nested_example(schema, obj)
stop = time.perf_counter()

print(f"Time spent is {stop-start} sec")
```
`encode_nested_example` becomes 492x faster (from 0.0769 to 0.0002 sec), respectively 322x (from 0.00814 to 0.00003 sec) for a list of length 10000, on a GH200 system, making it unnoticeable when loading data with tokenization.

Another change is made to avoid creating arrays from scalars and afterwards re-extracting them during casting to python (`obj == obj.__array__()[()]` in that case), which avoids a regression in the array write benchmarks.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7393). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: push_to_hub payload too large error when using large ClassLabel feature
Author: DavidRConnell (NONE)
State: open
Created: 2025-02-11T17:51:34Z

### Describe the bug

When using `datasets.DatasetDict.push_to_hub` an `HfHubHTTPError: 413 Client Error: Payload Too Large for url` is raised if the dataset contains a large `ClassLabel` feature. Even if the total size of the dataset is small.

### Steps to reproduce the bug

``` python
import random
import sys

import datasets

random.seed(42)


def random_str(sz):
    return "".join(chr(random.randint(ord("a"), ord("z"))) for _ in range(sz))


data = datasets.DatasetDict(
    {
        str(i): datasets.Dataset.from_dict(
            {
                "label": [list(range(3)) for _ in range(10)],
                "abstract": [random_str(10_000) for _ in range(10)],
            },
        )
        for i in range(3)
    }
)
features = data["1"].features.copy()
features["label"] = datasets.Sequence(
    datasets.ClassLabel(names=[str(i) for i in range(50_000)])
)
data = data.map(lambda examples: {}, features=features)

feat_size = sys.getsizeof(data["1"].features["label"].feature.names)
print(f"Size of ClassLabel names: {feat_size}")
# Size of ClassLabel names: 444376


data.push_to_hub("dconnell/pubtator3_test")
```

Note that this succeeds if `ClassLabel` has fewer names or if `ClassLabel` is replaced with `Value("int64")`


### Expected behavior

Should push the dataset to hub.

### Environment info

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 3.2.0
- Platform: Linux-5.15.0-126-generic-x86_64-with-glibc2.35
- Python version: 3.12.8
- `huggingface_hub` version: 0.28.1
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments:
See also <https://discuss.huggingface.co/t/datasetdict-push-to-hub-failing-with-payload-to-large/140083/8>
Title: Re-add py.typed
Author: NeilGirdhar (CONTRIBUTOR)
State: open
Created: 2025-02-10T22:12:52Z
Labels: enhancement

### Feature request

The motivation for removing py.typed no longer seems to apply.  Would a solution like [this one](https://github.com/huggingface/huggingface_hub/pull/2752) work here?

### Motivation

MyPy support is broken.  As more type checkers come out, such as RedKnot, these may also be broken.  It would be good to be PEP 561 compliant as long as it's not too onerous.

### Your contribution

I can re-add py.typed, but I don't know how to make sur all of the `__all__` files are provided (although you may not need to with modern PyRight).

Comments:
Title: Getting statistics about filtered examples
Author: jonathanasdf (NONE)
State: closed
Created: 2025-02-10T20:48:29Z

@lhoestq wondering if the team has thought about this and if there are any recommendations?

Currently when processing datasets some examples are bound to get filtered out, whether it's due to bad format, or length is too long, or any other custom filters that might be getting applied. Let's just focus on the filter by length for now, since that would be something that gets applied dynamically for each training run. Say we want to show a graph in W&B with the running total of the number of filtered examples so far.

What would be a good way to go about hooking this up? Because the map/filter operations happen before the DataLoader batches are created, at training time if we're just grabbing batches from the DataLoader then we won't know how many things have been filtered already. But there's not really a good way to include a 'num_filtered' key into the dataset itself either because dataset map/filter process examples independently and don't have a way to track a running sum.

The only approach I can kind of think of is having a 'is_filtered' key in the dataset, and then creating a custom batcher/collator that reads that and tracks the metric?

Comments:
You can actually track a running sum in map() or filter() :)

```python
num_filtered = 0

def f(x):
    global num_filtered
    condition = len(x["text"]) < 1000
    if not condition:
        num_filtered += 1
    return condition

ds = ds.filter(f)
print(num_filtered)
```

and if you want to use multiprocessing, make sure to use a variable that is shared across processes


```python
from multiprocess import Manager

manager = Manager()
num_filtered = manager.Value('i', 0)

def f(x):
    global num_filtered
    condition = len(x["text"]) < 1000
    if not condition:
        num_filtered.value += 1
    return condition

ds = ds.filter(f, num_proc=4)
print(num_filtered.value)
```

PS: `datasets` uses `multiprocess` instead of the `multiprocessing` package to support lambda functions in map() and filter()

Oh that's great to know!

I guess this value would not be exactly synced with the batch in cases of pre-fetch and shuffle buffers and so on, but that's probably fine. Thanks!
Title: OSError: [Errno 22] Invalid argument forbidden character
Author: langflogit (NONE)
State: closed
Created: 2025-02-10T17:46:31Z

### Describe the bug

I'm on Windows and i'm trying to load a datasets but i'm having title error because files in the repository are named with charactere like < >which can't be in a name file. Could it be possible to load this datasets but removing those charactere ?

### Steps to reproduce the bug

load_dataset("CATMuS/medieval") on Windows

### Expected behavior

Making the function to erase the forbidden character to allow loading the datasets who have those characters. 

### Environment info

- `datasets` version: 3.2.0
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.12.2
- `huggingface_hub` version: 0.28.1
- PyArrow version: 19.0.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments:
You can probably copy the dataset in your HF account and rename the files (without having to download them to your disk). Or alternatively feel free to open a Pull Request to this dataset with the renamed file

Thank you, that will help me work around this problem
Title: Dynamic adjusting dataloader sampling weight
Author: whc688 (NONE)
State: open
Created: 2025-02-10T03:18:47Z

Hi,
Thanks for your wonderful work! I'm wondering is there a way to dynamically adjust the sampling weight of each data in the dataset during training? Looking forward to your reply, thanks again.

Comments:
You mean based on a condition that has to be checked on-the-fly during training ? Otherwise if you know in advance after how many samples you need to change the sampling you can simply concatenate the two mixes

Yes, like during training, if one data sample's prediction is consistently wrong, its sampling weight gets higher and higher, and if one data sample's prediction is already correct, then we rarely sample it

it's not possible to use `interleave_datasets()` and modify the probabilities while iterating on the dataset at the moment, so you'd have to implement your own torch `Sampler` or your own`IterableDataset` to implement this logic
Title: Add bookfolder Dataset Builder for Digital Book Formats
Author: shikanime (NONE)
State: closed
Created: 2025-02-08T14:27:55Z
Labels: enhancement

### Feature request

This feature proposes adding a new dataset builder called bookfolder to the datasets library. This builder would allow users to easily load datasets consisting of various digital book formats, including: AZW, AZW3, CB7, CBR, CBT, CBZ, EPUB, MOBI, and PDF.

### Motivation

Currently, loading datasets of these digital book files requires manual effort. This would also lower the barrier to entry for working with these formats, enabling more diverse and interesting datasets to be used within the Hugging Face ecosystem.

### Your contribution

This feature is rather simple as it will be based on the folder-based builder, similar to imagefolder. I'm willing to contribute to this feature by submitting a PR

Comments:
On second thought, probably not a good idea.
Title: Make IterableDataset (optionally) resumable
Author: yzhangcs (CONTRIBUTOR)
State: open
Created: 2025-02-04T15:55:33Z

### What does this PR do?
This PR introduces a new `stateful` option to the `dataset.shuffle` method, which defaults to `False`. 
When enabled, this option allows for resumable shuffling of `IterableDataset` instances, albeit with some additional memory overhead.

Key points:
* All tests have passed
* Docstrings have been updated to reflect the new functionality

I'm very looking forward to receiving feedback on this implementation! @lhoestq

Comments:
@lhoestq Hi again~ Just circling back on this
Wondering if there’s anything I can do to help move this forward. 🤗 
Thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7385). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Support async functions in map()
Author: lhoestq (MEMBER)
State: closed
Created: 2025-02-03T18:18:40Z

e.g. to download images or call an inference API like HF or vLLM

```python
import asyncio
import random

from datasets import Dataset


async def f(x):
    await asyncio.sleep(random.random())

ds = Dataset.from_dict({"data": range(100)})
ds.map(f)
# Map: 100%|█████████████████████████████| 100/100 [00:01<00:00, 99.81 examples/s]
```

TODO
- [x] clean code (right now it's a big copy paste)
- [x] batched
- [x] Dataset.map()
- [x] IterableDataset.map()
- [x] Dataset.filter()
- [x] IterableDataset.filter()
- [x] test
- [x] docs

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7384). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

example of what you can do with it:

```python
import aiohttp
from huggingface_hub import get_token

from datasets import Dataset


API_URL = "https://api-inference.huggingface.co/models/microsoft/Phi-3-mini-4k-instruct/v1/chat/completions"
PROMPT = "What is this text mainly about ? Here is the text:\n\n```\n{Problem}\n```\n\nReply in one or two words."

async def query(example):
    headers = {"Authorization": f"Bearer {get_token()}", "Content-Type": "application/json"}
    json = {"messages": [{"role": "user", "content": PROMPT.format(Problem=example["Problem"])}], "max_tokens": 20, "seed": 42}
    async with aiohttp.ClientSession() as session, session.post(API_URL, headers=headers, json=json) as response:
        output = await response.json()
        return {"output": output["choices"][0]["message"]["content"]}

ds = Dataset.from_dict({"Problem": ["1 + 1"] * 10})
ds = ds.map(query)
print(ds[0])
# {'Problem': '1 + 1', 'output': 'Arithmetic'}
```
Title: Add Pandas, PyArrow and Polars docs
Author: lhoestq (MEMBER)
State: closed
Created: 2025-01-31T13:22:59Z

(also added the missing numpy docs and fixed a small bug in pyarrow formatting)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7382). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: fix: dill default for version bigger 0.3.8
Author: sam-hey (NONE)
State: closed
Created: 2025-01-26T13:37:16Z

Fixes def log for dill version >= 0.3.9 
https://pypi.org/project/dill/

This project uses dill with the release of version 0.3.9 the datasets lib.

Comments:
`datasets` doesn't support `dill` 0.3.9 yet afaik since `dill` made some changes related to the determinism of dumps

It would be cool to investigate (maybe run the `datasets` test) with recent `dill` to see excactly what breaks and if we can make `dill` 0.3.9 work with `datasets`
Title: Allow pushing config version to hub
Author: momeara (NONE)
State: open
Created: 2025-01-21T22:35:07Z
Labels: enhancement

### Feature request

Currently, when datasets are created, they can be versioned by passing the `version` argument to `load_dataset(...)`. For example creating `outcomes.csv` on the command line

```
echo "id,value\n1,0\n2,0\n3,1\n4,1\n" > outcomes.csv
```
and creating it
```
import datasets
dataset = datasets.load_dataset(
    "csv",
    data_files ="outcomes.csv",
    keep_in_memory = True,
    version = '1.0.0')
```

The version info is stored in the `info` and can be accessed e.g. by `next(iter(dataset.values())).info.version`

This dataset can be uploaded to the hub with `dataset.push_to_hub(repo_id = "maomlab/example_dataset")`. This will create a dataset  on the hub with the following in the `README.md`, but it doesn't upload the version information:

```
---
dataset_info:
  features:
  - name: id
    dtype: int64
  - name: value
    dtype: int64
  splits:
  - name: train
    num_bytes: 64
    num_examples: 4
  download_size: 1332
  dataset_size: 64
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
---
```

However, when I download from the hub, the version information is missing:

```
dataset_from_hub_no_version = datasets.load_dataset("maomlab/example_dataset")
next(iter(dataset.values())).info.version
```

I can add the version information manually to the hub, by appending it to the end of config section:

```
...
configs:
- config_name: default
  data_files:
  - split: train
    path: data/train-*
  version: 1.0.0
---
```

And then when I download it, the version information is correct.





### Motivation

### Why adding version information for each config makes sense
  1. The version information is already recorded in the dataset config info data structure and is able to parse it correctly, so it makes sense to sync it with `push_to_hub`. 
  2. Keeping the version info in at the config level is different from version info at the branch level. As the former relates to the version of the specific dataset the config refers to rather than the version of the dataset curation itself.

## A explanation for the current behavior:
In [datasets/src/datasets/info.py:159](https://github.com/huggingface/datasets/blob/fb91fd3c9ea91a818681a777faf8d0c46f14c680/src/datasets/info.py#L159C1-L160C1
), the `_INCLUDED_INFO_IN_YAML` variable doesn't include `"version"`.

If my reading of the code is right, adding `"version"` to `_INCLUDED_INFO_IN_YAML`, would allow the version information to be uploaded to the hub.

### Your contribution

Request: add `"version"` to `_INCLUDE_INFO_IN_YAML` in [datasets/src/datasets/info.py:159](https://github.com/huggingface/datasets/blob/fb91fd3c9ea91a818681a777faf8d0c46f14c680/src/datasets/info.py#L159C1-L160C1
)

Comments:
Hi ! This sounds reasonable to me, feel free to open a PR :)
Title: Support for sparse arrays with the Arrow Sparse Tensor format?
Author: JulesGM (NONE)
State: open
Created: 2025-01-21T20:14:35Z
Labels: enhancement

### Feature request

AI in biology is becoming a big thing. One thing that would be a huge benefit to the field that Huggingface Datasets doesn't currently have is native support for **sparse arrays**. 


Arrow has support for sparse tensors. 
https://arrow.apache.org/docs/format/Other.html#sparse-tensor


It would be a big deal if Hugging Face Datasets supported sparse tensors as a feature type, natively. 


### Motivation

This is important for example in the field of transcriptomics (modeling and understanding gene expression), because a large fraction of the genes are not expressed (zero). More generally, in science, sparse arrays are very common, so adding support for them would be very benefitial, it would make just using Hugging Face Dataset objects a lot more straightforward and clean.


### Your contribution

We can discuss this further once the team comments of what they think about the feature, and if there were previous attempts at making it work, and understanding their evaluation of how hard it would be. My intuition is that it should be fairly straightforward, as the Arrow backend already supports it.

Comments:
Hi ! Unfortunately the Sparse Tensor structure in Arrow is not part of the Arrow format (yes it's confusing...), so it's not possible to use it in `datasets`. It's a separate structure that doesn't correspond to any type or extension type in Arrow.

The Arrow community recently added an extension type for fixed shape tensors at https://arrow.apache.org/docs/format/CanonicalExtensions.html#fixed-shape-tensor, it should be possible to contribute an extension type for sparse tensors as well.
Title: vllm批量推理报错
Author: YuShengzuishuai (NONE)
State: open
Created: 2025-01-21T03:22:23Z

### Describe the bug

![Image](https://github.com/user-attachments/assets/3d958e43-28dc-4467-9333-5990c7af3b3f)

### Steps to reproduce the bug

![Image](https://github.com/user-attachments/assets/3067eeca-a54d-4956-b0fd-3fc5ea93dabb)

### Expected behavior

![Image](https://github.com/user-attachments/assets/77d32936-488f-4572-9365-bfb4170e555b)

### Environment info

![Image](https://github.com/user-attachments/assets/230335c4-825f-4db1-b07d-4776ef63ead8)

Comments:
Make sure you have installed a recent version of `soundfile`
Title: Excessive RAM Usage After Dataset Concatenation concatenate_datasets
Author: sam-hey (NONE)
State: open
Created: 2025-01-16T16:33:10Z

### Describe the bug

When loading a dataset from disk, concatenating it, and starting the training process, the RAM usage progressively increases until the kernel terminates the process due to excessive memory consumption.

https://github.com/huggingface/datasets/issues/2276

### Steps to reproduce the bug

```python
from datasets import  DatasetDict, concatenate_datasets

dataset = DatasetDict.load_from_disk("data")

...
...

combined_dataset = concatenate_datasets(
        [dataset[split] for split in dataset]
    )

#start SentenceTransformer training
```

### Expected behavior

I would not expect RAM utilization to increase after concatenation. Removing the concatenation step resolves the issue

### Environment info

sentence-transformers==3.1.1
datasets==3.2.0

python3.10

Comments:
![Image](https://github.com/user-attachments/assets/b6f8bcbd-44af-413e-bc06-65380eb0f746)

![Image](https://github.com/user-attachments/assets/a241fcd8-4b62-495c-926c-685f82015dfb)

Adding a img from memray
https://gist.github.com/sam-hey/00c958f13fb0f7b54d17197fe353002f

I'm having the same issue where concatenation seems to use a huge amount of RAM.

```python
# Load all chunks and concatenate them into a final dataset.
        chunk_datasets = [
            Dataset.load_from_disk(file, keep_in_memory=False)
            for file in tqdm(chunk_files, desc="Loading chunk datasets")
        ]
        logging.info("Concatenating chunk datasets...")
        final_dataset = concatenate_datasets(chunk_datasets)
```

This is a real issue for me as the final dataset is a few terabytes in size. I'm using datasets version `3.1.0`. Also tested with version `3.4.1`

I did have a short look, the error seems to be from `memory_map` and the stream not being closed. 

https://github.com/huggingface/datasets/blob/5f8d2ad9a1b0bccfd962d998987228addfd5be9f/src/datasets/table.py#L48-L50


Did not have the time to test jet: https://github.com/sam-hey/datasets/tree/fix/concatenate_datasets

I will probably have a better look in a couple of days.
Title: Inconsistent Behavior Between `load_dataset` and `load_from_disk` When Loading Sharded Datasets
Author: gaohongkui (NONE)
State: open
Created: 2025-01-16T05:47:20Z

### Description

I encountered an inconsistency in behavior between `load_dataset` and `load_from_disk` when loading sharded datasets. Here is a minimal example to reproduce the issue:

#### Code 1: Using `load_dataset`
```python
from datasets import Dataset, load_dataset

# First save with max_shard_size=10
Dataset.from_dict({"id": range(1000)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Second save with max_shard_size=10
Dataset.from_dict({"id": range(500)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Load the DatasetDict
loaded_datasetdict = load_dataset("my_sharded_datasetdict")
print(loaded_datasetdict)
```
**Output**:
- `train` has 1350 samples.
- `test` has 150 samples.

#### Code 2: Using `load_from_disk`
```python
from datasets import Dataset, load_from_disk

# First save with max_shard_size=10
Dataset.from_dict({"id": range(1000)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Second save with max_shard_size=10
Dataset.from_dict({"id": range(500)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Load the DatasetDict
loaded_datasetdict = load_from_disk("my_sharded_datasetdict")
print(loaded_datasetdict)
```
**Output**:
- `train` has 450 samples.
- `test` has 50 samples.

### Expected Behavior
I expected both `load_dataset` and `load_from_disk` to load the same dataset, as they are pointing to the same directory. However, the results differ significantly:
- `load_dataset` seems to merge all shards, resulting in a combined dataset.
- `load_from_disk` only loads the last saved dataset, ignoring previous shards.

### Questions
1. Is this behavior intentional? If so, could you clarify the difference between `load_dataset` and `load_from_disk` in the documentation?
2. If this is not intentional, could this be considered a bug?
3. What is the recommended way to handle cases where multiple datasets are saved to the same directory?


Thank you for your time and effort in maintaining this great library! I look forward to your feedback.

Comments:
Title: Support faster processing using pandas or polars functions in `IterableDataset.map()`
Author: lhoestq (MEMBER)
State: closed
Created: 2025-01-14T18:14:13Z

Following the polars integration :)

Allow super fast processing using pandas or polars functions in `IterableDataset.map()` by adding support to pandas and polars formatting in `IterableDataset`

```python
import polars as pl
from datasets import Dataset

ds = Dataset.from_dict({"i": range(10)}).to_iterable_dataset()
ds = ds.with_format("polars")
ds = ds.map(lambda df: df.with_columns(pl.col("i").add(1).alias("i+1")), batched=True)
ds = ds.with_format(None)
print(next(iter(ds)))
# {'i': 0, 'i+1': 1}
```

It leverages arrow's zero-copy features from/to pandas and polars.

related to https://github.com/huggingface/datasets/issues/3444 #6762

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7370). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

merging this and will make some docs and communications around using polars for optimizing data processing :)
Title: Importing dataset gives unhelpful error message when filenames in metadata.csv are not found in the directory
Author: svencornetsdegroot (NONE)
State: open
Created: 2025-01-14T13:53:21Z

### Describe the bug

While importing an audiofolder dataset, where the names of the audiofiles don't correspond to the filenames in the metadata.csv, we get an unclear error message that is not helpful for the debugging, i.e. 
```
ValueError: Instruction "train" corresponds to no data!
```

### Steps to reproduce the bug

Assume an audiofolder with audiofiles, filename1.mp3, filename2.mp3 etc and a file metadata.csv which contains the columns file_name and sentence. The file_names are formatted like filename1.mp3, filename2.mp3 etc. 

Load the audio
```
from datasets import load_dataset
load_dataset("audiofolder", data_dir='/path/to/audiofolder')
```
When the file_names in the csv are not in sync with the filenames in the audiofolder, then we get an Error message:
```
File /opt/conda/lib/python3.12/site-packages/datasets/arrow_reader.py:251, in BaseReader.read(self, name, instructions, split_infos, in_memory)
    249 if not files:
    250     msg = f'Instruction "{instructions}" corresponds to no data!'
--> 251     raise ValueError(msg)
    252 return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)

ValueError: Instruction "train" corresponds to no data!
```
load_dataset has a default value for the argument split = 'train'. 


### Expected behavior

It would be better to get an error report something like:
```
The metadata.csv file has different filenames than the files in the datadirectory. 
```
It would have saved me 4 hours of debugging. 

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-427.40.1.el9_4.x86_64-x86_64-with-glibc2.39
- Python version: 3.12.8
- `huggingface_hub` version: 0.27.0
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments:
I'd prefer even more verbose errors; like `"file123.mp3" is referenced in metadata.csv, but not found in the data directory '/path/to/audiofolder' ! (and 100+ more missing files)` Or something along those lines.
Title: Add with_split to DatasetDict.map
Author: jp1924 (CONTRIBUTOR)
State: closed
Created: 2025-01-13T15:09:56Z

#7356

Comments:
Can you check this out, @lhoestq?

cc @lhoestq @albertvillanova 

@lhoestq


@lhoestq


@lhoestq

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7368). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq 

@lhoestq please...

Thank you so much for reviewing this PR! Have a great weekend~
Title: Dataset.from_dict() can't handle large dict
Author: CSU-OSS (NONE)
State: open
Created: 2025-01-11T02:05:21Z

### Describe the bug

I have 26,000,000 3-tuples. When I use Dataset.from_dict() to load, neither. py nor Jupiter notebook can run successfully. This is my code:
```
    # len(example_data) is 26,000,000, 'diff' is a text
    diff1_list = [example_data[i].texts[0] for i in range(len(example_data))]
    diff2_list = [example_data[i].texts[1] for i in range(len(example_data))]
    label_list = [example_data[i].label for i in range(len(example_data))]

    embedding_dataset = Dataset.from_dict({
        "diff1": diff1_list,
        "diff2": diff2_list,
        "label": label_list
    })
```

### Steps to reproduce the bug

1. Initialize a large 3-tuple, e.g. 26,000,000
2. Use Dataset.from_dict() to load

### Expected behavior

Dataset.from_dict() run successfully

### Environment info

sentence-transformers                   3.3.1

Comments:
Title: A parameter is specified but not used in datasets.arrow_dataset.Dataset.from_pandas()
Author: NourOM02 (NONE)
State: open
Created: 2025-01-10T13:39:33Z

### Describe the bug

I am interested in creating train, test and eval splits from a pandas Dataframe, therefore I was looking at the possibilities I can follow. I noticed the split parameter and was hopeful to use it in order to generate the 3 at once, however, while trying to understand the code, i noticed that it has no added value (correct me if I am wrong or misunderstood the code). 


from_pandas function code :

```python
    if info is not None and features is not None and info.features != features:
            raise ValueError(
                f"Features specified in `features` and `info.features` can't be different:\n{features}\n{info.features}"
            )
        features = features if features is not None else info.features if info is not None else None
        if info is None:
            info = DatasetInfo()
        info.features = features
        table = InMemoryTable.from_pandas(
            df=df,
            preserve_index=preserve_index,
        )
        if features is not None:
            # more expensive cast than InMemoryTable.from_pandas(..., schema=features.arrow_schema)
            # needed to support the str to Audio conversion for instance
            table = table.cast(features.arrow_schema)
        return cls(table, info=info, split=split)
```

### Steps to reproduce the bug

```python
from datasets import Dataset
# Filling the split parameter with whatever causes no harm at all
data = Dataset.from_pandas(self.raw_data, split='egiojegoierjgoiejgrefiergiuorenvuirgurthgi')
```

### Expected behavior

Would be great if there is no split parameter (if it isn't working), or to add a concrete example of how it can be used.

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.15.0-127-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.27.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0

Comments: