real-jiakai/github-issues huggingface dataset:
Slow iteration for iterable dataset with numpy formatting for array data
Author: alex-hh
State: open

### Describe the bug

When working with large arrays, setting with_format to e.g. numpy then applying map causes a significant slowdown for iterable datasets.

### Steps to reproduce the bug

```python
import numpy as np
import time
from datasets import Dataset, Features, Array3D

features=Features(**{"array0": Array3D((None, 10, 10), dtype="float32"), "array1": Array3D((None,10,10), dtype="float32")})
dataset = Dataset.from_dict({f"array{i}": [np.zeros((x,10,10), dtype=np.float32) for x in [2000,1000]*25] for i in range(2)}, features=features)
```

Then
```python
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy").map(lambda x: x)
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
print(t1-t0)
```

takes 27 s, whereas

```python
ds = dataset.to_iterable_dataset()
ds = ds.with_format("numpy")
ds = dataset.to_iterable_dataset()
t0 = time.time()
for ex in ds:
    pass
t1 = time.time()
print(t1 - t0)
```

takes ~1s

### Expected behavior

Map should not introduce a slowdown when formatting is enabled.

### Environment info

3.0.2

The below easily eats up 32G of RAM. Leaving it for a while bricked the laptop with 16GB.

```
dataset = load_dataset("Voxel51/OxfordFlowers102", data_dir="data").with_format("numpy")
processed_dataset = dataset.map(lambda x: x)
```

![image](https://github.com/user-attachments/assets/c1863a69-b18f-4014-89dc-98994336df96)

Similar problems occur if using a real transform function in `.map()`.
fix ci benchmark
Author: lhoestq
State: closed

we're not using the benchmarks anymore + they were not working anyway due to token permissions

I keep the code in case we ever want to re-run the benchmark manually

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7205). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix unbatched arrow map for iterable datasets
Author: alex-hh
State: closed

Fixes the bug when applying map to an arrow-formatted iterable dataset described here:

https://github.com/huggingface/datasets/issues/6833#issuecomment-2399903885

```python

from datasets import load_dataset
ds = load_dataset("rotten_tomatoes", split="train", streaming=True)
ds = ds.with_format("arrow").map(lambda x: x)
for ex in ds:
    pass
```

@lhoestq 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7204). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix the environment variable for huggingface cache
Author: torotoki
State: closed

Resolve  #6256. As far as I tested, `HF_DATASETS_CACHE` was ignored and I could not specify the cache directory at all except for the default one by this environment variable. `HF_HOME` has worked. Perhaps the recent change on file downloading by `huggingface_hub` could affect this bug.

In my testing, I could not specify the cache directory even by `load_dataset("dataset_name" cache_dir="...")`. It might be another issue. I also welcome any advice to solve this issue.

Hi ! yes now `datasets` uses `huggingface_hub` to download and cache files from the HF Hub so you need to use `HF_HOME` (or manually `HF_HUB_CACHE` and `HF_DATASETS_CACHE` if you want to separate HF Hub cached files and cached datasets Arrow files)

So in your change I guess it needs to be `HF_HOME` instead of `HF_CACHE` ?

Thank you for your comment. You are right. I am sorry for my mistake, I fixed it.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7200). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I just had this issue, and needed to move the setting the env code in the python file to top, before the import of the lib 
ie. 
```python
import os
LOCAL_DISK_MOUNT = '/mnt/data'

os.environ['HF_HOME'] = f'{LOCAL_DISK_MOUNT}/hf_cache/'
os.environ['HF_DATASETS_CACHE'] = f'{LOCAL_DISK_MOUNT}/datasets/'

from datasets import load_dataset
from datasets import load_dataset_builder
from psutil._common import bytes2human


```
Add repeat method to datasets
Author: alex-hh
State: open

Following up on discussion in #6623 and #7198 I thought this would be pretty useful for my case so had a go at implementing.

My main motivation is to be able to call iterable_dataset.repeat(None).take(samples_per_epoch) to safely avoid timeout issues in a distributed training setting. This would provide a straightforward workaround for several open issues related to this situation:  https://github.com/huggingface/datasets/issues/6437, https://github.com/huggingface/datasets/issues/6594, https://github.com/huggingface/datasets/issues/6623, https://github.com/huggingface/datasets/issues/6719.

@lhoestq let me know if this looks on the right track!
concatenate_datasets does not preserve shuffling state
Author: alex-hh
State: open

### Describe the bug

After concatenate datasets on an iterable dataset, the shuffling state is destroyed, similar to #7156 

This means concatenation cant be used for resolving uneven numbers of samples across devices when using iterable datasets in a distributed setting as discussed in #6623 

I also noticed that the number of shards is the same after concatenation, which I found surprising, but I don't understand the internals well enough to know whether this is actually surprising or not

### Steps to reproduce the bug

```python
import datasets
import torch.utils.data


def gen(shards):
    yield {"shards": shards}


def main():
    dataset1 = datasets.IterableDataset.from_generator(
        gen, gen_kwargs={"shards": list(range(25))}  # TODO: how to understand this?
    )
    dataset2 = datasets.IterableDataset.from_generator(
        gen, gen_kwargs={"shards": list(range(25, 50))}  # TODO: how to understand this?
    )
    dataset1 = dataset1.shuffle(buffer_size=1)
    dataset2 = dataset2.shuffle(buffer_size=1)
    print(dataset1.n_shards)
    print(dataset2.n_shards)

    dataset = datasets.concatenate_datasets(
        [dataset1, dataset2]
    )
    print(dataset.n_shards)
    # dataset = dataset1

    dataloader = torch.utils.data.DataLoader(
        dataset,
        batch_size=8,
        num_workers=0,
    )

    for i, batch in enumerate(dataloader):
        print(batch)
    print("\nNew epoch")

    dataset = dataset.set_epoch(1)

    for i, batch in enumerate(dataloader):
        print(batch)


if __name__ == "__main__":
    main()
```

### Expected behavior

Shuffling state should be preserved

### Environment info

Latest datasets
Add support for 3D datasets
Author: severo
State: open

See https://huggingface.co/datasets/allenai/objaverse for example

maybe related: https://github.com/huggingface/datasets/issues/6388

Also look at https://github.com/huggingface/dataset-viewer/blob/f5fd117ceded990a7766e705bba1203fa907d6ad/services/worker/src/worker/job_runners/dataset/modalities.py#L241 which lists the 3D file formats that will assign the 3D modality to a dataset.

~~we can brainstorm about the UX maybe (i don't expect we should load all models on the page at once – IMO there should be a manual action from user to load + maybe load first couple of row by default) cc @gary149 @cfahlgren1~~

it's more for the viewer issue (https://github.com/huggingface/dataset-viewer/issues/1003)
Support of num_workers (multiprocessing) in map for IterableDataset
Author: getao
State: open

### Feature request

Currently, IterableDataset doesn't support setting num_worker in .map(), which results in slow processing here. Could we add support for it? As .map() can be run in the batch fashion (e.g., batch_size is default to 1000 in datasets), it seems to be doable for IterableDataset as the regular Dataset.

### Motivation

Improving data processing efficiency

### Your contribution

Testing

I was curious about the same - since map is applied on the fly I was assuming that setting num_workers>1 in DataLoader would effectively do the map in parallel, have you tried that?
Add repeat() for iterable datasets
Author: alex-hh
State: open

### Feature request

It would be useful to be able to straightforwardly repeat iterable datasets indefinitely, to provide complete control over starting and ending of iteration to the user.

An IterableDataset.repeat(n) function could do this automatically

### Motivation

This feature was discussed in this issue https://github.com/huggingface/datasets/issues/7147, and would resolve the need to use the hack of interleave datasets with probability 0 as a simple way to achieve this functionality.

An additional benefit might be the simplification of the use of iterable datasets in a distributed setting:
If the user can assume that datasets will repeat indefinitely, then issues around different numbers of samples appearing on different devices (e.g. https://github.com/huggingface/datasets/issues/6437, https://github.com/huggingface/datasets/issues/6594, https://github.com/huggingface/datasets/issues/6623, https://github.com/huggingface/datasets/issues/6719) can potentially be straightforwardly resolved by simply doing:

ids.repeat(None).take(n_samples_per_epoch)


### Your contribution

I'm not familiar enough with the codebase to assess how straightforward this would be to implement.

If it might be very straightforward, I could possibly have a go.

perhaps concatenate_datasets can already be used to achieve almost the same effect? 

`concatenate_datasets` does the job when there is a finite number of repetitions, but in case of `.repeat()` forever we need a new logic in `iterable_dataset.py`