rIsHu009/github-issues-updated huggingface dataset:
Title: A bug of Dataset.to_json() function
Author: LinglingGreat (NONE)
State: open
Created: 2024-07-10T09:11:22Z
Labels: bug

### Describe the bug

When using the Dataset.to_json() function, an unexpected error occurs if the parameter is set to lines=False. The stored data should be in the form of a list, but it actually turns into multiple lists, which causes an error when reading the data again.
The reason is that to_json() writes to the file in several segments based on the batch size. This is not a problem when lines=True, but it is incorrect when lines=False, because writing in several times will produce multiple lists(when len(dataset) > batch_size).

### Steps to reproduce the bug

try this code:
```python
from datasets import load_dataset
import json

train_dataset = load_dataset("Anthropic/hh-rlhf", data_dir="harmless-base")["train"]
output_path = "./harmless-base_hftojs.json"
print(len(train_dataset))
train_dataset.to_json(output_path, lines=False, force_ascii=False, indent=2)

with open(output_path, encoding="utf-8") as f:
    data = json.loads(f.read())

```
it raise error: json.decoder.JSONDecodeError: Extra data: line 4003 column 1 (char 1373709)

Extra square brackets have appeared here:
<img width="265" alt="image" src="https://github.com/huggingface/datasets/assets/26499566/81492332-386d-42e8-88d1-b6d4ae3682cc">


### Expected behavior

The code runs normally.

### Environment info

datasets=2.20.0

Comments:
Thanks for reporting, @LinglingGreat.

I confirm this is a bug.

@albertvillanova I would like to take a shot at this if you aren't working on it currently. Let me know!
Title: Docs are not generated when a parameter defaults to a NamedSplit value
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-07-10T07:51:24Z
Labels: maintenance

While generating the docs, we get an error when some parameter defaults to a `NamedSplit` value, like:
```python
def call_function(split=Split.TRAIN):
    ...
```
The error is: ValueError: Equality not supported between split train and <class 'inspect._empty'>

See: https://github.com/huggingface/datasets/actions/runs/9869660902/job/27254359863?pr=7015
```
Building the MDX files:  97%|█████████▋| 58/60 [00:00<00:00, 91.94it/s] 
Traceback (most recent call last):
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 197, in build_mdx_files
    content, new_anchors, source_files, errors = resolve_autodoc(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 123, in resolve_autodoc
    doc = autodoc(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/autodoc.py", line 499, in autodoc
    method_doc, check = document_object(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/autodoc.py", line 395, in document_object
    signature = format_signature(obj)
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/autodoc.py", line 126, in format_signature
    if param.default != inspect._empty:
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/datasets/splits.py", line 136, in __ne__
    return not self.__eq__(other)
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/datasets/splits.py", line 379, in __eq__
    raise ValueError(f"Equality not supported between split {self} and {other}")
ValueError: Equality not supported between split train and <class 'inspect._empty'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/datasets/datasets/.venv/bin/doc-builder", line 8, in <module>
    sys.exit(main())
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/commands/doc_builder_cli.py", line 47, in main
    args.func(args)
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/commands/build.py", line 102, in build_command
    build_doc(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 367, in build_doc
    anchors_mapping, source_files_mapping = build_mdx_files(
  File "/home/runner/work/datasets/datasets/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 230, in build_mdx_files
    raise type(e)(f"There was an error when converting {file} to the MDX format.\n" + e.args[0]) from e
ValueError: There was an error when converting ../datasets/docs/source/package_reference/main_classes.mdx to the MDX format.
Equality not supported between split train and <class 'inspect._empty'>
```

Comments:
Title: `from_generator` does not allow to specify the split name
Author: pminervini (CONTRIBUTOR)
State: closed
Created: 2024-07-09T07:47:58Z

### Describe the bug

I'm building train, dev, and test using `from_generator`; however, in all three cases, the logger prints `Generating train split:`
It's not possible to change the split name since it seems to be hardcoded: https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/generator/generator.py

### Steps to reproduce the bug

```
In [1]: from datasets import Dataset

In [2]: def gen():
   ...:     yield {"pokemon": "bulbasaur", "type": "grass"}
   ...: 

In [3]: ds = Dataset.from_generator(gen)
Generating train split: 1 examples [00:00, 133.89 examples/s]
```

### Expected behavior

It should be possible to specify any split name

### Environment info

- `datasets` version: 2.19.2
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.5
- `huggingface_hub` version: 0.23.3
- PyArrow version: 15.0.0
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0

Comments:
Thanks for reporting, @pminervini.

I agree we should give the option to define the split name.

Indeed, there is a PR that addresses precisely this issue:
- #7015

I am reviewing it.

Booom! thank you guys :)
Title: Register `.zstd` extension for zstd-compressed files
Author: polinaeterna (CONTRIBUTOR)
State: closed
Created: 2024-07-08T12:39:50Z

For example, https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0 dataset files have `.zstd` extension which is currently ignored (only `.zst` is registered).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7032). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@albertvillanova hm I don't know tbh, it's just that "mlfoundations/dclm-baseline-1.0" dataset contains [files](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0/tree/main/global-shard_01_of_10/local-shard_0_of_10) with this extension and these files seem to be valid 

not sure why CI is failing but seems to be unrelated to this pr? can I merge @lhoestq @albertvillanova ?

yes you can merge, the CI failure is unrelated (surely an issue with hub-ci)

ah why not, you could try opening a PR

btw there is a channel with them at (internal) https://app.slack.com/client/T1RCG4490/C079AKTV11P if you want to let them know

@lhoestq, your previous comment was addressed to me or Polina?

@polinaeterna let me know if it is OK for you.

I opened https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0/discussions/7

Should we close this PR then?
Title: CI quality is broken: use ruff check instead
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-07-08T11:42:24Z

CI quality is broken: https://github.com/huggingface/datasets/actions/runs/9838873879/job/27159697027
```
error: `ruff <path>` has been removed. Use `ruff check <path>` instead.
```

Comments:
Title: Add option to disable progress bar when reading a dataset ("Loading dataset from disk")
Author: yuvalkirstain (NONE)
State: closed
Created: 2024-07-06T05:43:37Z
Labels: enhancement

### Feature request

Add an option in load_from_disk to disable the progress bar even if the number of files is larger than 16.

### Motivation

I am reading a lot of datasets that it creates lots of logs.

<img width="1432" alt="image" src="https://github.com/huggingface/datasets/assets/57996478/8d4bbf03-6b89-44b6-937c-932f01b4eb2a">


### Your contribution

Seems like an easy fix to make.  I can create a PR if necessary.

Comments:
You can disable progress bars for all of `datasets` with `disable_progress_bars`. [Link](https://huggingface.co/docs/datasets/en/package_reference/utilities#datasets.enable_progress_bars)

So you could do something like:

```python
from datasets import load_from_disk, enable_progress_bars, disable_progress_bars

disable_progress_bars()
# Your code
load_from_disk(....)

enable_progress_bars()
```


Thank you! Closing the issue.
Title: load_dataset on AWS lambda throws OSError(30, 'Read-only file system') error
Author: sugam-nexusflow (NONE)
State: open
Created: 2024-07-04T19:15:16Z

### Describe the bug

I'm using AWS lambda to run a python application. I run the `load_dataset` function with cache_dir="/tmp" and is still throws the OSError(30, 'Read-only file system') error. Is even updated all the HF envs to point to /tmp dir but the issue still persists. I can confirm that the I can write to /tmp directory. 

### Steps to reproduce the bug

```python
d = load_dataset(
      path=hugging_face_link,
      split=split,
      token=token,
      cache_dir="/tmp/hugging_face_cache",
 )
```

### Expected behavior

Everything written to the file system as part of the load_datasets function should be in the /tmp directory. 

### Environment info

datasets version: 2.16.1
Platform: Linux-5.10.216-225.855.amzn2.x86_64-x86_64-with-glibc2.26
Python version: 3.11.9
huggingface_hub version: 0.19.4
PyArrow version: 16.1.0
Pandas version: 2.2.2
fsspec version: 2023.10.0

Comments:
hi ! can you share the full stack trace ? this should help locate what files is not written in the cache_dir
Title: Streaming dataset not returning data
Author: johnwee1 (NONE)
State: open
Created: 2024-07-04T07:21:47Z

### Describe the bug

I'm deciding to post here because I'm still not sure what the issue is, or if I am using IterableDatasets wrongly.
I'm following the guide on here https://huggingface.co/learn/cookbook/en/fine_tuning_code_llm_on_single_gpu pretty much to a tee and have verified that it works when I'm fine-tuning on the provided dataset.

However, I'm doing some data preprocessing steps (filtering out entries), when I try to swap out the dataset for mine, it fails to train. However, I eventually fixed this by simply setting `stream=False` in `load_dataset`.

Coud this be some sort of network / firewall issue I'm facing?

### Steps to reproduce the bug

I made a post with greater description about how I reproduced this problem before I found my workaround: https://discuss.huggingface.co/t/problem-with-custom-iterator-of-streaming-dataset-not-returning-anything/94551

Here is the problematic dataset snippet, which works when streaming=False (and with buffer keyword removed from shuffle)

```
commitpackft = load_dataset(
    "chargoddard/commitpack-ft-instruct", split="train", streaming=True
).filter(lambda example: example["language"] == "Python")
 
 
def form_template(example):
    """Forms a template for each example following the alpaca format for CommitPack"""
    example["content"] = (
        "### Human: " + example["instruction"] + " " + example["input"] + " ### Assistant: " + example["output"]
    )
    return example
 
 
dataset = commitpackft.map(
    form_template,
    remove_columns=["id", "language", "license", "instruction", "input", "output"],
).shuffle(
    seed=42, buffer_size=10000
)  # remove everything since its all inside "content" now
validation_data = dataset.take(4000)
train_data = dataset.skip(4000)
```

The annoying part about this is that it only fails during training and I don't know when it will fail, except that it always fails during evaluation.

### Expected behavior

The expected behavior is that I should be able to get something from the iterator when called instead of getting nothing / stuck in a loop somewhere.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-5.4.0-121-generic-x86_64-with-glibc2.31
- Python version: 3.11.7
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0

Comments:
Title: There is dead code after we require pyarrow >= 15.0.0
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-07-03T08:52:57Z
Labels: maintenance

There are code lines specific for pyarrow versions < 15.0.0.

However, we require pyarrow >= 15.0.0 since the merge of PR:
- #6892

Those code lines are now dead code and should be removed.

Comments:
Title: Casting list array to fixed size list raises error
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-07-03T07:54:49Z
Labels: bug

When trying to cast a list array to fixed size list, an AttributeError is raised:
> AttributeError: 'pyarrow.lib.FixedSizeListType' object has no attribute 'length'

Steps to reproduce the bug:
```python
import pyarrow as pa
from datasets.table import array_cast

arr = pa.array([[0, 1]])
array_cast(arr, pa.list_(pa.int64(), 2))
```

Stack trace:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-6cb90a1d8216> in <module>
      3 
      4 arr = pa.array([[0, 1]])
----> 5 array_cast(arr, pa.list_(pa.int64(), 2))

~/huggingface/datasets/src/datasets/table.py in wrapper(array, *args, **kwargs)
   1802             return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
   1803         else:
-> 1804             return func(array, *args, **kwargs)
   1805 
   1806     return wrapper

~/huggingface/datasets/src/datasets/table.py in array_cast(array, pa_type, allow_primitive_to_str, allow_decimal_to_str)
   1920                 else:
   1921                     array_values = array.values[
-> 1922                         array.offset * pa_type.length : (array.offset + len(array)) * pa_type.length
   1923                     ]
   1924                     return pa.FixedSizeListArray.from_arrays(_c(array_values, pa_type.value_type), pa_type.list_size)

AttributeError: 'pyarrow.lib.FixedSizeListType' object has no attribute 'length'
```

Comments:
Title: `load_dataset` fails to load dataset saved by `save_to_disk`
Author: sliedes (NONE)
State: open
Created: 2024-07-01T12:19:19Z

### Describe the bug

This code fails to load the dataset it just saved:

```python
from datasets import load_dataset
from transformers import AutoTokenizer

MODEL = "google-bert/bert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(MODEL)

dataset = load_dataset("yelp_review_full")

def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)

tokenized_datasets = dataset.map(tokenize_function, batched=True)
tokenized_datasets.save_to_disk("dataset")

tokenized_datasets = load_dataset("dataset/")  # raises
```

It raises `ValueError: Couldn't infer the same data file format for all splits. Got {NamedSplit('train'): ('arrow', {}), NamedSplit('test'): ('json', {})}`.

I believe this bug is caused by the [logic that tries to infer dataset format](https://github.com/huggingface/datasets/blob/9af8dd3de7626183a9a9ec8973cebc672d690400/src/datasets/load.py#L556). It counts the most common file extension. However, a small dataset can fit in a single `.arrow` file and have two JSON metadata files, causing the format to be inferred as JSON:

```shell
$ ls -l dataset/test
-rw-r--r-- 1 sliedes sliedes 191498784 Jul  1 13:55 data-00000-of-00001.arrow
-rw-r--r-- 1 sliedes sliedes      1730 Jul  1 13:55 dataset_info.json
-rw-r--r-- 1 sliedes sliedes       249 Jul  1 13:55 state.json
```

### Steps to reproduce the bug

Execute the code above.

### Expected behavior

The dataset is loaded successfully.

### Environment info

- `datasets` version: 2.20.0
- Platform: Linux-6.9.3-arch1-1-x86_64-with-glibc2.39
- Python version: 3.12.4
- `huggingface_hub` version: 0.23.4
- PyArrow version: 16.1.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.5.0

Comments:
In my case the error was:
```
ValueError: You are trying to load a dataset that was saved using `save_to_disk`. Please use `load_from_disk` instead.
```
Did you try `load_from_disk`?

More generally, any reason there is no API consistency between save_to_disk and push_to_hub ? 

Would be nice to be able to save_to_disk and then upload manually to the hub and load_dataset (which works in some situations but not all)...

I have the exact same problem !

`load_from_disk` managed to load the dataset, but the bug with `load_dataset` needs to be fixed. 

any update ? I need some function from load dataset like num_proc, or streaming to optimize ram, ... but got this error
Title: `drop_duplicates` method
Author: MohamedAliRashad (NONE)
State: open
Created: 2024-07-01T09:01:06Z
Labels: duplicate

enhancement

### Feature request

`drop_duplicates` method for huggingface datasets (similiar in simplicity to the `pandas` one)

### Motivation

Ease of use

### Your contribution

I don't think i am good enough to help

Comments:
There is an open issue #2514 about this which also proposes solutions.
Title: CI is broken for faiss tests on Windows: node down: Not properly terminated
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-07-01T06:40:03Z
Labels: maintenance

Faiss tests on Windows make the CI run indefinitely until maximum execution time (360 minutes) is reached.

See: https://github.com/huggingface/datasets/actions/runs/9712659783
```
test (integration, windows-latest, deps-minimum)
The job running on runner GitHub Actions 60 has exceeded the maximum execution time of 360 minutes.
test (integration, windows-latest, deps-latest)
The job running on runner GitHub Actions 238 has exceeded the maximum execution time of 360 minutes.
```
```
____________________________ tests/test_search.py _____________________________
[gw1] win32 -- Python 3.8.10 C:\hostedtoolcache\windows\Python\3.8.10\x64\python.exe
worker 'gw1' crashed while running 'tests/test_search.py::IndexableDatasetTest::test_add_faiss_index'
____________________________ tests/test_search.py _____________________________
[gw2] win32 -- Python 3.8.10 C:\hostedtoolcache\windows\Python\3.8.10\x64\python.exe
worker 'gw2' crashed while running 'tests/test_search.py::IndexableDatasetTest::test_add_faiss_index'
```
```
tests/test_search.py::IndexableDatasetTest::test_add_faiss_index 
[gw0] node down: Not properly terminated
[gw0] FAILED tests/test_search.py::IndexableDatasetTest::test_add_faiss_index 

replacing crashed worker gw0

tests/test_search.py::IndexableDatasetTest::test_add_faiss_index 
[gw1] node down: Not properly terminated
[gw1] FAILED tests/test_search.py::IndexableDatasetTest::test_add_faiss_index 

replacing crashed worker gw1

tests/test_search.py::IndexableDatasetTest::test_add_faiss_index 
[gw2] node down: Not properly terminated
[gw2] FAILED tests/test_search.py::IndexableDatasetTest::test_add_faiss_index 

replacing crashed worker gw2
```

Comments:
Title: Re-enable raising error from huggingface-hub FutureWarning in CI
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-06-28T07:23:40Z
Labels: maintenance

Re-enable raising error from huggingface-hub FutureWarning in CI, which was disabled by PR:
- #6876

Note that this can only be done once transformers releases the fix:
- https://github.com/huggingface/transformers/pull/31007

Comments:
Title: CI is broken after ruff-0.5.0: E721
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-06-28T05:03:28Z
Labels: maintenance

After ruff-0.5.0 release (https://github.com/astral-sh/ruff/releases/tag/0.5.0), our CI is broken due to E721 rule. 

See: https://github.com/huggingface/datasets/actions/runs/9707641618/job/26793170961?pr=6983
> src/datasets/features/features.py:844:12: E721 Use `is` and `is not` for type comparisons, or `isinstance()` for isinstance checks

Comments:
Title: Datasetbuilder Local Download FileNotFoundError
Author: purefall (NONE)
State: open
Created: 2024-06-25T15:02:34Z

### Describe the bug

So I was trying to download a dataset and save it as parquet and I follow the [tutorial](https://huggingface.co/docs/datasets/filesystems#download-and-prepare-a-dataset-into-a-cloud-storage) of Huggingface. However, during the excution I face a FileNotFoundError.

I debug the code and it seems there is a bug there:
So first it creates a .incomplete folder and before moving its contents the following code deletes the directory
[Code](https://github.com/huggingface/datasets/blob/98fdc9e78e6d057ca66e58a37f49d6618aab8130/src/datasets/builder.py#L984)
hence as a result I face with: 

``` FileNotFoundError: [Errno 2] No such file or directory: '~/data/Parquet/.incomplete '```

### Steps to reproduce the bug

```
from datasets import load_dataset_builder
from pathlib import Path

parquet_dir = "~/data/Parquet/" 
Path(parquet_dir).mkdir(parents=True, exist_ok=True)
builder = load_dataset_builder(
    "rotten_tomatoes",
)
builder.download_and_prepare(parquet_dir, file_format="parquet")
```

### Expected behavior

Downloads the files and saves as parquet

### Environment info

Ubuntu,
Python 3.10
```
datasets 2.19.1
```

Comments:
Ok it seems the solution is to use the directory string without the trailing "/" which in my case  as: 

`parquet_dir = "~/data/Parquet" `

Still i think this is a weird behavior...
Title: IterableDataset: Unsupported ScalarType BFloat16
Author: stoical07 (NONE)
State: closed
Created: 2024-06-25T14:43:26Z

### Describe the bug

`IterableDataset.from_generator` crashes when using BFloat16:

```
  File "/usr/local/lib/python3.11/site-packages/datasets/utils/_dill.py", line 169, in _save_torchTensor
    args = (obj.detach().cpu().numpy(),)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^

TypeError: Got unsupported ScalarType BFloat16
```

### Steps to reproduce the bug

```python
import torch
from datasets import IterableDataset


def demo(x):
    yield {"x": x}


x = torch.tensor([1.], dtype=torch.bfloat16)

dataset = IterableDataset.from_generator(
    demo,
    gen_kwargs=dict(x=x),
)

example = next(iter(dataset))
print(example)
```

### Expected behavior

Code sample should print:
```python
{'x': tensor([1.], dtype=torch.bfloat16)}
```

### Environment info

```
datasets==2.20.0
torch==2.2.2
```

Comments:
@lhoestq Thank you for merging #6607, but unfortunately the issue persists for `IterableDataset` :pensive: 

Hi ! I opened https://github.com/huggingface/datasets/pull/7002 to fix this bug

Amazing, thank you so much @lhoestq! :pray:
Title: CI is broken for tests using hf-internal-testing/librispeech_asr_dummy
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-06-25T07:55:44Z
Labels: maintenance

CI is broken: https://github.com/huggingface/datasets/actions/runs/9657882317/job/26637998686?pr=6996
```
FAILED tests/test_inspect.py::test_get_dataset_config_names[hf-internal-testing/librispeech_asr_dummy-expected4] - AssertionError: assert ['clean'] == ['clean', 'other']
  
  Right contains one more item: 'other'
  
  Full diff:
    [
        'clean',
  -     'other',
    ]
FAILED tests/test_inspect.py::test_get_dataset_default_config_name[hf-internal-testing/librispeech_asr_dummy-None] - AssertionError: assert 'clean' is None
```

Note that repository was recently converted to Parquet: https://huggingface.co/datasets/hf-internal-testing/librispeech_asr_dummy/commit/5be91486e11a2d616f4ec5db8d3fd248585ac07a

Comments:
Title: Problematic rank after calling `split_dataset_by_node` twice
Author: yzhangcs (CONTRIBUTOR)
State: closed
Created: 2024-06-21T14:25:26Z

### Describe the bug

I'm trying to split `IterableDataset` by `split_dataset_by_node`.
But when doing split on a already split dataset, the resulting `rank` is greater than `world_size`.

### Steps to reproduce the bug

Here is the minimal code for reproduction:
```py
>>> from datasets import load_dataset
>>> from datasets.distributed import split_dataset_by_node
>>> dataset = load_dataset('fla-hub/slimpajama-test', split='train', streaming=True)      
>>> dataset = split_dataset_by_node(dataset, 1, 32)
>>> dataset._distributed
DistributedConfig(rank=1, world_size=32)
>>> dataset = split_dataset_by_node(dataset, 1, 15)
>>> dataset._distributed                           
DistributedConfig(rank=481, world_size=480)
```
As you can see, the second rank 481 > 480, which is problematic.

### Expected behavior

I think this error comes from this line @lhoestq
https://github.com/huggingface/datasets/blob/a6ccf944e42c1a84de81bf326accab9999b86c90/src/datasets/iterable_dataset.py#L2943-L2944
We may need to obtain the rank first. Then the above code gives
```py
>>> dataset._distributed                           
DistributedConfig(rank=16, world_size=480)
```

### Environment info

datasets==2.20.0

Comments:
ah yes good catch ! feel free to open a PR with your suggested fix
Title: Convert polars DataFrame back to datasets
Author: ljw20180420 (NONE)
State: closed
Created: 2024-06-19T11:38:48Z
Labels: enhancement

### Feature request

This returns error.
```python
from datasets import Dataset

dsdf = Dataset.from_dict({"x": [[1, 2], [3, 4, 5]], "y": ["a", "b"]})
Dataset.from_polars(dsdf.to_polars())
```
ValueError: Arrow type large_list<item: int64> does not have a datasets dtype equivalent.

### Motivation

When datasets contain Sequence data type, it will be converted to Arrow type large_list. However, the reverse (from large_list to Sequence) does not work.

### Your contribution

No

Comments:
Hi ! Thanks for reporting :)

We don't support `large_list` yet, though it should be added to `Sequence` IMO (maybe with a parameter `large=True` ?)
Title: Support NumPy 2.0
Author: NeilGirdhar (CONTRIBUTOR)
State: closed
Created: 2024-06-18T23:30:22Z
Labels: enhancement

### Feature request

Support NumPy 2.0.

### Motivation

NumPy introduces the Array API, which bridges the gap between machine learning libraries.  Many clients of HuggingFace are eager to start using the Array API.

Besides that, NumPy 2 provides a cleaner interface than NumPy 1.

### Tasks

NumPy 2.0 was released for testing so that libraries could ensure compatibility [since mid-March](https://github.com/numpy/numpy/issues/24300#issuecomment-1986815755).  What needs to be done for HuggingFace to support Numpy 2?

- [x] Fix use of `array`: https://github.com/huggingface/datasets/pull/6976
- [ ] Remove [NumPy version limit](https://github.com/huggingface/datasets/pull/6975): https://github.com/huggingface/datasets/pull/6991

Comments:
Title: IndexError during training with Squad dataset and T5-small model
Author: ramtunguturi36 (NONE)
State: closed
Created: 2024-06-16T07:53:54Z

### Describe the bug

I am encountering an IndexError while training a T5-small model on the Squad dataset using the transformers and datasets libraries. The error occurs even with a minimal reproducible example, suggesting a potential bug or incompatibility.

### Steps to reproduce the bug

1.Install the required libraries: !pip install transformers datasets
2.Run the following code:
!pip install transformers datasets

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, TrainingArguments, Trainer, DataCollatorWithPadding

# Load a small, publicly available dataset
from datasets import load_dataset
dataset = load_dataset("squad", split="train[:100]") # Use a small subset for testing

# Load a pre-trained model and tokenizer
model_name = "t5-small"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Define a basic data collator
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)

# Define training arguments
training_args = TrainingArguments(
    output_dir="./results",
    per_device_train_batch_size=2,
    num_train_epochs=1,
)

# Create a trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    data_collator=data_collator,
)

# Train the model
trainer.train()

### Expected behavior

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
[<ipython-input-23-f13a4b23c001>](https://localhost:8080/#) in <cell line: 34>()
     32 
     33 # Train the model
---> 34 trainer.train()

10 frames
[/usr/local/lib/python3.10/dist-packages/datasets/formatting/formatting.py](https://localhost:8080/#) in _check_valid_index_key(key, size)
    427     if isinstance(key, int):
    428         if (key < 0 and key + size < 0) or (key >= size):
--> 429             raise IndexError(f"Invalid key: {key} is out of bounds for size {size}")
    430         return
    431     elif isinstance(key, slice):

IndexError: Invalid key: 42 is out of bounds for size 0

### Environment info

transformers version:4.41.2
datasets version:1.18.4
Python version:3.10.12

Comments:
add remove_unused_columns=False to training_args
https://github.com/huggingface/datasets/issues/6535#issuecomment-1874024704

Closing this issue because it was a reported and fixed in transformers.
Title: Method to load Laion400m
Author: humanely (NONE)
State: open
Created: 2024-06-12T16:04:04Z
Labels: enhancement

### Feature request

Large datasets like Laion400m are provided as embeddings. The provided methods in load_dataset are not straightforward for loading embedding files, i.e. img_emb_XX.npy ; XX = 0 to 99


### Motivation

The trial and experimentation is the key pivot of HF. It would be great if HF can load embeddings files s,ealessly.

### Your contribution

I cam write the loader with some help.

Comments:
Title: Manual downloads should count as downloads
Author: umarbutler (NONE)
State: open
Created: 2024-06-09T04:52:06Z
Labels: enhancement

### Feature request

I would like to request that manual downloads of data files from Hugging Face dataset repositories count as downloads of a dataset. According to the documentation for the Hugging Face Hub, that is currently not the case: https://huggingface.co/docs/hub/en/datasets-download-stats

### Motivation

This would ensure that downloads are accurately reported to end users.

### Your contribution

N/A

Comments:
We're unlikely to add more features/support for datasets with python loading scripts, which include datasets with manual download. Sorry for the inconvenience
Title: My Private Dataset doesn't exist on the Hub or cannot be accessed
Author: wangguan1995 (NONE)
State: closed
Created: 2024-06-06T06:52:19Z

### Describe the bug


```
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1852, in dataset_module_factory
    raise DatasetNotFoundError(msg + f" at revision '{revision}'" if revision else msg)
datasets.exceptions.DatasetNotFoundError: Dataset 'xxx' doesn't exist on the Hub or cannot be accessed
>>> dataset = load_dataset("xxxx", token=True)

404 error 404 Client Error. (Request ID: Root=xxxx)

Repository Not Found for url: https://huggingface.co/api/datasets/xxx/xxx.
Please make sure you specified the correct `repo_id` and `repo_type`.
If you are trying to access a private or gated repo, make sure you are authenticated.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 2593, in load_dataset
    builder_instance = load_dataset_builder(
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 2265, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1910, in dataset_module_factory
    raise e1 from None
  File "/root/miniconda3/envs/gino_conda/lib/python3.9/site-packages/datasets/load.py", line 1852, in dataset_module_factory
    raise DatasetNotFoundError(msg + f" at revision '{revision}'" if revision else msg)
datasets.exceptions.DatasetNotFoundError: Dataset 'xxx' doesn't exist on the Hub or cannot be accessed
```

### Steps to reproduce the bug

123

### Expected behavior

123

### Environment info

123

Comments:
I can load public dataset, but for my private dataset it fails

https://huggingface.co/docs/datasets/upload_dataset

I have checked the API HTTP link. Repository Not Found for url: https://huggingface.co/api/datasets/xxx/xxx.

![image](https://github.com/huggingface/datasets/assets/39621324/4aceef59-0c65-4161-9665-676d25d73225)

It just works fine.

It seems that everything is in a mass huh....

![image](https://github.com/huggingface/datasets/assets/39621324/fb2fe12c-4f0a-4bf6-9656-63ba50347b10)


https://huggingface.co/datasets/rajpurkar/squad/blob/main/squad.py fails again

https://github.com/huggingface/datasets/blob/main/templates/new_dataset_script.py#L81 can not use this, too complex. I just need a def to load my file to a dict

I am facing the same issue. Did you find a fix?

You should authenticate to be able to access private or gated repos: https://huggingface.co/docs/hub/datasets-gated#access-gated-datasets-as-a-user
Title: `Dataset.with_format` behaves inconsistently with documentation
Author: iansheng (NONE)
State: closed
Created: 2024-06-04T09:18:32Z
Labels: documentation

### Describe the bug

The actual behavior of the interface `Dataset.with_format` is inconsistent with the documentation.
https://huggingface.co/docs/datasets/use_with_pytorch#n-dimensional-arrays
https://huggingface.co/docs/datasets/v2.19.0/en/use_with_tensorflow#n-dimensional-arrays

> If your dataset consists of N-dimensional arrays, you will see that by default they are considered as nested lists.
> In particular, a PyTorch formatted dataset outputs nested lists instead of a single tensor.
> A TensorFlow formatted dataset outputs a RaggedTensor instead of a single tensor.

But I get a single tensor by default, which is  inconsistent with the description.

Actually the current behavior seems more reasonable to me. Therefore, the document needs to be modified.

### Steps to reproduce the bug

```python
>>> from datasets import Dataset
>>> data = [[[1, 2],[3, 4]],[[5, 6],[7, 8]]]
>>> ds = Dataset.from_dict({"data": data})
>>> ds = ds.with_format("torch")
>>> ds[0]
{'data': tensor([[1, 2],
        [3, 4]])}
>>> ds = ds.with_format("tf")
>>> ds[0]
{'data': <tf.Tensor: shape=(2, 2), dtype=int64, numpy=
array([[1, 2],
       [3, 4]])>}
```

### Expected behavior

```python
>>> from datasets import Dataset
>>> data = [[[1, 2],[3, 4]],[[5, 6],[7, 8]]]
>>> ds = Dataset.from_dict({"data": data})
>>> ds = ds.with_format("torch")
>>> ds[0]
{'data': [tensor([1, 2]), tensor([3, 4])]}
>>> ds = ds.with_format("tf")
>>> ds[0]
{'data': <tf.RaggedTensor [[1, 2], [3, 4]]>}
```

### Environment info

datasets==2.19.1
torch==2.1.0
tensorflow==2.13.1

Comments:
Hi ! It seems the documentation was outdated in this paragraph

I fixed it here: https://github.com/huggingface/datasets/pull/6956

Fixed.
Title: load_dataset error
Author: frederichen01 (NONE)
State: closed
Created: 2024-06-04T01:24:45Z

### Describe the bug

Why does the program get stuck when I use load_dataset method, and it still gets stuck after loading for several hours? In fact, my json file is only 21m, and I can load it in one go using open('', 'r').

### Steps to reproduce the bug

1. pip install datasets==2.19.2
2. from datasets import Dataset, DatasetDict, NamedSplit, Split, load_dataset
3. data = load_dataset('json', data_files='train.json')

### Expected behavior

It is able to load my json correctly

### Environment info

datasets==2.19.2

Comments:
Hi, @lion-ops.

In our Continuous Integration we have many tests on loading JSON files and all of them work properly.

Could you please share your "train.json" file, so that we can try to reproduce the issue you have? 

> Hi, @lion-ops.
> 
> In our Continuous Integration we have many tests on loading JSON files and all of them work properly.
> 
> Could you please share your "train.json" file, so that we can try to reproduce the issue you have?

Thank you for your reply. I can load it normally in another server. Is it possible that the disk of my server is a network disk in the LAN, so it will be downloaded from the LAN and get stuck?
Title: to_tf_dataset: Visible devices cannot be modified after being initialized
Author: logasja (NONE)
State: open
Created: 2024-06-03T18:10:57Z

### Describe the bug

When trying to use to_tf_dataset with a custom data_loader collate_fn when I use parallelism I am met with the following error as many times as number of workers there were in ``num_workers``.

  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/multiprocess/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/multiprocess/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/datasets/utils/tf_utils.py", line 438, in worker_loop
    tf.config.set_visible_devices([], "GPU")  # Make sure workers don't try to allocate GPU memory
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/tensorflow/python/framework/config.py", line 566, in set_visible_devices
    context.context().set_visible_devices(devices, device_type)
  File "/opt/miniconda/envs/env/lib/python3.11/site-packages/tensorflow/python/eager/context.py", line 1737, in set_visible_devices
    raise RuntimeError(
RuntimeError: Visible devices cannot be modified after being initialized

### Steps to reproduce the bug

1. Download a dataset using HuggingFace load_dataset
2. Define a function that transforms the data in some way to be used in the collate_fn argument
3. Provide a ``batch_size`` and ``num_workers`` value in the ``to_tf_dataset`` function
4. Either retrieve directly or use tfds benchmark to test the dataset

``` python
from datasets import load_datasets
import tensorflow_datasets as tfds
from keras_cv.layers import Resizing

def data_loader(examples):
      x = Resizing(examples[0]['image'], 256, 256, crop_to_aspect_ratio=True)
      return {X[0]: x}

ds = load_datasets("logasja/FDF", split="test")

ds = ds.to_tf_dataset(collate_fn=data_loader, batch_size=16, num_workers=2)

tfds.benchmark(ds)
```

### Expected behavior

Use multiple processes to apply transformations from the collate_fn to the tf dataset on the CPU.

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-6.5.0-1023-oracle-x86_64-with-glibc2.35
- Python version: 3.11.8
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0

Comments:
Title: Release 2.19.2
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-06-03T05:01:50Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6943). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Import sorting is disabled by flake8 noqa directive after switching to ruff linter
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-06-02T09:43:34Z
Labels: maintenance

When we switched to `ruff` linter in PR:
- #5519

import sorting was disabled in all files containing the `# flake8: noqa` directive
- https://github.com/astral-sh/ruff/issues/11679

We should re-enable import sorting on those files.

Comments:
Title: Supporting FFCV: Fast Forward Computer Vision
Author: Luciennnnnnn (NONE)
State: open
Created: 2024-06-01T05:34:52Z
Labels: enhancement

### Feature request

Supporting FFCV, https://github.com/libffcv/ffcv

### Motivation

According to the benchmark, FFCV seems to be fastest image loading method.

### Your contribution

no

Comments:
Title: Enable Sharding to Equal Sized Shards
Author: yuvalkirstain (NONE)
State: open
Created: 2024-05-31T21:55:50Z
Labels: enhancement

### Feature request

Add an option when sharding a dataset to have all shards the same size. Will be good to provide both an option of duplication, and by truncation.

### Motivation

Currently the behavior of sharding is "If n % i == l, then the first l shards will have length (n // i) + 1, and the remaining shards will have length (n // i).". However, when using FSDP we want the shards to have the same size. This requires the user to manually handle this situation, but it will be nice if we had an option to shard the dataset into equally sized shards. 

### Your contribution

For now just a PR. I can also add code that does what is needed, but probably not efficient.
Shard to equal size by duplication:
```
remainder = len(dataset) % num_shards
num_missing_examples = num_shards - remainder
duplicated = dataset.select(list(range(num_missing_examples)))
dataset = concatenate_datasets([dataset, duplicated])
shard = dataset.shard(num_shards, shard_idx)
```
Or by truncation:
```
shard = dataset.shard(num_shards, shard_idx)
num_examples_per_shard = len(dataset) // num_shards
shard = shard.select(list(range(num_examples_per_shard)))
```

Comments:
Title: ExpectedMoreSplits error when using data_dir
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-05-31T15:08:42Z
Labels: bug

As reported by @regisss, an `ExpectedMoreSplits` error is raised when passing `data_dir`:
```python
from datasets import load_dataset

dataset = load_dataset(
    "lvwerra/stack-exchange-paired",
    split="train",
    cache_dir=None,
    data_dir="data/rl",
)
```
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2609, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1027, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1140, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/info_utils.py", line 92, in verify_splits
    raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))
datasets.utils.info_utils.ExpectedMoreSplits: {'test'}
```

Comments:
Title: Fix expected splits when passing data_files or dir
Author: lhoestq (MEMBER)
State: closed
Created: 2024-05-31T11:04:22Z

reported on slack:

The following code snippet gives an error with v2.19 but not with v2.18:
from datasets import load_dataset
```
dataset = load_dataset(
    "lvwerra/stack-exchange-paired",
    split="train",
    cache_dir=None,
    data_dir="data/rl",
)
```
and the error is:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/datasets/load.py", line 2609, in load_dataset
    builder_instance.download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1027, in download_and_prepare
    self._download_and_prepare(
  File "/usr/local/lib/python3.10/dist-packages/datasets/builder.py", line 1140, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/usr/local/lib/python3.10/dist-packages/datasets/utils/info_utils.py", line 92, in verify_splits
    raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))
datasets.utils.info_utils.ExpectedMoreSplits: {'test'}
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6938). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

fix is included in https://github.com/huggingface/datasets/pull/6925
Title: save_to_disk() freezes when saving on s3 bucket with multiprocessing
Author: ycattan (NONE)
State: open
Created: 2024-05-30T16:48:39Z

### Describe the bug

I'm trying to save a `Dataset` using the `save_to_disk()` function with:
- `num_proc > 1`
- `dataset_path` being a s3 bucket path e.g. "s3://{bucket_name}/{dataset_folder}/"

The hf progress bar shows up but the saving does not seem to start. 
When using one processor only (`num_proc=1`), everything works fine.
When saving the dataset on local disk (as opposed to s3 bucket) with `num_proc > 1`, everything works fine.

Thank you for your help! :)

### Steps to reproduce the bug

I tried without any storage options:

```
from datasets import load_dataset

sandbox_ds = load_dataset("openai_humaneval")
sandbox_ds["test"].save_to_disk(
    "s3://bucket-name/test_multiprocessing_saving/",
    num_proc=4,
)
```


and with the specific s3fs storage options: 
```
from datasets import load_dataset
from s3fs import S3FileSystem

def get_s3fs():
    return S3FileSystem()

sandbox_ds = load_dataset("openai_humaneval")
sandbox_ds["test"].save_to_disk(
    "s3://bucket-name/test_multiprocessing_saving/",
    num_proc=4,
    storage_options=get_s3fs().storage_options, # also tried: storage_options=S3FileSystem().storage_options
)
```

I'm guessing I might use `storage_options` parameter wrongly, but I didn't find anything online that made it work.

**NB**: Behavior is the same when trying to save the whole `DatasetDict`.

### Expected behavior

Progress bar fills in and saving is carried out.

### Environment info

`datasets==2.18.0`

Comments:
I got the same issue. Any updates so far for this issue?

Same here. Any updates?

+1, experiencing this as well
Title: Support for pathlib.Path in datasets 2.19.0
Author: lamyiowce (NONE)
State: open
Created: 2024-05-30T12:53:36Z

### Describe the bug

After the recent update of `datasets`, Dataset.save_to_disk does not accept a pathlib.Path anymore. It was supported in 2.18.0 and previous versions. Is this intentional? Was it supported before only because of a Python dusk-typing miracle?

### Steps to reproduce the bug

```
from datasets import Dataset
import pathlib

path = pathlib.Path("./my_out_path")
Dataset.from_dict(
    {"text": ["hello world"], "label": [777], "split": ["train"]}
.save_to_disk(path)
```
This results in an error when using datasets 2.19:
```
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/Users/jb/scratch/venv/lib/python3.11/site-packages/datasets/arrow_dataset.py", line 1515, in save_to_disk
    fs, _ = url_to_fs(dataset_path, **(storage_options or {}))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jb/scratch/venv/lib/python3.11/site-packages/fsspec/core.py", line 383, in url_to_fs
    chain = _un_chain(url, kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jb/scratch/venv/lib/python3.11/site-packages/fsspec/core.py", line 323, in _un_chain
    if "::" in path
       ^^^^^^^^^^^^
TypeError: argument of type 'PosixPath' is not iterable
```

Converting to str works, however.
```
Dataset.from_dict(
     {"text": ["hello world"], "label": [777], "split": ["train"]}
).save_to_disk(str(path))
```

### Expected behavior

My dataset gets saved to disk without an error.

### Environment info

aiohttp==3.9.5
aiosignal==1.3.1
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
datasets==2.19.0
dill==0.3.8
filelock==3.14.0
frozenlist==1.4.1
fsspec==2024.3.1
huggingface-hub==0.23.2
idna==3.7
multidict==6.0.5
multiprocess==0.70.16
numpy==1.26.4
packaging==24.0
pandas==2.2.2
pyarrow==16.1.0
pyarrow-hotfix==0.6
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
requests==2.32.3
six==1.16.0
tqdm==4.66.4
typing_extensions==4.12.0
tzdata==2024.1
urllib3==2.2.1
xxhash==3.4.1
yarl==1.9.4

Comments:
+1 I just noticed this when I tried to update `datasets` today.

The same issue, I also get error.