sjwsjw/github-issues huggingface dataset:
GitHub issue: ValueError
State: closed

### Describe the bug

ValueError                                Traceback (most recent call last)
[<ipython-input-11-9b99bc80ec23>](https://localhost:8080/#) in <cell line: 11>()
      9 import numpy as np
     10 import matplotlib.pyplot as plt
---> 11 from datasets import DatasetDict, Dataset
     12 from transformers import AutoTokenizer, AutoModelForSequenceClassification
     13 from transformers import Trainer, TrainingArguments

5 frames
[/usr/local/lib/python3.10/dist-packages/datasets/__init__.py](https://localhost:8080/#) in <module>
     16 __version__ = "2.17.0"
     17 
---> 18 from .arrow_dataset import Dataset
     19 from .arrow_reader import ReadInstruction
     20 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in <module>
     65 
     66 from . import config
---> 67 from .arrow_reader import ArrowReader
     68 from .arrow_writer import ArrowWriter, OptimizedTypedSequence
     69 from .data_files import sanitize_patterns

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_reader.py](https://localhost:8080/#) in <module>
     27 
     28 import pyarrow as pa
---> 29 import pyarrow.parquet as pq
     30 from tqdm.contrib.concurrent import thread_map
     31 

[/usr/local/lib/python3.10/dist-packages/pyarrow/parquet/__init__.py](https://localhost:8080/#) in <module>
     18 # flake8: noqa
     19 
---> 20 from .core import *

[/usr/local/lib/python3.10/dist-packages/pyarrow/parquet/core.py](https://localhost:8080/#) in <module>
     34 import pyarrow as pa
     35 import pyarrow.lib as lib
---> 36 import pyarrow._parquet as _parquet
     37 
     38 from pyarrow._parquet import (ParquetReader, Statistics,  # noqa

/usr/local/lib/python3.10/dist-packages/pyarrow/_parquet.pyx in init pyarrow._parquet()

ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Expected behavior

Resolve the binary incompatibility

### Environment info

Google Colab Note book
GitHub issue: Default config for squad is incorrect
State: open

### Describe the bug

If you download Squad, it will download the plain_text version, but the config still specifies "default", so if you set the offline mode the cache will try to look it up according to the config_id which is "default" and this will say;
ValueError: Couldn't find cache for squad for config 'default'
Available configs in the cache: ['plain_text']

### Steps to reproduce the bug

1. export HF_DATASETS_OFFLINE=0
2. load_dataset("squad")
3. export HF_DATASETS_OFFLINE=1
4. load_dataset("squad")

### Expected behavior

We should change the config_name I guess? 

### Environment info

linux, latest version of datasets
GitHub issue: Allow SplitDict setitem to replace existing SplitInfo
State: closed

Fix this code provided by @clefourrier 

```python
import datasets
import os

token = os.getenv("TOKEN")

results = datasets.load_dataset("gaia-benchmark/results_public", "2023", token=token, download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)
results["test"] = datasets.Dataset.from_list([row for row in results["test"] if row["model"] != "StateFlow"])
results["test"].push_to_hub("gaia-benchmark/results_public", "2023", token=token, split="test")
```

```
ValueError                                Traceback (most recent call last)
Cell In[43], line 1
----> 1 results["test"].push_to_hub("gaia-benchmark/results_public", "2023", token=token, split="test")

File ~/miniconda3/envs/default310/lib/python3.10/site-packages/datasets/arrow_dataset.py:5498, in Dataset.push_to_hub(self, repo_id, config_name, split, private, token, branch, max_shard_size, num_shards, embed_external_files)
   5496         repo_info.dataset_size = (repo_info.dataset_size or 0) + dataset_nbytes
   5497         repo_info.size_in_bytes = repo_info.download_size + repo_info.dataset_size
-> 5498         repo_info.splits[split] = SplitInfo(
   5499             split, num_bytes=dataset_nbytes, num_examples=len(self), dataset_name=dataset_name
   5500         )
   5501         info_to_dump = repo_info
   5502 # create the metadata configs if it was uploaded with push_to_hub before metadata configs existed

File ~/miniconda3/envs/default310/lib/python3.10/site-packages/datasets/splits.py:541, in SplitDict.__setitem__(self, key, value)
    539     raise ValueError(f"Cannot add elem. (key mismatch: '{key}' != '{value.name}')")
    540 if key in self:
--> 541     raise ValueError(f"Split {key} already present")
    542 super().__setitem__(key, value)

ValueError: Split test already present
```
GitHub issue: `write_examples_on_file` and `write_batch` are broken in `ArrowWriter`
State: closed

### Describe the bug

`write_examples_on_file` and `write_batch` are broken in `ArrowWriter` since #6636. The order between the columns and the schema is not preserved anymore. So these functions don't work anymore unless the order happens to align well.

### Steps to reproduce the bug

Try to do `write_batch` with anything that has many columns, and it's likely to break.

### Expected behavior

I expect these functions to work, instead of it trying to cast a column to its incorrect type.

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-5.15.0-1040-aws-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.19.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
GitHub issue: Import error on Google Colab
State: closed

### Describe the bug

Cannot be imported on Google Colab, the import throws the following error:
ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

1. `! pip install -U datasets`
2. `import datasets`

### Expected behavior

Should be possible to use the library

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0
GitHub issue: Automatic Conversion for uint16/uint32 to Compatible PyTorch Dtypes
State: closed

This PR addresses an issue encountered when utilizing uint16 or uint32 datatypes with datasets, followed by attempting to convert these datasets into PyTorch-compatible formats. Currently, doing so results in a TypeError due to incompatible datatype conversion, as illustrated by the following example:
```python
from datasets import Dataset, Sequence, Value, Features

def gen():
    for i in range(100):
        yield {'seq': list(range(i, i + 20))}

ds = Dataset.from_generator(gen, features=Features({'seq': Sequence(feature=Value(dtype='uint16'), length=-1)}))

ds.set_format('torch')

print(ds[0])
```
This code snippet triggers the following error due to the inability to convert numpy.uint16 arrays to a PyTorch-supported format:
```
TypeError: can't convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.
```

This PR introduces an automatic mechanism to convert np.uint16 and np.uint32 datatypes to np.int64 for seamless compatibility with PyTorch formats, simplifying workflows and improving developer experience by eliminating the need for manual conversion handling.
GitHub issue: Change default compression argument for JsonDatasetWriter
State: closed

Change default compression type from `None` to "infer", to align with pandas' defaults.

Documentation asks the user to supply `to_json_kwargs` with arguments suitable for pandas' `to_json` method. At the same time, while pandas' by default uses ["infer"](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html) for compression, datasets enforce `None` as default. This, likely, confuses user, as they expect the same behaviour, i.e they expect that if they name their output file as "dataset.jsonl.zst" then the compression would be inferred as "zstd" and file will be compressed before writing.

Moreover, while it is probably outside of the scope of this pull request, `compression` argument needs to be capable of taking `dict` as input (along with `str`), as it does in pandas, in order to allow user to specify compression parameters. Current implementation will likely fail with `NotImplementedError`, as it expects either `None` or `str` specifying compression algo.
GitHub issue: [Resumable IterableDataset] Add IterableDataset state_dict
State: closed

A simple implementation of a mechanism to resume an IterableDataset.
It works by restarting at the latest shard and skip samples. It provides fast resuming (though not instantaneous).

Example:

```python
from datasets import Dataset, concatenate_datasets


ds = Dataset.from_dict({"a": range(5)}).to_iterable_dataset(num_shards=3)
ds = concatenate_datasets([ds] * 2)

print(f"{ds.state_dict()=}")
for i, example in enumerate(ds):
    print(example)
    if i == 6:
        state_dict = ds.state_dict()
        print("checkpoint")
ds.load_state_dict(state_dict)
print(f"resuming from checkpoint {ds.state_dict()=}")
for example in ds:
    print(example)
```

returns

```
ds.state_dict()={'ex_iterable_idx': 0, 'ex_iterables': [{'shard_idx': 0, 'shard_example_idx': 0}, {'shard_idx': 0, 'shard_example_idx': 0}]}
{'a': 0}
{'a': 1}
{'a': 2}
{'a': 3}
{'a': 4}
{'a': 0}
{'a': 1}
checkpoint
{'a': 2}
{'a': 3}
{'a': 4}
resuming from checkpoint ds.state_dict()={'ex_iterable_idx': 1, 'ex_iterables': [{'shard_idx': 3, 'shard_example_idx': 0}, {'shard_idx': 0, 'shard_example_idx': 2}]}
{'a': 2}
{'a': 3}
{'a': 4}
```

using torchdata:
```python
from datasets import load_dataset
from torchdata.stateful_dataloader import StatefulDataLoader
my_iterable_dataset = load_dataset("deepmind/code_contests", streaming=True, split="train")
dataloader = StatefulDataLoader(my_iterable_dataset, batch_size=32, num_workers=4)
# save in the middle of training
state_dict = dataloader.state_dict()
# and resume later
dataloader.load_state_dict(state_dict)
```

docs: https://huggingface.co/docs/datasets/main/en/use_with_pytorch#checkpoint-and-resume
GitHub issue: Release not pushed to conda channel
State: closed

### Describe the bug

The github actions step to publish the release 2.17.0 to conda channel has failed due to expired token. Can some one please update the anaconda token rerun the failed action? @albertvillanova ?
 
![image](https://github.com/huggingface/datasets/assets/7138162/1b56ad3d-7643-4778-9cce-4bf531717700)


### Steps to reproduce the bug

Please see this actions [link](https://github.com/huggingface/datasets/actions/runs/7842473662)

### Expected behavior

The action runs successfully and the latest release is pushed to HuggingFace conda channel

### Environment info

Not applicable.
GitHub issue: Error when loading a big local json file
State: open

### Describe the bug

When trying to load big json files from a local directory, `load_dataset` throws the following error

```
Traceback (most recent call last):
  File "/miniconda3/envs/conda-env/lib/python3.10/site-packages/datasets/builder.py", line 1989, in _prepare_split_single
    writer.write_table(table)
  File "miniconda3/envs/conda-env/lib/python3.10/site-packages/datasets/arrow_writer.py", line 573, in write_table
    pa_table = pa_table.combine_chunks()
  File "pyarrow/table.pxi", line 3638, in pyarrow.lib.Table.combine_chunks
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays
```

### Steps to reproduce the bug

1. Download a big file, e.g. `https://dl.fbaipublicfiles.com/dpr/data/retriever/biencoder-nq-train.json.gz`
2. Load it like `data = load_dataset("json", data_files=["nq-train.json"], split="train")`

```python
from datasets import load_dataset

data = load_dataset("json", data_files=["nq-train.json"], split="train")
```

A similarly formatted but smaller file, e.g. e.g. `https://dl.fbaipublicfiles.com/dpr/data/retriever/biencoder-nq-dev.json.gz` is loaded without issues

```python
from datasets import load_dataset

data = load_dataset("json", data_files=["nq-dev.json"], split="train")
```

### Expected behavior

It should load normally

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.18.10-76051810-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
GitHub issue: Slice splits support for datasets.load_from_disk
State: open

### Feature request

Support for slice splits in `datasets.load_from_disk`, similar to how it's already supported for `datasets.load_dataset`.

### Motivation

Slice splits are convienient in a numer of cases - adding support to `datasets.load_from_disk` would make working with local datasets easier and homogenize the APIs of load_from_disk and load_dataset. 

### Your contribution

Sure, if the devs think the feature request is sensible.
GitHub issue: Differently dataset object saved than it is loaded.
State: closed

### Describe the bug

Differently sized object is saved than it is loaded.

### Steps to reproduce the bug

Hi, I save dataset in a following way:
```
        dataset = load_dataset("json",
                               data_files={
                                   "train": os.path.join(input_folder, f"{task_meta_type}_{task_type}_train.jsonl"),
                                   "test": os.path.join(input_folder, f"{task_meta_type}_{task_type}_test.jsonl")})
        print(os.path.join(output_folder, f"{task_meta_type}_{task_type}"))
        print(f"Length of train dataset: {len(dataset['train'])}")
        print(f"Length of test dataset: {len(dataset['test'])}")
        dataset.save_to_disk(os.path.join(output_folder, f"{task_meta_type}_{task_type}"))
```

this yields output
```
.data/hf_dataset/propaganda_zanr
Length of train dataset: 7642
Length of test dataset: 1000
```

Everything looks fine.

Then I load the dataset
```python
from datasets import load_dataset

dataset_path = ".data/hf_dataset/propaganda_zanr"
dataset = load_dataset(dataset_path)

print(f"Length of train dataset: {len(dataset['train'])}")
print(f"Length of test dataset: {len(dataset['test'])}")

```

this prints
```
Generating train split: 1 examples [00:00, 72.10 examples/s]
Generating test split: 1 examples [00:00, 100.69 examples/s]
Length of train dataset: 1
Length of test dataset: 1
```

I dont' understand :(


### Expected behavior

same object is loaded

### Environment info

datasets==2.16.1
GitHub issue: Sign Language Support
State: open

### Feature request

Currently, there are only several Sign Language labels, I would like to propose adding all the Signed Languages as new labels which are described in this ISO standard: https://www.evertype.com/standards/iso639/sign-language.html

### Motivation

Datasets currently only have labels for several signed languages. There are more signed languages in the world. Furthermore, some signed languages that have a lot of online data cannot be found because of this reason (for instance, German Sign Language, and there is no German Sign Language label on huggingface datasets even though there are a lot of readily available sign language datasets exist for German Sign Language, which are used very frequently in Sign Language Processing papers, and models.)

### Your contribution

I can submit a PR for this as well, adding the ISO codes and languages to the labels in datasets.
GitHub issue: 'with_format' is extremely slow when used together with 'interleave_datasets' or 'shuffle' on IterableDatasets
State: open

### Describe the bug

If you:

1. Interleave two iterable datasets together with the interleave_datasets function, or shuffle an iterable dataset
2. Set the output format to torch tensors with .with_format('torch')

Then iterating through the dataset becomes over 100x slower than it is if you don't apply the torch formatting.

### Steps to reproduce the bug

```python
import datasets
import torch
from tqdm import tqdm

rand_a = torch.randn(3,224,224)
rand_b = torch.randn(3,224,224)

a = torch.stack([rand_a] * 1000)
b = torch.stack([rand_b] * 1000)

features = datasets.Features({"tensor": datasets.Array3D(shape=(3,224,224), dtype="float32")})
ds_a = datasets.Dataset.from_dict({"tensor": a}, features=features).to_iterable_dataset()
ds_b = datasets.Dataset.from_dict({"tensor": b}, features=features).to_iterable_dataset()

# Iterating through either dataset with torch formatting is really fast (2000it/s on my machine)
for example in tqdm(ds_a.with_format('torch')):
    pass

# Iterating through either dataset shuffled is also pretty fast (100it/s on my machine)
for example in tqdm(ds_a.shuffle()):
    pass

# Iterating through this interleaved dataset is pretty fast (200it/s on my machine)
ds_fast = datasets.interleave_datasets([ds_a, ds_b])
for example in tqdm(ds_fast):
    pass

# Iterating through either dataset with torch formatting *after shuffling* is really slow... (<2it/s on my machine)
for example in tqdm(ds_a.shuffle().with_format('torch')):
    pass

# Iterating through this torch formatted interleaved dataset is also really slow (<2it/s on my machine)...
ds_slow = datasets.interleave_datasets([ds_a, ds_b]).with_format('torch')
for example in tqdm(ds_slow):
    pass

# Even doing this is way faster!! (70it/s on my machine)
for example in tqdm(ds_fast):
    test = torch.tensor(example['tensor'])

```

### Expected behavior

Applying torch formatting to the interleaved dataset shouldn't increase the time taken to iterate through the dataset by very much, since even explicitly converting every example is over 70x faster than calling .with_format('torch').

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.5.0-15-generic-x86_64-with-glibc2.38
- Python version: 3.11.6
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
GitHub issue: Faster column validation and reordering
State: closed

I work with bioinformatics data and often these tables have thousands and even tens of thousands of features. These tables are also accompanied by metadata that I do not want to pass in the model. When I perform `set_format('pt', columns=large_column_list)` , it can take several minutes before it finishes. The culprit is when the following check is performed: `any(col not in self._data.column_names for col in columns)`. Replacing this by  `set(columns) - (self._data.column_names)` is more efficient.
GitHub issue: Fix missing info when loading some datasets from Parquet export
State: closed

Fix getting the info for script-based datasets with Parquet export with a single config not named "default".

E.g.

```python
from datasets import load_dataset_builder

b = load_dataset_builder("bookcorpus")
print(b.info.features)
# should print {'text': Value(dtype='string', id=None)}
```

I fixed this by setting the default config name when there is only one config.
GitHub issue: Fix reload cache with data dir
State: closed

The cache used to only check for the latest cache directory with a given config_name, but it was wrong (e.g. `default-data_dir=data%2Ffortran-data_dir=data%2Ffortran` instead of `default-data_dir=data%2Ffortran`)

I fixed this by not passing the `config_kwargs` to the parent Builder `__init__`, and passing the config_id forged from the `config_kwargs` directly

close https://github.com/huggingface/datasets/issues/6609
GitHub issue: Support push_to_hub without org/user to default to logged-in user
State: closed

This behavior is aligned with:
- the behavior of `datasets` before merging #6519
  - the behavior described in the corresponding docstring
- the behavior of `huggingface_hub.create_repo`

Revert "Support push_to_hub canonical datasets (#6519)"
- This reverts commit a887ee78835573f5d80f9e414e8443b4caff3541.

Fix #6597.
GitHub issue: streaming datasets doesn't work properly with multi-node
State: open

### Feature request

Let’s say I have a dataset with 5 samples with values [1, 2, 3, 4, 5], with 2 GPUs (for DDP) and batch size of 2. This dataset is an `IterableDataset` since I am streaming it.

Now I split the dataset using `split_dataset_by_node` to ensure it doesn’t get repeated. And since it’s already splitted, I don’t have to use `DistributedSampler` (also they don't work with iterable datasets anyway)?

But in this case I noticed that the:

First iteraton:
first GPU will get → [1, 2]
first GPU will get → [3, 4]

Second iteraton:
first GPU will get → [5]
first GPU will get → Nothing

which actually creates an issue since in case of `DistributedSampler`, the samples are repeated internally to ensure non of the GPUs at any iteration is missing any data for gradient sync.

So my questions are:

1. Here since splitting is happening before hand, how to make sure each GPU get’s a batch at each iteration to avoid gradient sync issues?
2. Do we need to use `DistributedSampler`? If yes, how?
3. in the docstrings of `split_dataset_by_node`, this is mentioned: *"If the dataset has a number of shards that is a factor of `world_size` (i.e. if `dataset.n_shards % world_size == 0`), then the shards are evenly assigned across the nodes, which is the most optimized. Otherwise, each node keeps 1 example out of `world_size`, skipping the other examples."* Can you explain the last part here?
4. If `dataset.n_shards % world_size != 0`, is it possible to shard the streaming dataset on the fly to avoid the case where data is missing?

### Motivation

Somehow streaming datasets should work with DDP since for big LLMs a lot of data is required and DDP/multi-node is mostly used to train such models and streaming can actually help solve the data part of it.

### Your contribution

Yes, I can help in submitting the PR once we get mutual understanding on how it should behave.
GitHub issue: multi-GPU map does not work
State: closed

### Describe the bug

Here is the code for single-GPU processing: https://pastebin.com/bfmEeK2y
Here is the code for multi-GPU processing: https://pastebin.com/gQ7i5AQy

Here is the video showing that the multi-GPU mapping does not work as expected (there are so many things wrong here, it's better to watch the 3-minute video than explain here):

https://youtu.be/RNbdPkSppc4

### Steps to reproduce the bug

-

### Expected behavior

-

### Environment info

x2 RTX A4000
GitHub issue: wiki_dpr.py error (ID mismatch between lines {id} and vector {vec_id}
State: closed

### Describe the bug

I'm trying to run a rag example, and the dataset is wiki_dpr.
wiki_dpr download and extracting have been completed successfully.
However, at the generating train split stage, an error from wiki_dpr.py keeps popping up.

Especially in "_generate_examples" :
1. The following error occurs in the line **id, text, title = line.strip().split("\t")**
ValueError: not enough values ​​to unpack (expected 3, got 2)
-> This part handles exceptions so that even if an error occurs, it passes.

2. **ID mismatch between lines {id} and vector {vec_id}**
This error seems to occur at the line " assert int(id) == int(vec_id),".
After I handled the exception in the split error, generating train split progressed to 80%, but an id mismatch error occurred at about the 16200000th vector id.

Debugging is even more difficult because it takes a long time to download and split wiki_dpr. I need help. thank you in advance!!

### Steps to reproduce the bug

Occurs in the generating train split step when running the rag example in the transformers repository.
Specifically, it is an error in wiki_dpr.py.

### Expected behavior

.

### Environment info

python 3.8
GitHub issue: `datasets/downloads` cleanup tool
State: open

### Feature request

Splitting off https://github.com/huggingface/huggingface_hub/issues/1997 - currently `huggingface-cli delete-cache` doesn't take care of cleaning `datasets` temp files

e.g. I discovered having millions of files under `datasets/downloads` cache, I had to do:

```
sudo find /data/huggingface/datasets/downloads -type f -mtime +3 -exec rm {} \+
sudo find /data/huggingface/datasets/downloads -type d -empty -delete
```
 
could the cleanup be integrated into `huggingface-cli` or a different tool provided to keep the folders tidy and not consume inodes and space 

e.g. there were tens of thousands of `.lock` files - I don't know why they never get removed - lock files should be temporary for the duration of the operation requiring the lock and not remain after the operation finished, IMHO.

Also I think one should be able to nuke `datasets/downloads` w/o hurting the cache, but I think there are some datasets that rely on files extracted under this dir - or at least they did in the past - which is very difficult to manage since one has no idea what is safe to delete and what not.

Thank you

@Wauplin (requested to be tagged)
GitHub issue: cnn_dailymail repeats itself
State: closed

### Describe the bug

When I try to load `cnn_dailymail` dataset, it takes longer than usual and when I checked the dataset it's 3x bigger than it's supposed to be.

Check https://huggingface.co/datasets/cnn_dailymail: it says 287k rows for train. But when I check length of train split it says 861339.

Also I checked data:

```
>>> ds['train']['highlights'][0]                                           
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday . Young actor says he has no plans to fritter his cash away . Radcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][0]                                           
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday . Young actor says he has no plans to fritter his cash away . Radcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][287113]
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][574226]
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ."
```

The datasets seems to be updated 6 days ago to convert it to Parquet. Probably, there is some issue with backward compatability.



### Steps to reproduce the bug

1.
```
from datasets import load_dataset
ds = load_dataset('cnn_dailymail', '3.0.0')
len(ds['train'])
```

### Expected behavior

It should not repeat itself.

### Environment info

datasets==2.13.2
Python==3.7.13
GitHub issue: cast_column to Sequence(subfeatures_dict) has err
State: closed

### Describe the bug

I am working with the following demo code:
```
from datasets import load_dataset
from datasets.features import Sequence, Value, ClassLabel, Features

ais_dataset = load_dataset("/data/ryan.gao/ais_dataset_cache/raw/1978/")
ais_dataset = ais_dataset["train"]

def add_class(example):
    example["my_labeled_bbox"] = {"bbox": [100,100,200,200], "label": "cat"}
    return example

ais_dataset = ais_dataset.map(add_class, batched=False, num_proc=32)
ais_dataset = ais_dataset.cast_column("my_labeled_bbox", Sequence(
                                                        {
                                                            "bbox": Sequence(Value(dtype="int64")),
                                                            "label": ClassLabel(names=["cat", "dog"])
                                                        }))

print(ais_dataset[0])
```
However, executing this code results in an error:

```
  File "/home/protoss.gao/.local/lib/python3.9/site-packages/datasets/table.py", line 2111, in cast_array_to_feature
    raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{feature}")
TypeError: Couldn't cast array of type
int64
to
Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None)
```
Upon examining the source code in datasets/table.py at line 2035:

```
        if isinstance(feature, Sequence) and isinstance(feature.feature, dict):
            feature = {
                name: Sequence(subfeature, length=feature.length) for name, subfeature in feature.feature.items()
            }
```

I noticed that if subfeature is of type Sequence, the code results in Sequence(Sequence(...), ...) and Sequence(ClassLabel(...), ...), which appears to be the source of the error.


### Steps to reproduce the bug

run my demo code

### Expected behavior

no exception

### Environment info

python 3.9 
datasets: 2.16.1
GitHub issue: Wrong path for cache directory in offline mode
State: closed

### Describe the bug

Dear huggingfacers,

I'm trying to use a subset of the-stack dataset. When I run the command the first time

```
dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )
```

It downloads the files and caches them normally.

Nevertheless, since my compute nodes are not online (`HF_DATASETS_OFFLINE=1`) . Whenever I try to run the command again, the library is passing the wrong cache path:


`Cache directory for the-stack doesn't exist at /Users/user/.cache/huggingface/datasets/bigcode___the-stack/default-data_dir=data%2Ffortran-data_dir=data%2Ffortran`

when the right path is:

`'/Users/user/.cache/huggingface/datasets/bigcode___the-stack/default-data_dir=data\%2Ffortran`

Not sure why those redundancies are included in the path.  If I try adding the correct path through the the cache_dir argument it throws an error:

ConnectionError: Couldn't reach the Hugging Face Hub for dataset 'bigcode/the-stack': Offline mode is enabled.


Your help with this issue is greatly appreciated. Thanks a lot for the great work.


### Steps to reproduce the bug

1: 

`dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )`


2: 

`HF_DATASETS_OFFLINE=1`

3:

`dataset = load_dataset(
    path='bigcode/the-stack',
    data_dir='data/fortran',
    split='train' )`


### Expected behavior

being able to use the cached data

### Environment info

several different systems
GitHub issue: ELI5 no longer available, but referenced in example code
State: closed

Here, an example code is given:
https://huggingface.co/docs/transformers/tasks/language_modeling

This code + article references the ELI5 dataset.
ELI5 is no longer available, as the ELI5 dataset page states: https://huggingface.co/datasets/eli5
"Defunct: Dataset "eli5" is defunct and no longer accessible due to unavailability of the source data.
Reddit recently [changed the terms of access](https://www.reddit.com/r/reddit/comments/12qwagm/an_update_regarding_reddits_api/) to its API, making the source data for this dataset unavailable.
"

Please change the example code to use a different dataset.
GitHub issue: Transform fingerprint collisions due to setting fixed random seed
State: closed

### Describe the bug

The transform fingerprinting logic relies on the `random` library for random bits when the function is not hashable (e.g. bound methods as used in `trl`: https://github.com/huggingface/trl/blob/main/trl/trainer/dpo_trainer.py#L356). This causes collisions when the training code sets a fixed random seed, which is common practice: https://github.com/huggingface/alignment-handbook/blob/main/recipes/zephyr-7b-beta/sft/config_full.yaml#L45.

This results in fingerprint collisions which leads to silently loading incorrect cache files corresponding to completely different datasets.

### Steps to reproduce the bug

n/a

### Expected behavior

Use `uuid` v4 instead of `random.getrandbits()`

### Environment info

`datasets` main branch
GitHub issue: datasets map `cache_file_name` does not work
State: open

### Describe the bug

In the documentation `datasets.Dataset.map` arg `cache_file_name` is said to be a string, but it doesn't work.

### Steps to reproduce the bug

1. pick a dataset
2. write a map function
3. do `ds.map(..., cache_file_name='some_filename')`
4. it crashes

### Expected behavior

It will tell you the filename you specified does not exist or it will generate a new file and tell you the filename does not exist.

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2
GitHub issue: Index error when data is large
State: open

### Describe the bug

At `save_to_disk` step, the `max_shard_size` by default is `500MB`. However, one row of the dataset might be larger than `500MB` then the saving will throw an index error. Without looking at the source code, the bug is due to wrong calculation of number of shards which i think is 
`total_size / min(max_shard_size, row_size)` which should be `total_size / max(max_shard_size, row_size)`

The fix is setting a larger `max_shard_size`

### Steps to reproduce the bug

1. create a dataset with large dense tensors per row
2. set a small `max_shard_size` say 1MB
3. `save_to_disk`

### Expected behavior

```
raise IndexError(f"Index {index} out of range for dataset of size {size}.")

IndexError: Index 10 out of range for dataset of size 10.
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2
GitHub issue: Loading CSV exported dataset has unexpected format
State: open

### Describe the bug

I wanted to be able to save a HF dataset for translations and load it again in another script, but I'm a bit confused with the documentation and the result I've got so I'm opening this issue to ask if this behavior is as expected.

### Steps to reproduce the bug

The documentation I've mainly consulted is https://huggingface.co/docs/datasets/v2.16.1/en/package_reference/loading_methods#datasets.load_dataset and https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset (where I've found `.to_csv()`)

```python
# Load a dataset of translations
test_dataset = load_dataset("opus100", name="en-fr", split="test")

# Save with .to_csv()
test_csv_path = "try_testset_save.csv"
test_dataset.to_csv(test_csv_path)

# Load dataset from the CSV
loaded_dataset = load_dataset("csv", data_files=test_csv_path)
print(test_dataset_fromfile[0]["translation"])
print(test_dataset_fromfile[0]["translation"]["en"])
```
```
Creating CSV from Arrow format: 100%
2/2 [00:00<00:00, 47.99ba/s]
Downloading data files: 100%
1/1 [00:00<00:00, 65.33it/s]
Extracting data files: 100%
1/1 [00:00<00:00, 42.10it/s]
Generating train split:
2000/0 [00:00<00:00, 47486.09 examples/s]

{'en': "She wasn't going to vaccinate her kid against polio,  no way.", 'fr': 'Elle ne vaccinerait pas son enfant contre la polio. Pas question.'}

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[29], line 11
      9 loaded_dataset = load_dataset("csv", data_files=test_csv_path)
     10 print(test_dataset_fromfile[0]["translation"])
---> 11 print(test_dataset_fromfile[0]["translation"]["en"])

TypeError: string indices must be integers, not 'str'
```


### Expected behavior

Each translation was saved as a stringified dict like `"{'en': ""She wasn't going to vaccinate her kid against polio,  no way."", 'fr': 'Elle ne vaccinerait pas son enfant contre la polio. Pas question.'}"` where I would have expected 2 columns (1st with English segments, and 2nd with French segments), and I was expecting `load_dataset` to infer the type of feature automatically as I haven't seen anything about it in the documentation.

Do you have an example of how to effectively save and load datasets of translations ?

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-3.10.0-1160.36.2.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.11.5
- `huggingface_hub` version: 0.16.4
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0
GitHub issue: Easy way to segment into 30s snippets given an m4a file and a vtt file 
State: closed

### Feature request

Uploading datasets is straightforward thanks to the ability to push Audio to hub. However, it would be nice if the data (text and audio) could be segmented when being pushed (if not possible already).

### Motivation

It's easy to create a vtt file from an audio file. If there could be auto-segmenting, this would make the creation of datasets much faster.

### Your contribution

I have made a custom script to do this but it's not all that clean - uses librosa and pydub.
GitHub issue: Dataset.push_to_hub of a canonical dataset creates an additional dataset under the user namespace
State: closed

While using `Dataset.push_to_hub` of a canonical dataset, an additional dataset was created under my user namespace.

## Steps to reproduce the bug
The command:
```python
commit_info = ds.push_to_hub(
    "caner",
    config_name="default",
    commit_message="Convert dataset to Parquet",
    commit_description="Convert dataset to Parquet.",
    create_pr=True,
    token=token,
)
```
creates the additional dataset `albertvillanova/caner`.
GitHub issue: IterableDataset sharding logic needs improvement
State: open

### Describe the bug

The sharding of IterableDatasets with respect to distributed and dataloader worker processes appears problematic with significant performance traps and inconsistencies wrt to distributed train processes vs worker processes.

Splitting across num_workers (per train process loader processes) and world_size (distributed training processes) appears inconsistent.
* worker split: https://github.com/huggingface/datasets/blob/9d6d16117a30ba345b0236407975f701c5b288d4/src/datasets/iterable_dataset.py#L1266-L1283
* distributed split: https://github.com/huggingface/datasets/blob/9d6d16117a30ba345b0236407975f701c5b288d4/src/datasets/iterable_dataset.py#L1335-L1356

In the case of the distributed split, there is a modulus check that flips between two very different behaviours, why is this different than splitting across the data loader workers? For IterableDatasets the DataLoaders worker processes are independent, so whether it's workers within one train process or across a distributed world the shards should be distributed the same, across `world_size * num_worker` independent workers in either case... 

Further, the fallback case when the `n_shards % world_size == 0` check fails is a rather extreme change. I argue it is not desirable to do that implicitly, it should be an explicit case for specific scenarios (ie reliable validation). A train scenario would likely be much better handled with improved wrapping / stopping behaviour to eg also fix #6437. Changing from stepping shards to stepping samples means that every single process reads ALL of the shards. This was never an intended default for sharded training, shards gain their performance advantage in large scale distributed training by explicitly avoiding the need to have every process overlapping in the data they read, by default, only the data allocated to each process via their assigned shards should be read in each pass of the dataset.

Using a large scale CLIP example, some of the larger datasets have 10-20k shards across 100+TB of data. Training with 1000 GPUs we are switching between reading 100 terabytes per epoch to 100 petabytes if say change 20k % 1000 and drop one gpu-node to 20k % 992.

The 'step over samples' case might be worth the overhead in specific validation scenarios where gaurantees of at least/most once samples seen are more important and do not make up a significant portion of train time or are done in smaller world sizes outside of train.

### Steps to reproduce the bug

N/A

### Expected behavior

We have an iterable dataset with N shards, to split across workers
* shuffle shards (same seed across all train processes)
* step shard iterator across distributed processes
* step shard iterator across dataloader worker processes
* shuffle samples in every worker via shuffle buffer (different seed in each worker, but ideally controllable (based on base seed + worker id + epoch).
* end up with (possibly uneven) number of shards per worker but each shard only ever accessed by 1 worker per pass (epoch)



### Environment info

N/A
GitHub issue: Logs are delayed when doing .map when `docker logs`
State: closed

### Describe the bug

When I run my SD training in a Docker image and then listen to logs like `docker logs train -f`, the progress bar is delayed.
It's updating every few percent.
When you have a large dataset that has to be mapped (like 1+ million samples), it's crucial to see the updates in real-time, not every couple hours to make sure nothing got frozen or broken

### Steps to reproduce the bug

1. Run any huge dataset processing as a Docker image
2. `docker logs image_name` to it

### Expected behavior

...

### Environment info

...
GitHub issue: The datasets models housed in Dropbox can't support a lot of users downloading them
State: closed

### Describe the bug

I'm using the datasets 


```
from datasets import load_dataset, Audio
dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
```


And it seems that sometimes when I imagine a lot of users are accessing the same resources, the Dropbox host fails:

`raise ConnectionError(f"Couldn't reach {url} (error {response.status_code})") ConnectionError: Couldn't reach https://www.dropbox.com/s/e2us0hcs3ilr20e/MInDS-14.zip?dl=1 (error 429)`

My question is if we can somehow host these files elsewhere or can you change the limit of simultaneous users accessing those resources or any other solution?

Also, has anyone had this issue before?

Thanks

### Steps to reproduce the bug

1: Create a python script like so:
```

from datasets import load_dataset, Audio
dataset = load_dataset("PolyAI/minds14", name="en-US", split="train")
```

2: Execute this by a certain number of users at the same time

### Expected behavior

I woudl expect that this shouldnt happen unless its a huge amount of users, which it is not the case

### Environment info

This was done in an Ubuntu 22 environment.
GitHub issue: Feature request: Multi-GPU dataset mapping for SDXL training
State: open

### Feature request

We need to speed up SDXL dataset pre-process. Please make it possible to use multiple GPUs for the [official SDXL trainer](https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_sdxl.py) :)

### Motivation

Pre-computing 3 million of images takes around 2 days.
Would be nice to be able to be able to do multi-GPU (or even better – multi-GPU + multi-node) vae and embedding precompute...

### Your contribution

I'm not sure i can wrap my head around the multi-GPU mapping...
Plus it's too expensive for me to take x2 A100 and spend a day just figuring out the staff since I don't have a job right now.
GitHub issue: After `2.16.0` version, there are `PermissionError` when users use shared cache_dir
State: closed

### Describe the bug

- We use shared `cache_dir` using `HF_HOME="{shared_directory}"` 
- After dataset version 2.16.0, datasets uses `filelock` package for file locking #6445 
- But, `filelock` package make `.lock` file with `644` permission
- Dataset is not available to other users except the user who created the lock file via `load_dataset`.

### Steps to reproduce the bug

1. `pip install datasets==2.16.0`
2. `export HF_HOME="{shared_directory}"`
3. download dataset with `load_dataset`
4. logout and login another user
5. `pip install datasets==2.16.0`
6. `export HF_HOME="{shared_directory}"`
7. download dataset with `load_dataset`
8. `PermissionError` occurs

### Expected behavior

- Users can share `cache_dir` using environment variable `HF_HOME`

### Environment info

- python == 3.9.10
- datasets == 2.16.0
- ubuntu 22.04
- shared_directory has ACL
![image (1)](https://github.com/huggingface/datasets/assets/106717516/5ca759db-ad0c-4883-9a97-9c8fccd00d8a)
- users are same group (developers)
GitHub issue: fix os.listdir return name is empty string
State: closed

### Describe the bug

xlistdir return name is empty string 
Overloaded os.listdir

### Steps to reproduce the bug

```python
from datasets.download.streaming_download_manager import xjoin
from datasets.download.streaming_download_manager import xlistdir
config = DownloadConfig(storage_options=options)
manger = StreamingDownloadManager("ILSVRC2012",download_config=config)
input_path = "lakefs://datalab/main/imagenet/ILSVRC2012.zip"

download_files = manger.download_and_extract(input_path)

current_dir = xjoin(download_files,"ILSVRC2012/Images/ILSVRC2012_img_train")
folder_list = xlistdir(current_dir)
```

in xlistdir function

Obj ["name"] ends with "/"

last return ""

### Expected behavior

Obj ["name"] ends with "/" 
return folder name

### Environment info

no
GitHub issue: Allow concatenation of datasets with mixed structs
State: closed

Fixes #6466 

The idea is to do a recursive check for structs. PyArrow handles it well enough.


For a demo you can do:

```python
from datasets import Dataset, concatenate_datasets

ds = Dataset.from_dict({'speaker': [{'name': 'Ben', 'email': None}]})
ds2 = Dataset.from_dict({'speaker': [{'name': 'Fred', 'email': 'abc@aol.com'}]})
print(concatenate_datasets([ds, ds2]).features)
print(concatenate_datasets([ds, ds2]).to_dict())
```

Now both the features and the rows are fixed.

I note that Sequence suffers from the same problem, so I can fix that in a future PR once this one is merged.
GitHub issue: losing DatasetInfo in Dataset.map when num_proc > 1
State: open

### Describe the bug

Hello and thanks for developing this package!

When I process a Dataset with the map function using multiple processors some set attributes of the DatasetInfo get lost and are None in the resulting Dataset.    

### Steps to reproduce the bug

```python
from datasets import Dataset, DatasetInfo


def run_map(num_proc):
    dataset = Dataset.from_dict(
        {"col1": [0, 1], "col2": [3, 4]},
        info=DatasetInfo(
            dataset_name="my_dataset",
        ),
    )

    ds = dataset.map(lambda x: x, num_proc=num_proc)
    print(ds.info.dataset_name)


run_map(1)
run_map(2)
```

This puts out:

```bash
Map: 100%|██████████| 2/2 [00:00<00:00, 724.66 examples/s]
my_dataset
Map (num_proc=2): 100%|██████████| 2/2 [00:00<00:00, 18.25 examples/s]
None
```

### Expected behavior

I expect the DatasetInfo to be kept as it was and there should be no difference in the output of running map with num_proc=1 and num_proc=2. 

Expected output:
```bash
Map: 100%|██████████| 2/2 [00:00<00:00, 724.66 examples/s]
my_dataset
Map (num_proc=2): 100%|██████████| 2/2 [00:00<00:00, 18.25 examples/s]
my_dataset
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.17
- Python version: 3.8.18
- `huggingface_hub` version: 0.20.2
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.9.2