xyx138/github-issues huggingface dataset:
Mosaic Streaming (MDS) Support

### Feature request

I'm a huge fan of the current HF Datasets `webdataset` integration (especially the built-in streaming support). However, I'd love to upload some robotics and multimodal datasets I've processed for use with [Mosaic Streaming](https://docs.mosaicml.com/projects/streaming/en/stable/), specifically their [MDS Format](https://docs.mosaicml.com/projects/streaming/en/stable/fundamentals/dataset_format.html#mds). 

Because the shard files have similar semantics to WebDataset, I'm hoping that adding such support won't be too much trouble?

### Motivation

One of the downsides with WebDataset is a lack of out-of-the-box determinism (especially for large-scale training and reproducibility), easy job resumption, and the ability to quickly debug / visualize individual examples. 

Mosaic Streaming provides a [great interface for this out of the box](https://docs.mosaicml.com/projects/streaming/en/stable/#key-features), so I'd love to see it supported in HF Datasets.

### Your contribution

Happy to help test things / provide example data. Can potentially submit a PR if maintainers could point me to the necessary WebDataset logic / steps for adding a new streaming format!
Tokenization slows towards end of dataset

### Describe the bug

Mapped tokenization slows down substantially towards end of dataset.

train set started off very slow, caught up to 20k then tapered off til the end.

what's particularly strange is that the tokenization crashed a few times before due to errors with invalid tokens somewhere or corrupted downloads, and the speed ups/downs consistently happened the same times

```bash
Running tokenizer on dataset (num_proc=48):   0%|          | 847000/881416735 [12:18<252:45:45, 967.72 examples/s]
Running tokenizer on dataset (num_proc=48):   0%|          | 848000/881416735 [12:19<224:16:10, 1090.66 examples/s]

Running tokenizer on dataset (num_proc=48):  10%|▉         | 84964000/881416735 [3:48:00<11:21:34, 19476.01 examples/s]
Running tokenizer on dataset (num_proc=48):  10%|▉         | 84967000/881416735 [3:48:00<12:04:01, 18333.79 examples/s]

Running tokenizer on dataset (num_proc=48):  61%|██████    | 538631977/881416735 [13:46:40<27:50:04, 3420.84 examples/s]
Running tokenizer on dataset (num_proc=48):  61%|██████    | 538632977/881416735 [13:46:40<23:48:20, 3999.77 examples/s]

Running tokenizer on dataset (num_proc=48): 100%|█████████▉| 881365886/881416735 [38:30:19<04:34, 185.10 examples/s]
Running tokenizer on dataset (num_proc=48): 100%|█████████▉| 881366886/881416735 [38:30:25<04:36, 180.57 examples/s]
```

and validation set as well

```bash
Running tokenizer on dataset (num_proc=48):  90%|████████▉ | 41544000/46390354 [28:44<02:37, 30798.76 examples/s]
Running tokenizer on dataset (num_proc=48):  90%|████████▉ | 41550000/46390354 [28:44<02:08, 37698.08 examples/s]

Running tokenizer on dataset (num_proc=48):  96%|█████████▋| 44747422/46390354 [2:15:48<12:22:44, 36.87 examples/s]
Running tokenizer on dataset (num_proc=48):  96%|█████████▋| 44747422/46390354 [2:16:00<12:22:44, 36.87 examples/s]

```



### Steps to reproduce the bug

    using the following kwargs
    
```python
with accelerator.main_process_first():
        lm_datasets = tokenized_datasets.map(
            group_texts,
            batched=True,
            num_proc=48
            load_from_cache_file=True,
            desc=f"Grouping texts in chunks of {block_size}",
        )
```

running through slurm script 
```bash
#SBATCH --partition=gpu-nvidia-a100
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gpus-per-task=8
#SBATCH --cpus-per-task=96
```

using this dataset https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T





### Expected behavior

Constant speed throughout

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.15.0-1049-aws-x86_64-with-glibc2.10
- Python version: 3.8.18
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0
EmptyDatasetError when loading dataset downloaded with HuggingFace cli

### Describe the bug

I am using a cluster that does not have access to the internet when given a job. I tried downloading the dataset using the huggingface-cli command and then loading it with load_dataset but I get an error:

```raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None```

The dataset I'm using is "lmsys/chatbot_arena_conversations". The folder structure is

- README.md
- data
  - train-00000-of-00001-cced8514c7ed782a.parquet



### Steps to reproduce the bug

1. Download dataset using HuggingFace CLI: ```huggingface-cli download lmsys/chatbot_arena_conversations --local-dir ./lmsys/chatbot_arena_conversations```
2. In Python
```
from datasets import load_dataset
load_dataset("lmsys/chatbot_arena_conversations")
```

### Expected behavior

Should return a Dataset Dict in the form of
```
DatasetDict({
    train: Dataset({
        features: [...],
        num_rows: 33,000
    })
})
```
### Environment info

Python 3.11.5
Datasets 2.18.0
Transformers 4.38.2
Pytorch 2.2.0
Pyarrow 15.0.1
Rocky Linux release 8.9 (Green Obsidian)
Unexpected behavior when using load_dataset with streaming=True in a for loop

### Describe the bug

### My Code
```
from datasets import load_dataset
res=[]
for i in [0,1]:
    di=load_dataset(
            "json", 
            data_files='path_to.json', 
            split='train',
            streaming=True, 
        ).map(lambda x: {"source": i})

    res.append(di)

for e in res[0]:
    print(e)
```

### Unexpected Behavior
Data in `res[0]` has `source=1`. However the expected value is 0.

### FYI
I further switch `streaming` to `False`. And the output value is as expected (0). So there may exist bugs in setting `streaming=True` in a for loop.

### Environment
Python 3.8.0
datasets==2.18.0
transformers==4.28.1

### Steps to reproduce the bug

1. Create a Json file with any content.
2. Run the provided code.
3. Switch `streaming` to `False` and run again to see the expected behavior.

### Expected behavior

The expected behavior is the data are mapped with its corresponding value in the for loop.

### Environment info

Python 3.8.0

datasets==2.18.0
transformers==4.28.1

Ubuntu 20.04
Using a registry instead of calling globals for fetching feature types

Hello,

When working with bio-data, each feature often has metadata associated with it (e.g. species, lineage, snp position, etc). To store this, I like to use the feature classes with the added `metadata` attribute. However, when saving or loading with custom features, you get an error since that class doesn't exist in the global namespace in `datasets.features.features`. Take for example,



```python
from dataclasses import dataclass, field
from datasets import Dataset
from datasets.features.features import Value, Features

@dataclass
class FeatureA(Value):
    metadata: dict = field(default=dict)
    _type: str = field(default="FeatureA", init=False, repr=False)

@dataclass
class FeatureB(Value):
    metadata: dict = field(default_factory=dict)
    _type: str = field(default="FeatureB", init=False, repr=False)

test_data = {
    "a": [1, 2, 3],
    "b": [4, 5, 6],
}
test_data = Dataset.from_dict(
    test_data, 
    features=Features({
        "a": FeatureA("int32", metadata={"species": "lactobacillus acetotolerans"}),
        "b": FeatureB("int32", metadata={"species": "lactobacillus iners"}),
    })
)

# returns an error since FeatureA and FeatureB are not in the global namespace
test_data.save_to_disk('./test_data')
```


    Saving the dataset (0/1 shards):   0%|          | 0/3 [00:00<?, ? examples/s]



    ---------------------------------------------------------------------------

    KeyError                                  Traceback (most recent call last)

    Cell In[2], line 28
         19 test_data = Dataset.from_dict(
         20     test_data, 
         21     features=Features({
       (...)
         24     })
         25 )
         27 # returns an error since FeatureA and FeatureB are not in the global namespace
    ---> 28 test_data.save_to_disk('./test_data')
    ...
    File ~\Documents\datasets\src\datasets\features\features.py:1361, in generate_from_dict(obj)
       1359     return {key: generate_from_dict(value) for key, value in obj.items()}
       1360 obj = dict(obj)
    -> 1361 class_type = globals()[obj.pop("_type")]
       1363 if class_type == Sequence:
       1364     return Sequence(feature=generate_from_dict(obj["feature"]), length=obj.get("length", -1))
    

    KeyError: 'FeatureA'


We can avoid this by having a registry (like formatters) and doing


```python
from datasets.features.features import register_feature
register_feature(FeatureA, "FeatureA")
register_feature(FeatureB, "FeatureB")
test_data.save_to_disk('./test_data')
```


    Saving the dataset (1/1 shards): 100%|------| 3/3 [00:00<00:00, 211.13 examples/s]


and loading from disk returns with all metadata information


```python
from datasets import load_from_disk
test_data = load_from_disk('./test_data')
test_data.features
```




    {'a': FeatureA(dtype='int32', id=None, metadata={'species': 'lactobacillus acetotolerans'}),
     'b': FeatureB(dtype='int32', id=None, metadata={'species': 'lactobacillus iners'})}
Non-deterministic `Dataset.builder_name` value

### Describe the bug

I'm not sure if this is a bug, but `print(ds.builder_name)` in the following code sometimes prints out `rotten_tomatoes` instead of `parquet`:

```python
import datasets

for _ in range(100):
    ds = datasets.load_dataset("rotten_tomatoes", split="train")
    print(ds.builder_name)  # prints out "rotten_tomatoes" sometimes instead of "parquet"
```

Output:

```
...
parquet
parquet
parquet
rotten_tomatoes
parquet
parquet
parquet
...
```

Here's a reproduction using GitHub Actions:
https://github.com/mlflow/mlflow/actions/runs/8153247984/job/22284263613?pr=11329#step:12:241

One of our tests is flaky because `builder_name` is not deterministic.

### Steps to reproduce the bug

1. Run the code above.

### Expected behavior

Always prints out `parquet`?

### Environment info

```
Copy-and-paste the text below in your GitHub issue.
- `datasets` version: 2.18.0
- Platform: Linux-6.5.0-1015-azure-x86_64-with-glibc2.34
- Python version: 3.8.18
- `huggingface_hub` version: 0.21.3
- PyArrow version: 15.0.0
- Pandas version: 2.0.3
- `fsspec` version: 2024.2.0
```
Fix sliced ConcatenationTable pickling with mixed schemas vertically

A sliced + pickled ConcatenationTable could end up with a different schema than the original schema, if the slice only contains blocks with only a subset of the columns.

This can lead to issues when saving datasets from a concatenation of datasets with mixed schemas

Reported in https://discuss.huggingface.co/t/datasetdict-save-to-disk-with-num-proc-1-seems-to-hang-with-error/75595
Bump huggingface-hub lower version to 0.21.2

This should fix the version compatibility issue when using `huggingface_hub` < 0.21.2 and latest fsspec (>=2023.12.0).

See my comment: https://github.com/huggingface/datasets/pull/6687#issuecomment-1976493336
>> EDIT: the fix has been released in `huggingface_hub` 0.21.2 - I removed my commits that were using `huggingface_hub@main`
>
>Please note that people using `huggingface_hub` < 0.21.2 and latest `fsspec` will have issues when using `datasets`:
>- https://github.com/huggingface/lighteval/actions/runs/8139147047/job/22241658122?pr=86
>- https://github.com/huggingface/lighteval/pull/84

CC: @clefourrier
3x Faster Text Preprocessing

I was preparing some datasets for AI training and noticed that `datasets` by HuggingFace uses the conventional `open` mechanism to read the file and split it into chunks. I thought it can be significantly accelerated, and [started with a benchmark](https://gist.github.com/ashvardanian/55c2052e9f78b05b8d614aa90cb12347):

```sh
$ pip install --upgrade --force-reinstall datasets
$ python benchmark_huggingface_datasets.py xlsum.csv 
Generating train split: 1004598 examples [00:47, 21116.16 examples/s]
Time taken to load the dataset: 48.66838526725769 seconds
Time taken to chunk the dataset into parts of size 10000: 0.11466407775878906 seconds
Total time taken: 48.78304934501648 seconds
```

For benchmarks I've used a [large CSV file with mixed UTF-8 content](https://github.com/ashvardanian/StringZilla/blob/main/CONTRIBUTING.md#benchmarking-datasets), most common in modern large-scale pre-training pipelines. I've later patched the `datasets` library to use `stringzilla`, which resulted in significantly lower memory consumption and in 2.9x  throughput improvement on the AWS `r7iz` instances. That's using slow SSDs mounted over the network. Performance on local SSDs on something like a DGX-H100 should be even higher:

```sh
$ pip install -e .
$ python benchmark_huggingface_datasets.py xlsum.csv 
Generating train split: 1004598 examples [00:15, 64529.90 examples/s]
Time taken to load the dataset: 16.45028805732727 seconds
Time taken to chunk the dataset into parts of size 10000: 0.1291060447692871 seconds
Total time taken: 16.579394102096558 seconds
```

I've already [pushed the patches to my fork](https://github.com/ashvardanian/datasets/tree/faster-text-parsers), and would love to contribute them to the upstream repository.

---

All the tests pass, but they leave a couple of important questions open. The default Python `open(..., newline=None)` uses universal newlines, where `\n`, `\r`, and `\r\n` are all converted to `\n` on the fly. I am not sure if its a good idea for a general purpose dataset preparation pipeline?

I can simulate the same behavior (which I don't yet do) for `"line"` splitter. Adjusting it for `"paragraph"`-splitter would be harder. Should we stick exactly to the old Pythonic behavior or stay closer to how C and other programming languages do that?
Persist IterableDataset epoch in workers

Use shared memory for the IterableDataset epoch.

This way calling `ds.set_epoch()` in the main process will update the epoch in the DataLoader workers as well.
This is useful especially because the epoch is used to compute the `effective_seed` used for shuffling.

I used torch's shared memory in case users want to send dataset copies without shared memory using pickle. I also find it easier to use than using `multiprocessing.shared_memory` than requires unlinking only in the main process, or `mp.Value` that is not picklable.

close https://github.com/huggingface/datasets/issues/6673

cc @rwightman
Fix data_files when passing data_dir

This code should not return empty data files

```python
from datasets import load_dataset_builder

revision = "3d406e70bc21c3ca92a9a229b4c6fc3ed88279fd"
b = load_dataset_builder("bigcode/the-stack-v2-dedup", data_dir="data/Dockerfile", revision=revision)
print(b.config.data_files)
```

Previously it would return no data files because it would apply the YAML `data_files: data/**/train-*` pattern to this directory

cc @anton-l
Improve default patterns resolution

Separate the default patterns that match directories from the ones matching files and ensure directories are checked first (reverts the change from https://github.com/huggingface/datasets/pull/6244, which merged these patterns). Also, ensure that the glob patterns do not overlap to avoid duplicates in the result. 

Additionally, replace `get_fs_token_paths` with `url_to_fs` to avoid [unnecessary glob calls](https://github.com/fsspec/filesystem_spec/blob/14dce8ca78f7aa509a20edb263bff83a7760c24d/fsspec/core.py#L655-L656).

fix https://github.com/huggingface/datasets/issues/6259
fix https://github.com/huggingface/datasets/issues/6272
Unable to load dataset that was saved with `save_to_disk`

### Describe the bug

I get the following error message: You are trying to load a dataset that was saved using `save_to_disk`. Please use `load_from_disk` instead.



### Steps to reproduce the bug

1. Save a dataset with `save_to_disk`
2. Try to load it with `load_datasets`

### Expected behavior

I am able to load the dataset again with `load_datasets` which most packages uses over `load_from_disk`. I want to have a workaround that allows me to create the same indexing that `push_to_hub` creates for you before using `save_to_disk` - how can that be achieved?

### Environment info

datasets 2.17.1, python 3.10
Push samples to dataset on hub without having the dataset locally

### Feature request

Say I have the following code:

```
from datasets import Dataset
import pandas as pd
new_data = {
    "column_1": ["value1", "value2"],
    "column_2": ["value3", "value4"],
}
df_new = pd.DataFrame(new_data)
dataset_new = Dataset.from_pandas(df_new)
# add these samples to a remote dataset
```
It would be great to have a way to push dataset_new to a remote dataset that respects the same schema. This way one would not have to do the following:
```
from datasets import load_dataset
dataset = load_dataset('username/dataset_name', use_auth_token='your_hf_token_here')
updated_dataset = dataset['train'].concatenate(dataset_new)
updated_dataset.push_to_hub('username/dataset_name', use_auth_token='your_hf_token_here')
```





### Motivation

No need to download the dataset.

### Your contribution

Maybe this feature already exists, didnt see it though. I do not have the expertise to do this.
Base parquet batch_size on parquet row group size

This allows to stream datasets like [Major-TOM/Core-S2L2A](https://huggingface.co/datasets/Major-TOM/Core-S2L2A) which have row groups with few rows (one row is ~10MB). Previously the cold start would take a lot of time and OOM because it would download many row groups before yielding the first example.

I tried on OpenOrca and imagenet-hard and it does't affect overall throughput.

Even if the overall throughput doesn't change for datasets like imagenet-hard with big rows, note that it does create shorter and more frequent pauses to download the next row group. Though I find it fine because previously the pauses were less frequent but very long (downloading multiple row groups at a time)
remove_columns is not in-place but the doc shows it is in-place

### Describe the bug

The doc of `datasets` v2.17.0/v2.17.1 shows that `remove_columns` is in-place. [link](https://huggingface.co/docs/datasets/v2.17.1/en/package_reference/main_classes#datasets.DatasetDict.remove_columns)

In the text classification example of transformers v4.38.1, the columns are not removed.
https://github.com/huggingface/transformers/blob/a0857740c0e6127485c11476650314df3accc2b6/examples/pytorch/text-classification/run_classification.py#L421

### Steps to reproduce the bug

https://github.com/huggingface/transformers/blob/a0857740c0e6127485c11476650314df3accc2b6/examples/pytorch/text-classification/run_classification.py#L421

### Expected behavior

Actually remove the columns.

### Environment info

1. datasets v2.17.0
2. transformers v4.38.1
`Dataset` unexpected changed dict data and may cause error

### Describe the bug

Will unexpected get keys with `None` value in the parsed json dict.

### Steps to reproduce the bug

```jsonl test.jsonl
{"id": 0, "indexs": {"-1": [0, 10]}}
{"id": 1, "indexs": {"-1": [0, 10]}}
```

```python
dataset = Dataset.from_json('.test.jsonl')
print(dataset[0])
```

Result:
```
{'id': 0, 'indexs': {'-1': [...], '-2': None, '-3': None, '-4': None, '-5': None, '-6': None, '-7': None, '-8': None, '-9': None, ...}}
```

Those keys with `None` value will unexpected appear in the dict.


### Expected behavior

Result should be
```
{'id': 0, 'indexs': {'-1': [0, 10]}}
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.5.0-14-generic-x86_64-with-glibc2.35
- Python version: 3.11.6
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0
__add__ for Dataset, IterableDataset

It's too cumbersome to write this command every time we perform a dataset merging operation. ```pythonfrom datasets import concatenate_datasets``` We have added a simple `__add__` magic method to each class using `concatenate_datasets.`

```python
from datasets import load_dataset

bookcorpus = load_dataset("bookcorpus", split="train")
wiki = load_dataset("wikimedia/wikipedia", "20231101.ab", split="train")
wiki = wiki.remove_columns([col for col in wiki.column_names if col != "text"])  # only keep the 'text' column

bookcorpus + wiki

#Dataset({
#   features: ['text'],
#   num_rows: 74004228
#})
#Dataset({
#    features: ['text'],
#    num_rows: 6152
#})
#Dataset({
#   features: ['text'],
#    num_rows: 74010380
#})
```
.load_dataset() method defaults to zstandard

### Describe the bug

Regardless of what method I use, datasets defaults to zstandard for unpacking my datasets.

This is poor behavior, because not only is zstandard not a dependency in the huggingface package (and therefore, your dataset loading will be interrupted while it asks you to install the package), but it happens on datasets that are uploaded in json format too, meaning the dataset loader will attempt to convert the data to a zstandard compatible format, and THEN try to unpackage it.

My 4tb drive runs out of room when using zstandard on slimpajama. It loads fine on 1.5tb when using json, however I lack the understanding of the "magic numbers" system used to select the unpackaging algorithm, so I can't push a change myself.

Commenting out this line, in "/datasets/utils/extract.py" fixes the issue, and causes SlimPajama to properly extract using rational amounts of storage, however it completely disables zstandard, which is probably undesirable behavior. Someone with an understanding of the "magic numbers" system should probably take a pass over this issue.

```
class Extractor:
    #  Put zip file to the last, b/c it is possible wrongly detected as zip (I guess it means: as tar or gzip)
    extractors: Dict[str, Type[BaseExtractor]] = {
        "tar": TarExtractor,
        "gzip": GzipExtractor,
        "zip": ZipExtractor,
        "xz": XzExtractor,
        #"zstd": ZstdExtractor,   # This line needs to go, in order for datasets to work w/o non-dependent packages
        "rar": RarExtractor,
        "bz2": Bzip2Extractor,
        "7z": SevenZipExtractor,  # <Added version="2.4.0"/>
        "lz4": Lz4Extractor,  # <Added version="2.4.0"/>
    }
```

### Steps to reproduce the bug

'''
from datasaets import load_dataset
load_dataset(path="/cerebras/SlimPajama-627B")
'''

This alone should trigger the error on any system that does not have zstandard pip installed.

### Expected behavior

This repository (which is encoded in json format, not zstandard) should check whether zstandard is installed before defaulting to it. Additionally, using zstandard should not use more than 3x the required space that other extraction mechanisms use.

### Environment info

- `datasets` version: 2.17.1
- Platform: Linux-6.5.0-18-generic-x86_64-with-glibc2.35
- Python version: 3.12.0
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
Tensor type (e.g. from `return_tensors`) ignored in map

### Describe the bug

I don't know if it is a bug or an expected behavior, but the tensor type seems to be ignored after applying map. For example, mapping over to tokenize text with a transformers' tokenizer always returns lists and it ignore the `return_tensors` argument.

If this is an expected behaviour (e.g., for caching/Arrow compatibility/etc.) it should be clearly documented. For example, current documentation (see [here](https://huggingface.co/docs/datasets/v2.17.1/en/nlp_process#map)) clearly state to "set  `return_tensors="np"` when you tokenize your text" to have Numpy arrays.

### Steps to reproduce the bug

```py
# %%%

import datasets
import numpy as np
import tensorflow as tf
import torch
from transformers import AutoTokenizer


# %%
ds = datasets.load_dataset("cnn_dailymail", "1.0.0", split="train[:1%]")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

#%%
for return_tensors in [None, "np", "pt", "tf", "jax"]:
  print(f"********** no map, return_tensors={return_tensors} **********")
  _ds = tokenizer(ds["article"], return_tensors=return_tensors, truncation=True, padding=True)
  print('Type <input_ids>:', type(_ds["input_ids"]))

# %%
for return_tensors in [None, "np", "pt", "tf", "jax"]:
  print(f"********** map, return_tensors={return_tensors} **********")
  _ds = ds.map(
    lambda examples: tokenizer(examples["article"], return_tensors=return_tensors, truncation=True, padding=True),
    batched=True,
    remove_columns=["article"],
  )

  print('Type <input_ids>:', type(_ds[0]["input_ids"]))
```


### Expected behavior

The output from the script above. I would expect the second half to be the same.

```
********** no map, return_tensors=None **********
Type <input_ids>: <class 'list'>
********** no map, return_tensors=np **********
Type <input_ids>: <class 'numpy.ndarray'>
********** no map, return_tensors=pt **********
Type <input_ids>: <class 'torch.Tensor'>
********** no map, return_tensors=tf **********
Type <input_ids>: <class 'tensorflow.python.framework.ops.EagerTensor'>
********** no map, return_tensors=jax **********
Type <input_ids>: <class 'jaxlib.xla_extension.ArrayImpl'>


********** map, return_tensors=None **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=np **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=pt **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=tf **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=jax **********
Type <input_ids>: <class 'list'>
```

### Environment info

- `datasets` version: 2.17.1
- Platform: Redacted (linux)
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0
fsspec: support fsspec>=2023.12.0 glob changes

- adds support for the `fs.glob` changes introduced in `fsspec==2023.12.0` and unpins the current upper bound

Should close #6644
Should close #6645

The `test_data_files` glob/pattern tests pass for me in:
- `fsspec==2023.10.0` (the pinned max version in datasets `main`)
- `fsspec==2023.12.0` (#6644)
- `fsspec==2024.2.0` (#6645)
Can't Read List of JSON Files Properly

### Describe the bug

Trying to read a bunch of JSON files into Dataset class but default approach doesn't work. I don't get why it works when I read it one by one but not when I pass as a list :man_shrugging: 

The code fails with 

```
ArrowInvalid: JSON parse error: Invalid value. in row 0  
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte 
DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

This doesn't work

```
from datasets import Dataset

# dir contains 100 json files.
Dataset.from_json("/PUT SOME PATH HERE/*")
```

This works:
```
from datasets import concatenate_datasets

ls_ds = []

for file in list_of_json_files:
    ls_ds.append(Dataset.from_json(file))

ds = concatenate_datasets(ls_ds)

```





### Expected behavior

I expect this to read json files properly as error is not clear

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-6.5.0-15-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
ValueError

### Describe the bug

ValueError                                Traceback (most recent call last)
[<ipython-input-11-9b99bc80ec23>](https://localhost:8080/#) in <cell line: 11>()
      9 import numpy as np
     10 import matplotlib.pyplot as plt
---> 11 from datasets import DatasetDict, Dataset
     12 from transformers import AutoTokenizer, AutoModelForSequenceClassification
     13 from transformers import Trainer, TrainingArguments

5 frames
[/usr/local/lib/python3.10/dist-packages/datasets/__init__.py](https://localhost:8080/#) in <module>
     16 __version__ = "2.17.0"
     17 
---> 18 from .arrow_dataset import Dataset
     19 from .arrow_reader import ReadInstruction
     20 from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_dataset.py](https://localhost:8080/#) in <module>
     65 
     66 from . import config
---> 67 from .arrow_reader import ArrowReader
     68 from .arrow_writer import ArrowWriter, OptimizedTypedSequence
     69 from .data_files import sanitize_patterns

[/usr/local/lib/python3.10/dist-packages/datasets/arrow_reader.py](https://localhost:8080/#) in <module>
     27 
     28 import pyarrow as pa
---> 29 import pyarrow.parquet as pq
     30 from tqdm.contrib.concurrent import thread_map
     31 

[/usr/local/lib/python3.10/dist-packages/pyarrow/parquet/__init__.py](https://localhost:8080/#) in <module>
     18 # flake8: noqa
     19 
---> 20 from .core import *

[/usr/local/lib/python3.10/dist-packages/pyarrow/parquet/core.py](https://localhost:8080/#) in <module>
     34 import pyarrow as pa
     35 import pyarrow.lib as lib
---> 36 import pyarrow._parquet as _parquet
     37 
     38 from pyarrow._parquet import (ParquetReader, Statistics,  # noqa

/usr/local/lib/python3.10/dist-packages/pyarrow/_parquet.pyx in init pyarrow._parquet()

ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Expected behavior

Resolve the binary incompatibility

### Environment info

Google Colab Note book
Allow SplitDict setitem to replace existing SplitInfo

Fix this code provided by @clefourrier 

```python
import datasets
import os

token = os.getenv("TOKEN")

results = datasets.load_dataset("gaia-benchmark/results_public", "2023", token=token, download_mode=datasets.DownloadMode.FORCE_REDOWNLOAD)
results["test"] = datasets.Dataset.from_list([row for row in results["test"] if row["model"] != "StateFlow"])
results["test"].push_to_hub("gaia-benchmark/results_public", "2023", token=token, split="test")
```

```
ValueError                                Traceback (most recent call last)
Cell In[43], line 1
----> 1 results["test"].push_to_hub("gaia-benchmark/results_public", "2023", token=token, split="test")

File ~/miniconda3/envs/default310/lib/python3.10/site-packages/datasets/arrow_dataset.py:5498, in Dataset.push_to_hub(self, repo_id, config_name, split, private, token, branch, max_shard_size, num_shards, embed_external_files)
   5496         repo_info.dataset_size = (repo_info.dataset_size or 0) + dataset_nbytes
   5497         repo_info.size_in_bytes = repo_info.download_size + repo_info.dataset_size
-> 5498         repo_info.splits[split] = SplitInfo(
   5499             split, num_bytes=dataset_nbytes, num_examples=len(self), dataset_name=dataset_name
   5500         )
   5501         info_to_dump = repo_info
   5502 # create the metadata configs if it was uploaded with push_to_hub before metadata configs existed

File ~/miniconda3/envs/default310/lib/python3.10/site-packages/datasets/splits.py:541, in SplitDict.__setitem__(self, key, value)
    539     raise ValueError(f"Cannot add elem. (key mismatch: '{key}' != '{value.name}')")
    540 if key in self:
--> 541     raise ValueError(f"Split {key} already present")
    542 super().__setitem__(key, value)

ValueError: Split test already present
```
`write_examples_on_file` and `write_batch` are broken in `ArrowWriter`

### Describe the bug

`write_examples_on_file` and `write_batch` are broken in `ArrowWriter` since #6636. The order between the columns and the schema is not preserved anymore. So these functions don't work anymore unless the order happens to align well.

### Steps to reproduce the bug

Try to do `write_batch` with anything that has many columns, and it's likely to break.

### Expected behavior

I expect these functions to work, instead of it trying to cast a column to its incorrect type.

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-5.15.0-1040-aws-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.19.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
Import error on Google Colab

### Describe the bug

Cannot be imported on Google Colab, the import throws the following error:
ValueError: pyarrow.lib.IpcWriteOptions size changed, may indicate binary incompatibility. Expected 88 from C header, got 72 from PyObject

### Steps to reproduce the bug

1. `! pip install -U datasets`
2. `import datasets`

### Expected behavior

Should be possible to use the library

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.6.0
Change default compression argument for JsonDatasetWriter

Change default compression type from `None` to "infer", to align with pandas' defaults.

Documentation asks the user to supply `to_json_kwargs` with arguments suitable for pandas' `to_json` method. At the same time, while pandas' by default uses ["infer"](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html) for compression, datasets enforce `None` as default. This, likely, confuses user, as they expect the same behaviour, i.e they expect that if they name their output file as "dataset.jsonl.zst" then the compression would be inferred as "zstd" and file will be compressed before writing.

Moreover, while it is probably outside of the scope of this pull request, `compression` argument needs to be capable of taking `dict` as input (along with `str`), as it does in pandas, in order to allow user to specify compression parameters. Current implementation will likely fail with `NotImplementedError`, as it expects either `None` or `str` specifying compression algo.
[Resumable IterableDataset] Add IterableDataset state_dict

A simple implementation of a mechanism to resume an IterableDataset.
It works by restarting at the latest shard and skip samples. It provides fast resuming (though not instantaneous).

Example:

```python
from datasets import Dataset, concatenate_datasets


ds = Dataset.from_dict({"a": range(5)}).to_iterable_dataset(num_shards=3)
ds = concatenate_datasets([ds] * 2)

print(f"{ds.state_dict()=}")
for i, example in enumerate(ds):
    print(example)
    if i == 6:
        state_dict = ds.state_dict()
        print("checkpoint")
ds.load_state_dict(state_dict)
print(f"resuming from checkpoint {ds.state_dict()=}")
for example in ds:
    print(example)
```

returns

```
ds.state_dict()={'ex_iterable_idx': 0, 'ex_iterables': [{'shard_idx': 0, 'shard_example_idx': 0}, {'shard_idx': 0, 'shard_example_idx': 0}]}
{'a': 0}
{'a': 1}
{'a': 2}
{'a': 3}
{'a': 4}
{'a': 0}
{'a': 1}
checkpoint
{'a': 2}
{'a': 3}
{'a': 4}
resuming from checkpoint ds.state_dict()={'ex_iterable_idx': 1, 'ex_iterables': [{'shard_idx': 3, 'shard_example_idx': 0}, {'shard_idx': 0, 'shard_example_idx': 2}]}
{'a': 2}
{'a': 3}
{'a': 4}
```

using torchdata:
```python
from datasets import load_dataset
from torchdata.stateful_dataloader import StatefulDataLoader
my_iterable_dataset = load_dataset("deepmind/code_contests", streaming=True, split="train")
dataloader = StatefulDataLoader(my_iterable_dataset, batch_size=32, num_workers=4)
# save in the middle of training
state_dict = dataloader.state_dict()
# and resume later
dataloader.load_state_dict(state_dict)
```

docs: https://huggingface.co/docs/datasets/main/en/use_with_pytorch#checkpoint-and-resume
Release not pushed to conda channel

### Describe the bug

The github actions step to publish the release 2.17.0 to conda channel has failed due to expired token. Can some one please update the anaconda token rerun the failed action? @albertvillanova ?
 
![image](https://github.com/huggingface/datasets/assets/7138162/1b56ad3d-7643-4778-9cce-4bf531717700)


### Steps to reproduce the bug

Please see this actions [link](https://github.com/huggingface/datasets/actions/runs/7842473662)

### Expected behavior

The action runs successfully and the latest release is pushed to HuggingFace conda channel

### Environment info

Not applicable.
Differently dataset object saved than it is loaded.

### Describe the bug

Differently sized object is saved than it is loaded.

### Steps to reproduce the bug

Hi, I save dataset in a following way:
```
        dataset = load_dataset("json",
                               data_files={
                                   "train": os.path.join(input_folder, f"{task_meta_type}_{task_type}_train.jsonl"),
                                   "test": os.path.join(input_folder, f"{task_meta_type}_{task_type}_test.jsonl")})
        print(os.path.join(output_folder, f"{task_meta_type}_{task_type}"))
        print(f"Length of train dataset: {len(dataset['train'])}")
        print(f"Length of test dataset: {len(dataset['test'])}")
        dataset.save_to_disk(os.path.join(output_folder, f"{task_meta_type}_{task_type}"))
```

this yields output
```
.data/hf_dataset/propaganda_zanr
Length of train dataset: 7642
Length of test dataset: 1000
```

Everything looks fine.

Then I load the dataset
```python
from datasets import load_dataset

dataset_path = ".data/hf_dataset/propaganda_zanr"
dataset = load_dataset(dataset_path)

print(f"Length of train dataset: {len(dataset['train'])}")
print(f"Length of test dataset: {len(dataset['test'])}")

```

this prints
```
Generating train split: 1 examples [00:00, 72.10 examples/s]
Generating test split: 1 examples [00:00, 100.69 examples/s]
Length of train dataset: 1
Length of test dataset: 1
```

I dont' understand :(


### Expected behavior

same object is loaded

### Environment info

datasets==2.16.1
'with_format' is extremely slow when used together with 'interleave_datasets' or 'shuffle' on IterableDatasets

### Describe the bug

If you:

1. Interleave two iterable datasets together with the interleave_datasets function, or shuffle an iterable dataset
2. Set the output format to torch tensors with .with_format('torch')

Then iterating through the dataset becomes over 100x slower than it is if you don't apply the torch formatting.

### Steps to reproduce the bug

```python
import datasets
import torch
from tqdm import tqdm

rand_a = torch.randn(3,224,224)
rand_b = torch.randn(3,224,224)

a = torch.stack([rand_a] * 1000)
b = torch.stack([rand_b] * 1000)

features = datasets.Features({"tensor": datasets.Array3D(shape=(3,224,224), dtype="float32")})
ds_a = datasets.Dataset.from_dict({"tensor": a}, features=features).to_iterable_dataset()
ds_b = datasets.Dataset.from_dict({"tensor": b}, features=features).to_iterable_dataset()

# Iterating through either dataset with torch formatting is really fast (2000it/s on my machine)
for example in tqdm(ds_a.with_format('torch')):
    pass

# Iterating through either dataset shuffled is also pretty fast (100it/s on my machine)
for example in tqdm(ds_a.shuffle()):
    pass

# Iterating through this interleaved dataset is pretty fast (200it/s on my machine)
ds_fast = datasets.interleave_datasets([ds_a, ds_b])
for example in tqdm(ds_fast):
    pass

# Iterating through either dataset with torch formatting *after shuffling* is really slow... (<2it/s on my machine)
for example in tqdm(ds_a.shuffle().with_format('torch')):
    pass

# Iterating through this torch formatted interleaved dataset is also really slow (<2it/s on my machine)...
ds_slow = datasets.interleave_datasets([ds_a, ds_b]).with_format('torch')
for example in tqdm(ds_slow):
    pass

# Even doing this is way faster!! (70it/s on my machine)
for example in tqdm(ds_fast):
    test = torch.tensor(example['tensor'])

```

### Expected behavior

Applying torch formatting to the interleaved dataset shouldn't increase the time taken to iterate through the dataset by very much, since even explicitly converting every example is over 70x faster than calling .with_format('torch').

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.5.0-15-generic-x86_64-with-glibc2.38
- Python version: 3.11.6
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
Faster column validation and reordering

I work with bioinformatics data and often these tables have thousands and even tens of thousands of features. These tables are also accompanied by metadata that I do not want to pass in the model. When I perform `set_format('pt', columns=large_column_list)` , it can take several minutes before it finishes. The culprit is when the following check is performed: `any(col not in self._data.column_names for col in columns)`. Replacing this by  `set(columns) - (self._data.column_names)` is more efficient.
Fix missing info when loading some datasets from Parquet export

Fix getting the info for script-based datasets with Parquet export with a single config not named "default".

E.g.

```python
from datasets import load_dataset_builder

b = load_dataset_builder("bookcorpus")
print(b.info.features)
# should print {'text': Value(dtype='string', id=None)}
```

I fixed this by setting the default config name when there is only one config.
Fix reload cache with data dir

The cache used to only check for the latest cache directory with a given config_name, but it was wrong (e.g. `default-data_dir=data%2Ffortran-data_dir=data%2Ffortran` instead of `default-data_dir=data%2Ffortran`)

I fixed this by not passing the `config_kwargs` to the parent Builder `__init__`, and passing the config_id forged from the `config_kwargs` directly

close https://github.com/huggingface/datasets/issues/6609
Support push_to_hub without org/user to default to logged-in user

This behavior is aligned with:
- the behavior of `datasets` before merging #6519
  - the behavior described in the corresponding docstring
- the behavior of `huggingface_hub.create_repo`

Revert "Support push_to_hub canonical datasets (#6519)"
- This reverts commit a887ee78835573f5d80f9e414e8443b4caff3541.

Fix #6597.
streaming datasets doesn't work properly with multi-node

### Feature request

Let’s say I have a dataset with 5 samples with values [1, 2, 3, 4, 5], with 2 GPUs (for DDP) and batch size of 2. This dataset is an `IterableDataset` since I am streaming it.

Now I split the dataset using `split_dataset_by_node` to ensure it doesn’t get repeated. And since it’s already splitted, I don’t have to use `DistributedSampler` (also they don't work with iterable datasets anyway)?

But in this case I noticed that the:

First iteraton:
first GPU will get → [1, 2]
first GPU will get → [3, 4]

Second iteraton:
first GPU will get → [5]
first GPU will get → Nothing

which actually creates an issue since in case of `DistributedSampler`, the samples are repeated internally to ensure non of the GPUs at any iteration is missing any data for gradient sync.

So my questions are:

1. Here since splitting is happening before hand, how to make sure each GPU get’s a batch at each iteration to avoid gradient sync issues?
2. Do we need to use `DistributedSampler`? If yes, how?
3. in the docstrings of `split_dataset_by_node`, this is mentioned: *"If the dataset has a number of shards that is a factor of `world_size` (i.e. if `dataset.n_shards % world_size == 0`), then the shards are evenly assigned across the nodes, which is the most optimized. Otherwise, each node keeps 1 example out of `world_size`, skipping the other examples."* Can you explain the last part here?
4. If `dataset.n_shards % world_size != 0`, is it possible to shard the streaming dataset on the fly to avoid the case where data is missing?

### Motivation

Somehow streaming datasets should work with DDP since for big LLMs a lot of data is required and DDP/multi-node is mostly used to train such models and streaming can actually help solve the data part of it.

### Your contribution

Yes, I can help in submitting the PR once we get mutual understanding on how it should behave.
multi-GPU map does not work

### Describe the bug

Here is the code for single-GPU processing: https://pastebin.com/bfmEeK2y
Here is the code for multi-GPU processing: https://pastebin.com/gQ7i5AQy

Here is the video showing that the multi-GPU mapping does not work as expected (there are so many things wrong here, it's better to watch the 3-minute video than explain here):

https://youtu.be/RNbdPkSppc4

### Steps to reproduce the bug

-

### Expected behavior

-

### Environment info

x2 RTX A4000
wiki_dpr.py error (ID mismatch between lines {id} and vector {vec_id}

### Describe the bug

I'm trying to run a rag example, and the dataset is wiki_dpr.
wiki_dpr download and extracting have been completed successfully.
However, at the generating train split stage, an error from wiki_dpr.py keeps popping up.

Especially in "_generate_examples" :
1. The following error occurs in the line **id, text, title = line.strip().split("\t")**
ValueError: not enough values ​​to unpack (expected 3, got 2)
-> This part handles exceptions so that even if an error occurs, it passes.

2. **ID mismatch between lines {id} and vector {vec_id}**
This error seems to occur at the line " assert int(id) == int(vec_id),".
After I handled the exception in the split error, generating train split progressed to 80%, but an id mismatch error occurred at about the 16200000th vector id.

Debugging is even more difficult because it takes a long time to download and split wiki_dpr. I need help. thank you in advance!!

### Steps to reproduce the bug

Occurs in the generating train split step when running the rag example in the transformers repository.
Specifically, it is an error in wiki_dpr.py.

### Expected behavior

.

### Environment info

python 3.8
cnn_dailymail repeats itself

### Describe the bug

When I try to load `cnn_dailymail` dataset, it takes longer than usual and when I checked the dataset it's 3x bigger than it's supposed to be.

Check https://huggingface.co/datasets/cnn_dailymail: it says 287k rows for train. But when I check length of train split it says 861339.

Also I checked data:

```
>>> ds['train']['highlights'][0]                                           
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday . Young actor says he has no plans to fritter his cash away . Radcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][0]                                           
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday . Young actor says he has no plans to fritter his cash away . Radcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][287113]
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ."````

>>> ds['train']['highlights'][574226]
"Harry Potter star Daniel Radcliffe gets £20M fortune as he turns 18 Monday .\nYoung actor says he has no plans to fritter his cash away .\nRadcliffe's earnings from first five Potter films have been held in trust fund ."
```

The datasets seems to be updated 6 days ago to convert it to Parquet. Probably, there is some issue with backward compatability.



### Steps to reproduce the bug

1.
```
from datasets import load_dataset
ds = load_dataset('cnn_dailymail', '3.0.0')
len(ds['train'])
```

### Expected behavior

It should not repeat itself.

### Environment info

datasets==2.13.2
Python==3.7.13