sjwsjw/github-issues huggingface dataset:
GitHub issue: Precision being changed from float16 to float32 unexpectedly
State: open

### Describe the bug

I'm loading a HuggingFace Dataset for images. 

I'm running a preprocessing (map operation) step that runs a few operations, one of them being conversion to float16.  The Dataset features also say that the 'img' is of type float16.  Whenever I take an image from that HuggingFace Dataset instance, the type turns out to be float32.  



### Steps to reproduce the bug

```python
import torchvision.transforms.v2 as transforms
from datasets import load_dataset

dataset = load_dataset('cifar10', split='test')
dataset = dataset.with_format("torch")

data_transform = transforms.Compose([transforms.Resize((32, 32)), 
                                     transforms.ToDtype(torch.float16, scale=True),
                                     transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]),
                                     ])
def _preprocess(examples):
    # Permutes from (BS x H x W x C) to (BS x C x H x W)
    images = torch.permute(examples['img'], (0, 3, 2, 1))
    examples['img'] = data_transform(images)
    return examples

dataset = dataset.map(_preprocess, batched=True, batch_size=8)
```

Now at this point the dataset.features are showing float16 which is great because that's what I want. 

```python
print(data_loader.features['img'])

Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='float16', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None)
```

But when I try to sample an image from this dataloader; I'm getting a float32 image, when I'm expecting float16:
```python
print(next(iter(data_loader))['img'].dtype)

torch.float32
```

### Expected behavior

I'm expecting the images loaded after the transformation to stay in float16.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.9
- `huggingface_hub` version: 0.21.4
- PyArrow version: 14.0.2
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0
GitHub issue: Strange slicing behavior
State: open

### Describe the bug

I have loaded a dataset, and then slice first 300 samples using `:` ops, however, the resulting dataset is not expected, as the output below:

```bash
len(dataset)=1050324
len(dataset[:300])=2
len(dataset[0:300])=2
len(dataset.select(range(300)))=300
```

### Steps to reproduce the bug

load a dataset then:
```bash
    dataset = load_from_disk(args.train_data_dir)

    print(f"{len(dataset)=}", flush=True)
    print(f"{len(dataset[:300])=}", flush=True)
    print(f"{len(dataset[0:300])=}", flush=True)
    print(f"{len(dataset.select(range(300)))=}", flush=True)
```

### Expected behavior

```bash
len(dataset)=1050324
len(dataset[:300])=300
len(dataset[0:300])=300
len(dataset.select(range(300)))=300
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- `huggingface_hub` version: 0.20.2
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0
GitHub issue: ExpectedMoreSplits error when loading C4 dataset
State: closed

### Describe the bug

I encounter bug when running the example command line 

```python
    python main.py \
    --model decapoda-research/llama-7b-hf \
    --prune_method wanda \
    --sparsity_ratio 0.5 \
    --sparsity_type unstructured \
    --save out/llama_7b/unstructured/wanda/ 
```

The bug occurred at these lines of code (when loading c4 dataset) 

```python
traindata = load_dataset('allenai/c4', 'allenai--c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train')
valdata = load_dataset('allenai/c4', 'allenai--c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation')
```

The error message states:
```    
raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))                                                                           
datasets.utils.info_utils.ExpectedMoreSplits: {'validation'}
```



### Steps to reproduce the bug

1. I encounter bug when running the example command line 


### Expected behavior

The error message states:
```    
raise ExpectedMoreSplits(str(set(expected_splits) - set(recorded_splits)))                                                                           
datasets.utils.info_utils.ExpectedMoreSplits: {'validation'}
```



### Environment info

I'm using cuda 12.4, so I use ```pip install pytorch``` instead of conda provided in install.md

Also, I've tried another environment using the same commands in install.md, but the same bug occured
GitHub issue: Option to disable file locking
State: open

### Feature request

Commands such as `load_dataset` creates file locks with `filelock.FileLock`. It would be good if there was a way to disable this.

### Motivation

File locking doesn't work on all file-systems (in my case NFS mounted Weka). If the `cache_dir` only had small files then it would be possible to point to local disk and the problem would be solved. However, as cache_dir is both where the small info files are written and the processed datasets are put this isn't a feasible solution.

Considering https://github.com/huggingface/datasets/issues/6395 I still do think this is something that belongs in HuggingFace. The possibility to control packages separately is valuable. It might be that a user has their dataset on a file-system that doesn't support file-locking while they are using file locking on local disk to control some other type of access.

### Your contribution

My suggested solution:
```
diff --git a/src/datasets/utils/_filelock.py b/src/datasets/utils/_filelock.py
index 19620e6e..58f41a02 100644
--- a/src/datasets/utils/_filelock.py
+++ b/src/datasets/utils/_filelock.py
@@ -18,11 +18,15 @@
 import os
 
 from filelock import FileLock as FileLock_
-from filelock import UnixFileLock
+from filelock import SoftFileLock, UnixFileLock
 from filelock import __version__ as _filelock_version
 from packaging import version
 
 
+if os.getenv('HF_USE_SOFTFILELOCK', 'false').lower() in ('true', '1'):
+    FileLock_ = SoftFileLock
+
+
 class FileLock(FileLock_):
     """
     A `filelock.FileLock` initializer that handles long paths.
```
GitHub issue: Fix offline mode with single config
State: closed

Reported in https://github.com/huggingface/datasets/issues/4760

The cache was not able to reload a dataset with a single config form the cache if the config name is not specificed

For example

```python
from datasets import load_dataset, config

config.HF_DATASETS_OFFLINE = True

load_dataset("openai_humaneval")
```

This was due to a regression in https://github.com/huggingface/datasets/pull/6632
GitHub issue: Support for loading geotiff files as a part of the ImageFolder
State: closed

### Feature request

Request for adding rasterio support to load geotiff as a part of ImageFolder, instead of using PIL

### Motivation

As of now, there are many datasets in HuggingFace Hub which are predominantly focussed towards RemoteSensing or are from RemoteSensing. The current ImageFolder (if I have understood correctly) uses PIL. This is not really optimized because mostly these datasets have images with many channels and additional metadata. Using PIL makes one loose it unless we provide a custom script. Hence, maybe an API could be added to have this in common?

### Your contribution

If the issue is accepted - i can contribute the code, because I would like to have it automated and generalised.
GitHub issue: Dict feature is non-nullable while nested dict feature is
State: closed

When i try to create a `Dataset` object with None values inside a dict column, like this: 
```python
from datasets import Dataset, Features, Value

Dataset.from_dict(
    {
        "dict": [{"a": 0, "b": 0}, None],
    }, features=Features(
        {"dict": {"a": Value("int16"), "b": Value("int16")}}
    )
)
```
i get `ValueError: Got None but expected a dictionary instead`.


At the same time, having None in _nested_ dict feature works, for example, this doesn't throw any errors:

```python
from datasets import Dataset, Features, Value, Sequence

dataset = Dataset.from_dict(
    {
        "list_dict": [[{"a": 0, "b": 0}], None],
        "sequence_dict": [[{"a": 0, "b": 0}], None],
    }, features=Features({
        "list_dict": [{"a": Value("int16"), "b": Value("int16")}],
        "sequence_dict": Sequence({"a": Value("int16"), "b": Value("int16")}),
    })
)
```
Other types of features also seem to be nullable (but I haven't checked all of them).

Version of `datasets` is the latest atm (2.18.0)


Is this an expected behavior or a bug?
GitHub issue: Invalid pattern: '**' can only be an entire path component
State: closed

### Describe the bug

ValueError: Invalid pattern: '**' can only be an entire path component
when loading any dataset


### Steps to reproduce the bug

import datasets
ds = datasets.load_dataset("TokenBender/code_instructions_122k_alpaca_style")

### Expected behavior

loading the dataset successfully 

### Environment info

- `datasets` version: 2.18.0
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.11.7
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2023.12.2
GitHub issue: Mosaic Streaming (MDS) Support
State: open

### Feature request

I'm a huge fan of the current HF Datasets `webdataset` integration (especially the built-in streaming support). However, I'd love to upload some robotics and multimodal datasets I've processed for use with [Mosaic Streaming](https://docs.mosaicml.com/projects/streaming/en/stable/), specifically their [MDS Format](https://docs.mosaicml.com/projects/streaming/en/stable/fundamentals/dataset_format.html#mds). 

Because the shard files have similar semantics to WebDataset, I'm hoping that adding such support won't be too much trouble?

### Motivation

One of the downsides with WebDataset is a lack of out-of-the-box determinism (especially for large-scale training and reproducibility), easy job resumption, and the ability to quickly debug / visualize individual examples. 

Mosaic Streaming provides a [great interface for this out of the box](https://docs.mosaicml.com/projects/streaming/en/stable/#key-features), so I'd love to see it supported in HF Datasets.

### Your contribution

Happy to help test things / provide example data. Can potentially submit a PR if maintainers could point me to the necessary WebDataset logic / steps for adding a new streaming format!
GitHub issue: Tokenization slows towards end of dataset
State: open

### Describe the bug

Mapped tokenization slows down substantially towards end of dataset.

train set started off very slow, caught up to 20k then tapered off til the end.

what's particularly strange is that the tokenization crashed a few times before due to errors with invalid tokens somewhere or corrupted downloads, and the speed ups/downs consistently happened the same times

```bash
Running tokenizer on dataset (num_proc=48):   0%|          | 847000/881416735 [12:18<252:45:45, 967.72 examples/s]
Running tokenizer on dataset (num_proc=48):   0%|          | 848000/881416735 [12:19<224:16:10, 1090.66 examples/s]

Running tokenizer on dataset (num_proc=48):  10%|▉         | 84964000/881416735 [3:48:00<11:21:34, 19476.01 examples/s]
Running tokenizer on dataset (num_proc=48):  10%|▉         | 84967000/881416735 [3:48:00<12:04:01, 18333.79 examples/s]

Running tokenizer on dataset (num_proc=48):  61%|██████    | 538631977/881416735 [13:46:40<27:50:04, 3420.84 examples/s]
Running tokenizer on dataset (num_proc=48):  61%|██████    | 538632977/881416735 [13:46:40<23:48:20, 3999.77 examples/s]

Running tokenizer on dataset (num_proc=48): 100%|█████████▉| 881365886/881416735 [38:30:19<04:34, 185.10 examples/s]
Running tokenizer on dataset (num_proc=48): 100%|█████████▉| 881366886/881416735 [38:30:25<04:36, 180.57 examples/s]
```

and validation set as well

```bash
Running tokenizer on dataset (num_proc=48):  90%|████████▉ | 41544000/46390354 [28:44<02:37, 30798.76 examples/s]
Running tokenizer on dataset (num_proc=48):  90%|████████▉ | 41550000/46390354 [28:44<02:08, 37698.08 examples/s]

Running tokenizer on dataset (num_proc=48):  96%|█████████▋| 44747422/46390354 [2:15:48<12:22:44, 36.87 examples/s]
Running tokenizer on dataset (num_proc=48):  96%|█████████▋| 44747422/46390354 [2:16:00<12:22:44, 36.87 examples/s]

```



### Steps to reproduce the bug

    using the following kwargs
    
```python
with accelerator.main_process_first():
        lm_datasets = tokenized_datasets.map(
            group_texts,
            batched=True,
            num_proc=48
            load_from_cache_file=True,
            desc=f"Grouping texts in chunks of {block_size}",
        )
```

running through slurm script 
```bash
#SBATCH --partition=gpu-nvidia-a100
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gpus-per-task=8
#SBATCH --cpus-per-task=96
```

using this dataset https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T





### Expected behavior

Constant speed throughout

### Environment info

- `datasets` version: 2.15.0
- Platform: Linux-5.15.0-1049-aws-x86_64-with-glibc2.10
- Python version: 3.8.18
- `huggingface_hub` version: 0.19.4
- PyArrow version: 14.0.1
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0
GitHub issue: EmptyDatasetError when loading dataset downloaded with HuggingFace cli
State: open

### Describe the bug

I am using a cluster that does not have access to the internet when given a job. I tried downloading the dataset using the huggingface-cli command and then loading it with load_dataset but I get an error:

```raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None```

The dataset I'm using is "lmsys/chatbot_arena_conversations". The folder structure is

- README.md
- data
  - train-00000-of-00001-cced8514c7ed782a.parquet



### Steps to reproduce the bug

1. Download dataset using HuggingFace CLI: ```huggingface-cli download lmsys/chatbot_arena_conversations --local-dir ./lmsys/chatbot_arena_conversations```
2. In Python
```
from datasets import load_dataset
load_dataset("lmsys/chatbot_arena_conversations")
```

### Expected behavior

Should return a Dataset Dict in the form of
```
DatasetDict({
    train: Dataset({
        features: [...],
        num_rows: 33,000
    })
})
```
### Environment info

Python 3.11.5
Datasets 2.18.0
Transformers 4.38.2
Pytorch 2.2.0
Pyarrow 15.0.1
Rocky Linux release 8.9 (Green Obsidian)
GitHub issue: Unexpected behavior when using load_dataset with streaming=True in a for loop
State: closed

### Describe the bug

### My Code
```
from datasets import load_dataset
res=[]
for i in [0,1]:
    di=load_dataset(
            "json", 
            data_files='path_to.json', 
            split='train',
            streaming=True, 
        ).map(lambda x: {"source": i})

    res.append(di)

for e in res[0]:
    print(e)
```

### Unexpected Behavior
Data in `res[0]` has `source=1`. However the expected value is 0.

### FYI
I further switch `streaming` to `False`. And the output value is as expected (0). So there may exist bugs in setting `streaming=True` in a for loop.

### Environment
Python 3.8.0
datasets==2.18.0
transformers==4.28.1

### Steps to reproduce the bug

1. Create a Json file with any content.
2. Run the provided code.
3. Switch `streaming` to `False` and run again to see the expected behavior.

### Expected behavior

The expected behavior is the data are mapped with its corresponding value in the for loop.

### Environment info

Python 3.8.0

datasets==2.18.0
transformers==4.28.1

Ubuntu 20.04
GitHub issue: Using a registry instead of calling globals for fetching feature types
State: closed

Hello,

When working with bio-data, each feature often has metadata associated with it (e.g. species, lineage, snp position, etc). To store this, I like to use the feature classes with the added `metadata` attribute. However, when saving or loading with custom features, you get an error since that class doesn't exist in the global namespace in `datasets.features.features`. Take for example,



```python
from dataclasses import dataclass, field
from datasets import Dataset
from datasets.features.features import Value, Features

@dataclass
class FeatureA(Value):
    metadata: dict = field(default=dict)
    _type: str = field(default="FeatureA", init=False, repr=False)

@dataclass
class FeatureB(Value):
    metadata: dict = field(default_factory=dict)
    _type: str = field(default="FeatureB", init=False, repr=False)

test_data = {
    "a": [1, 2, 3],
    "b": [4, 5, 6],
}
test_data = Dataset.from_dict(
    test_data, 
    features=Features({
        "a": FeatureA("int32", metadata={"species": "lactobacillus acetotolerans"}),
        "b": FeatureB("int32", metadata={"species": "lactobacillus iners"}),
    })
)

# returns an error since FeatureA and FeatureB are not in the global namespace
test_data.save_to_disk('./test_data')
```


    Saving the dataset (0/1 shards):   0%|          | 0/3 [00:00<?, ? examples/s]



    ---------------------------------------------------------------------------

    KeyError                                  Traceback (most recent call last)

    Cell In[2], line 28
         19 test_data = Dataset.from_dict(
         20     test_data, 
         21     features=Features({
       (...)
         24     })
         25 )
         27 # returns an error since FeatureA and FeatureB are not in the global namespace
    ---> 28 test_data.save_to_disk('./test_data')
    ...
    File ~\Documents\datasets\src\datasets\features\features.py:1361, in generate_from_dict(obj)
       1359     return {key: generate_from_dict(value) for key, value in obj.items()}
       1360 obj = dict(obj)
    -> 1361 class_type = globals()[obj.pop("_type")]
       1363 if class_type == Sequence:
       1364     return Sequence(feature=generate_from_dict(obj["feature"]), length=obj.get("length", -1))
    

    KeyError: 'FeatureA'


We can avoid this by having a registry (like formatters) and doing


```python
from datasets.features.features import register_feature
register_feature(FeatureA, "FeatureA")
register_feature(FeatureB, "FeatureB")
test_data.save_to_disk('./test_data')
```


    Saving the dataset (1/1 shards): 100%|------| 3/3 [00:00<00:00, 211.13 examples/s]


and loading from disk returns with all metadata information


```python
from datasets import load_from_disk
test_data = load_from_disk('./test_data')
test_data.features
```




    {'a': FeatureA(dtype='int32', id=None, metadata={'species': 'lactobacillus acetotolerans'}),
     'b': FeatureB(dtype='int32', id=None, metadata={'species': 'lactobacillus iners'})}
GitHub issue: Request for a comparison of huggingface datasets compared with other data format especially webdataset
State: open

### Feature request

Request for a comparison of huggingface datasets compared with other data format especially webdataset

### Motivation

I see huggingface datasets uses Apache Arrow as its backend, it seems to be great, but I'm curious about how it is good compared with other dataset format, like webdataset, what's the pros/cons of them.

### Your contribution

More information
GitHub issue: Dataset with loading script does not work in renamed repos
State: open

### Describe the bug

My data repository was first called `BramVanroy/hplt-mono-v1-2` but I then renamed to use underscores instead of dashes. However, it seems that `datasets` retrieves the old repo name when it checks whether the repo contains data loading scripts in this line.

https://github.com/huggingface/datasets/blob/6fb6c834f008996c994b0a86c3808d0a33d44525/src/datasets/load.py#L1845

When I print `filename` it returns `hplt-mono-v1-2.py` but the files in the repo are of course `['.gitattributes', 'README.md', 'hplt_mono_v1_2.py']`. So the `filename` is the original reponame instead of the renamed one.

I am not sure if this is a caching issue or not or how I can resolve it.

### Steps to reproduce the bug

```
from datasets import load_dataset


ds = load_dataset(
    "BramVanroy/hplt-mono-v1-2",
    "ky",
    trust_remote_code=True
)
```

### Expected behavior

That the most recent repo name is used when `filename` is generated.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.14.0-284.25.1.el9_2.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0
GitHub issue: Is there any way to solve hanging of IterableDataset using split by node + filtering during inference
State: open

### Describe the bug

I am using an iterable dataset in a multi-node setup, trying to do training/inference while filtering the data on the fly. I usually do not use `split_dataset_by_node` but it is very slow using the IterableDatasetShard in `accelerate` and `transformers`. When I filter after applying `split_dataset_by_node`, it results in shards that are not equal sizes due to unequal samples filtered from each one. 

The distributed process hangs when trying to accomplish this. Is there any way to resolve this or is it impossible to implement?



### Steps to reproduce the bug

Here is a toy example of what I am trying to do that reproduces the behavior

```
# torchrun --nproc-per-node 2 file.py


import os

import pandas as pd
import torch
from accelerate import Accelerator
from datasets import Features, Value, load_dataset
from datasets.distributed import split_dataset_by_node
from torch.utils.data import DataLoader

accelerator = Accelerator(device_placement=True, dispatch_batches=False)
if accelerator.is_main_process:
    if not os.path.exists("scratch_data"):
        os.mkdir("scratch_data")

    n_shards = 4
    for i in range(n_shards):
        df = pd.DataFrame({"id": list(range(10 * i, 10 * (i + 1)))})
        df.to_parquet(f"scratch_data/shard_{i}.parquet")


world_size = accelerator.num_processes
local_rank = accelerator.process_index


def collate_fn(examples):
    input_ids = []
    for example in examples:
        input_ids.append(example["id"])
    return torch.LongTensor(input_ids)


dataset = load_dataset(
    "parquet", data_dir="scratch_data", split="train", streaming=True
)
dataset = (
    split_dataset_by_node(dataset, rank=local_rank, world_size=world_size)
    .filter(lambda x: x["id"] < 35)
    .shuffle(seed=42, buffer_size=100)
)

batch_size = 2
train_dataloader = DataLoader(
    dataset,
    batch_size=batch_size,
    collate_fn=collate_fn,
    num_workers=2
)  

for x in train_dataloader:
    x = x.to(accelerator.device)
    print({"rank": local_rank, "id": x})
   
    y = accelerator.gather_for_metrics(x)
    if accelerator.is_main_process:
        print("gathered", y)


```

### Expected behavior

Is there any way to continue training/inference on the GPUs that have remaining data left without waiting for the others? Is it impossible to filter when 

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.10.209-198.812.amzn2.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.21.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2023.6.0
GitHub issue: Non-deterministic `Dataset.builder_name` value
State: closed

### Describe the bug

I'm not sure if this is a bug, but `print(ds.builder_name)` in the following code sometimes prints out `rotten_tomatoes` instead of `parquet`:

```python
import datasets

for _ in range(100):
    ds = datasets.load_dataset("rotten_tomatoes", split="train")
    print(ds.builder_name)  # prints out "rotten_tomatoes" sometimes instead of "parquet"
```

Output:

```
...
parquet
parquet
parquet
rotten_tomatoes
parquet
parquet
parquet
...
```

Here's a reproduction using GitHub Actions:
https://github.com/mlflow/mlflow/actions/runs/8153247984/job/22284263613?pr=11329#step:12:241

One of our tests is flaky because `builder_name` is not deterministic.

### Steps to reproduce the bug

1. Run the code above.

### Expected behavior

Always prints out `parquet`?

### Environment info

```
Copy-and-paste the text below in your GitHub issue.
- `datasets` version: 2.18.0
- Platform: Linux-6.5.0-1015-azure-x86_64-with-glibc2.34
- Python version: 3.8.18
- `huggingface_hub` version: 0.21.3
- PyArrow version: 15.0.0
- Pandas version: 2.0.3
- `fsspec` version: 2024.2.0
```
GitHub issue: Fix sliced ConcatenationTable pickling with mixed schemas vertically
State: closed

A sliced + pickled ConcatenationTable could end up with a different schema than the original schema, if the slice only contains blocks with only a subset of the columns.

This can lead to issues when saving datasets from a concatenation of datasets with mixed schemas

Reported in https://discuss.huggingface.co/t/datasetdict-save-to-disk-with-num-proc-1-seems-to-hang-with-error/75595
GitHub issue: Bump huggingface-hub lower version to 0.21.2
State: closed

This should fix the version compatibility issue when using `huggingface_hub` < 0.21.2 and latest fsspec (>=2023.12.0).

See my comment: https://github.com/huggingface/datasets/pull/6687#issuecomment-1976493336
>> EDIT: the fix has been released in `huggingface_hub` 0.21.2 - I removed my commits that were using `huggingface_hub@main`
>
>Please note that people using `huggingface_hub` < 0.21.2 and latest `fsspec` will have issues when using `datasets`:
>- https://github.com/huggingface/lighteval/actions/runs/8139147047/job/22241658122?pr=86
>- https://github.com/huggingface/lighteval/pull/84

CC: @clefourrier
GitHub issue: 3x Faster Text Preprocessing
State: open

I was preparing some datasets for AI training and noticed that `datasets` by HuggingFace uses the conventional `open` mechanism to read the file and split it into chunks. I thought it can be significantly accelerated, and [started with a benchmark](https://gist.github.com/ashvardanian/55c2052e9f78b05b8d614aa90cb12347):

```sh
$ pip install --upgrade --force-reinstall datasets
$ python benchmark_huggingface_datasets.py xlsum.csv 
Generating train split: 1004598 examples [00:47, 21116.16 examples/s]
Time taken to load the dataset: 48.66838526725769 seconds
Time taken to chunk the dataset into parts of size 10000: 0.11466407775878906 seconds
Total time taken: 48.78304934501648 seconds
```

For benchmarks I've used a [large CSV file with mixed UTF-8 content](https://github.com/ashvardanian/StringZilla/blob/main/CONTRIBUTING.md#benchmarking-datasets), most common in modern large-scale pre-training pipelines. I've later patched the `datasets` library to use `stringzilla`, which resulted in significantly lower memory consumption and in 2.9x  throughput improvement on the AWS `r7iz` instances. That's using slow SSDs mounted over the network. Performance on local SSDs on something like a DGX-H100 should be even higher:

```sh
$ pip install -e .
$ python benchmark_huggingface_datasets.py xlsum.csv 
Generating train split: 1004598 examples [00:15, 64529.90 examples/s]
Time taken to load the dataset: 16.45028805732727 seconds
Time taken to chunk the dataset into parts of size 10000: 0.1291060447692871 seconds
Total time taken: 16.579394102096558 seconds
```

I've already [pushed the patches to my fork](https://github.com/ashvardanian/datasets/tree/faster-text-parsers), and would love to contribute them to the upstream repository.

---

All the tests pass, but they leave a couple of important questions open. The default Python `open(..., newline=None)` uses universal newlines, where `\n`, `\r`, and `\r\n` are all converted to `\n` on the fly. I am not sure if its a good idea for a general purpose dataset preparation pipeline?

I can simulate the same behavior (which I don't yet do) for `"line"` splitter. Adjusting it for `"paragraph"`-splitter would be harder. Should we stick exactly to the old Pythonic behavior or stay closer to how C and other programming languages do that?
GitHub issue: Persist IterableDataset epoch in workers
State: closed

Use shared memory for the IterableDataset epoch.

This way calling `ds.set_epoch()` in the main process will update the epoch in the DataLoader workers as well.
This is useful especially because the epoch is used to compute the `effective_seed` used for shuffling.

I used torch's shared memory in case users want to send dataset copies without shared memory using pickle. I also find it easier to use than using `multiprocessing.shared_memory` than requires unlinking only in the main process, or `mp.Value` that is not picklable.

close https://github.com/huggingface/datasets/issues/6673

cc @rwightman
GitHub issue: Fix data_files when passing data_dir
State: closed

This code should not return empty data files

```python
from datasets import load_dataset_builder

revision = "3d406e70bc21c3ca92a9a229b4c6fc3ed88279fd"
b = load_dataset_builder("bigcode/the-stack-v2-dedup", data_dir="data/Dockerfile", revision=revision)
print(b.config.data_files)
```

Previously it would return no data files because it would apply the YAML `data_files: data/**/train-*` pattern to this directory

cc @anton-l
GitHub issue: Improve default patterns resolution
State: closed

Separate the default patterns that match directories from the ones matching files and ensure directories are checked first (reverts the change from https://github.com/huggingface/datasets/pull/6244, which merged these patterns). Also, ensure that the glob patterns do not overlap to avoid duplicates in the result. 

Additionally, replace `get_fs_token_paths` with `url_to_fs` to avoid [unnecessary glob calls](https://github.com/fsspec/filesystem_spec/blob/14dce8ca78f7aa509a20edb263bff83a7760c24d/fsspec/core.py#L655-L656).

fix https://github.com/huggingface/datasets/issues/6259
fix https://github.com/huggingface/datasets/issues/6272
GitHub issue: Unable to load dataset that was saved with `save_to_disk`
State: closed

### Describe the bug

I get the following error message: You are trying to load a dataset that was saved using `save_to_disk`. Please use `load_from_disk` instead.



### Steps to reproduce the bug

1. Save a dataset with `save_to_disk`
2. Try to load it with `load_datasets`

### Expected behavior

I am able to load the dataset again with `load_datasets` which most packages uses over `load_from_disk`. I want to have a workaround that allows me to create the same indexing that `push_to_hub` creates for you before using `save_to_disk` - how can that be achieved?

### Environment info

datasets 2.17.1, python 3.10
GitHub issue: Push samples to dataset on hub without having the dataset locally
State: closed

### Feature request

Say I have the following code:

```
from datasets import Dataset
import pandas as pd
new_data = {
    "column_1": ["value1", "value2"],
    "column_2": ["value3", "value4"],
}
df_new = pd.DataFrame(new_data)
dataset_new = Dataset.from_pandas(df_new)
# add these samples to a remote dataset
```
It would be great to have a way to push dataset_new to a remote dataset that respects the same schema. This way one would not have to do the following:
```
from datasets import load_dataset
dataset = load_dataset('username/dataset_name', use_auth_token='your_hf_token_here')
updated_dataset = dataset['train'].concatenate(dataset_new)
updated_dataset.push_to_hub('username/dataset_name', use_auth_token='your_hf_token_here')
```





### Motivation

No need to download the dataset.

### Your contribution

Maybe this feature already exists, didnt see it though. I do not have the expertise to do this.
GitHub issue: Base parquet batch_size on parquet row group size
State: closed

This allows to stream datasets like [Major-TOM/Core-S2L2A](https://huggingface.co/datasets/Major-TOM/Core-S2L2A) which have row groups with few rows (one row is ~10MB). Previously the cold start would take a lot of time and OOM because it would download many row groups before yielding the first example.

I tried on OpenOrca and imagenet-hard and it does't affect overall throughput.

Even if the overall throughput doesn't change for datasets like imagenet-hard with big rows, note that it does create shorter and more frequent pauses to download the next row group. Though I find it fine because previously the pauses were less frequent but very long (downloading multiple row groups at a time)
GitHub issue: remove_columns is not in-place but the doc shows it is in-place
State: closed

### Describe the bug

The doc of `datasets` v2.17.0/v2.17.1 shows that `remove_columns` is in-place. [link](https://huggingface.co/docs/datasets/v2.17.1/en/package_reference/main_classes#datasets.DatasetDict.remove_columns)

In the text classification example of transformers v4.38.1, the columns are not removed.
https://github.com/huggingface/transformers/blob/a0857740c0e6127485c11476650314df3accc2b6/examples/pytorch/text-classification/run_classification.py#L421

### Steps to reproduce the bug

https://github.com/huggingface/transformers/blob/a0857740c0e6127485c11476650314df3accc2b6/examples/pytorch/text-classification/run_classification.py#L421

### Expected behavior

Actually remove the columns.

### Environment info

1. datasets v2.17.0
2. transformers v4.38.1
GitHub issue: `Dataset` unexpected changed dict data and may cause error
State: open

### Describe the bug

Will unexpected get keys with `None` value in the parsed json dict.

### Steps to reproduce the bug

```jsonl test.jsonl
{"id": 0, "indexs": {"-1": [0, 10]}}
{"id": 1, "indexs": {"-1": [0, 10]}}
```

```python
dataset = Dataset.from_json('.test.jsonl')
print(dataset[0])
```

Result:
```
{'id': 0, 'indexs': {'-1': [...], '-2': None, '-3': None, '-4': None, '-5': None, '-6': None, '-7': None, '-8': None, '-9': None, ...}}
```

Those keys with `None` value will unexpected appear in the dict.


### Expected behavior

Result should be
```
{'id': 0, 'indexs': {'-1': [0, 10]}}
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-6.5.0-14-generic-x86_64-with-glibc2.35
- Python version: 3.11.6
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.10.0
GitHub issue: __add__ for Dataset, IterableDataset
State: open

It's too cumbersome to write this command every time we perform a dataset merging operation. ```pythonfrom datasets import concatenate_datasets``` We have added a simple `__add__` magic method to each class using `concatenate_datasets.`

```python
from datasets import load_dataset

bookcorpus = load_dataset("bookcorpus", split="train")
wiki = load_dataset("wikimedia/wikipedia", "20231101.ab", split="train")
wiki = wiki.remove_columns([col for col in wiki.column_names if col != "text"])  # only keep the 'text' column

bookcorpus + wiki

#Dataset({
#   features: ['text'],
#   num_rows: 74004228
#})
#Dataset({
#    features: ['text'],
#    num_rows: 6152
#})
#Dataset({
#   features: ['text'],
#    num_rows: 74010380
#})
```
GitHub issue: .load_dataset() method defaults to zstandard
State: closed

### Describe the bug

Regardless of what method I use, datasets defaults to zstandard for unpacking my datasets.

This is poor behavior, because not only is zstandard not a dependency in the huggingface package (and therefore, your dataset loading will be interrupted while it asks you to install the package), but it happens on datasets that are uploaded in json format too, meaning the dataset loader will attempt to convert the data to a zstandard compatible format, and THEN try to unpackage it.

My 4tb drive runs out of room when using zstandard on slimpajama. It loads fine on 1.5tb when using json, however I lack the understanding of the "magic numbers" system used to select the unpackaging algorithm, so I can't push a change myself.

Commenting out this line, in "/datasets/utils/extract.py" fixes the issue, and causes SlimPajama to properly extract using rational amounts of storage, however it completely disables zstandard, which is probably undesirable behavior. Someone with an understanding of the "magic numbers" system should probably take a pass over this issue.

```
class Extractor:
    #  Put zip file to the last, b/c it is possible wrongly detected as zip (I guess it means: as tar or gzip)
    extractors: Dict[str, Type[BaseExtractor]] = {
        "tar": TarExtractor,
        "gzip": GzipExtractor,
        "zip": ZipExtractor,
        "xz": XzExtractor,
        #"zstd": ZstdExtractor,   # This line needs to go, in order for datasets to work w/o non-dependent packages
        "rar": RarExtractor,
        "bz2": Bzip2Extractor,
        "7z": SevenZipExtractor,  # <Added version="2.4.0"/>
        "lz4": Lz4Extractor,  # <Added version="2.4.0"/>
    }
```

### Steps to reproduce the bug

'''
from datasaets import load_dataset
load_dataset(path="/cerebras/SlimPajama-627B")
'''

This alone should trigger the error on any system that does not have zstandard pip installed.

### Expected behavior

This repository (which is encoded in json format, not zstandard) should check whether zstandard is installed before defaulting to it. Additionally, using zstandard should not use more than 3x the required space that other extraction mechanisms use.

### Environment info

- `datasets` version: 2.17.1
- Platform: Linux-6.5.0-18-generic-x86_64-with-glibc2.35
- Python version: 3.12.0
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
GitHub issue: Tensor type (e.g. from `return_tensors`) ignored in map
State: open

### Describe the bug

I don't know if it is a bug or an expected behavior, but the tensor type seems to be ignored after applying map. For example, mapping over to tokenize text with a transformers' tokenizer always returns lists and it ignore the `return_tensors` argument.

If this is an expected behaviour (e.g., for caching/Arrow compatibility/etc.) it should be clearly documented. For example, current documentation (see [here](https://huggingface.co/docs/datasets/v2.17.1/en/nlp_process#map)) clearly state to "set  `return_tensors="np"` when you tokenize your text" to have Numpy arrays.

### Steps to reproduce the bug

```py
# %%%

import datasets
import numpy as np
import tensorflow as tf
import torch
from transformers import AutoTokenizer


# %%
ds = datasets.load_dataset("cnn_dailymail", "1.0.0", split="train[:1%]")
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

#%%
for return_tensors in [None, "np", "pt", "tf", "jax"]:
  print(f"********** no map, return_tensors={return_tensors} **********")
  _ds = tokenizer(ds["article"], return_tensors=return_tensors, truncation=True, padding=True)
  print('Type <input_ids>:', type(_ds["input_ids"]))

# %%
for return_tensors in [None, "np", "pt", "tf", "jax"]:
  print(f"********** map, return_tensors={return_tensors} **********")
  _ds = ds.map(
    lambda examples: tokenizer(examples["article"], return_tensors=return_tensors, truncation=True, padding=True),
    batched=True,
    remove_columns=["article"],
  )

  print('Type <input_ids>:', type(_ds[0]["input_ids"]))
```


### Expected behavior

The output from the script above. I would expect the second half to be the same.

```
********** no map, return_tensors=None **********
Type <input_ids>: <class 'list'>
********** no map, return_tensors=np **********
Type <input_ids>: <class 'numpy.ndarray'>
********** no map, return_tensors=pt **********
Type <input_ids>: <class 'torch.Tensor'>
********** no map, return_tensors=tf **********
Type <input_ids>: <class 'tensorflow.python.framework.ops.EagerTensor'>
********** no map, return_tensors=jax **********
Type <input_ids>: <class 'jaxlib.xla_extension.ArrayImpl'>


********** map, return_tensors=None **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=np **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=pt **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=tf **********
Type <input_ids>: <class 'list'>
********** map, return_tensors=jax **********
Type <input_ids>: <class 'list'>
```

### Environment info

- `datasets` version: 2.17.1
- Platform: Redacted (linux)
- Python version: 3.10.12
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0
GitHub issue: fsspec: support fsspec>=2023.12.0 glob changes
State: closed

- adds support for the `fs.glob` changes introduced in `fsspec==2023.12.0` and unpins the current upper bound

Should close #6644
Should close #6645

The `test_data_files` glob/pattern tests pass for me in:
- `fsspec==2023.10.0` (the pinned max version in datasets `main`)
- `fsspec==2023.12.0` (#6644)
- `fsspec==2024.2.0` (#6645)
GitHub issue: Question: Is there any way for uploading a large image dataset?
State: open

I am uploading an image dataset like this:
```
dataset = load_dataset(
    "json",
    data_files={"train": "data/custom_dataset/train.json", "validation": "data/custom_dataset/val.json"},
)
dataset = dataset.cast_column("images", Sequence(Image()))
dataset.push_to_hub("StanfordAIMI/custom_dataset", max_shard_size="1GB")
```
where it takes a long time in the `Map` process. Do you think I can use multi-processing to map all the image data to the memory first? For the `Map()` function, I can set `num_proc`. But for `push_to_hub` and `cast_column`, I can not find it.

Thanks in advance!

Best,
GitHub issue: Node.js 16 GitHub Actions are deprecated
State: closed

`Node.js` 16 GitHub Actions are deprecated. See: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/

We should update them to Node 20.

See warnings in our CI, e.g.: https://github.com/huggingface/datasets/actions/runs/7957295009?pr=6678
> Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/checkout@v3, actions/setup-python@v4. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.
GitHub issue: Can't Read List of JSON Files Properly
State: open

### Describe the bug

Trying to read a bunch of JSON files into Dataset class but default approach doesn't work. I don't get why it works when I read it one by one but not when I pass as a list :man_shrugging: 

The code fails with 

```
ArrowInvalid: JSON parse error: Invalid value. in row 0  
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte 
DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

This doesn't work

```
from datasets import Dataset

# dir contains 100 json files.
Dataset.from_json("/PUT SOME PATH HERE/*")
```

This works:
```
from datasets import concatenate_datasets

ls_ds = []

for file in list_of_json_files:
    ls_ds.append(Dataset.from_json(file))

ds = concatenate_datasets(ls_ds)

```





### Expected behavior

I expect this to read json files properly as error is not clear

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-6.5.0-15-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
GitHub issue: Allow image model (color conversion) to be specified as part of datasets Image() decode
State: closed

### Feature request

Typical torchvision / torch Datasets in image applications apply color conversion in the Dataset portion of the code as part of image decode, separately from the image transform stack. This is true for PIL.Image where convert is usually called in dataset, for native torchvision https://pytorch.org/vision/main/generated/torchvision.io.decode_jpeg.html, and similarly in tensorflow.data pipelines decode_jpeg or https://www.tensorflow.org/api_docs/python/tf/io/decode_and_crop_jpeg have a channels arg that allows controlling the image mode in the decode step.

datasets currently requires this pattern (from [examples](https://huggingface.co/docs/datasets/main/en/image_process)):
```
from torchvision.transforms import Compose, ColorJitter, ToTensor

jitter = Compose(
    [
         ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.7),
         ToTensor(),
    ]
)

def transforms(examples):
    examples["pixel_values"] = [jitter(image.convert("RGB")) for image in examples["image"]]
    return examples
```




### Motivation

It would be nice to be able to handle `image.convert("RGB")` (or other modes) in the decode step, before applying torchvision transforms, this would reduce differences in code when handling pipelines that can handle torchvision, webdatset, or hf datasets with fewer code differences and without needing to handle image mode argument passing in two different stages of the pipelines...

### Your contribution

Can do a PR with guidance on how mode should be passed / set on the dataset.
GitHub issue:  Depprcated Overview.ipynb Link to new Quickstart Notebook invalid
State: closed

### Describe the bug

For the dreprecated notebook found  [here](https://github.com/huggingface/datasets/blob/main/notebooks/Overview.ipynb). The link to the new notebook is broken. 

### Steps to reproduce the bug

Click the [Quickstart notebook](https://github.com/huggingface/notebooks/blob/main/datasets_doc/quickstart.ipynb) link in the notebook.

### Expected behavior

I believe is it suposed to link [here](https://github.com/huggingface/notebooks/blob/main/datasets_doc/en/quickstart.ipynb). That is mentioned in the readme.

### Environment info

Colab
GitHub issue: IterableDataset `set_epoch` is ignored when DataLoader `persistent_workers=True`
State: closed

### Describe the bug

When persistent workers are enabled, the epoch that's set via the IterableDataset instance held by the training process is ignored by the workers as they are disconnected across processes.

PyTorch samplers for non-iterable datasets have a mechanism to sync this, datasets.IterableDataset does not.

In my own use of IterableDatasets I usually track the epoch count which crosses process boundaries in a multiprocessing.Value 

### Steps to reproduce the bug

Use a streaming dataset (Iterable) w/ the recommended pattern below and `persistent_workers=True` in the torch DataLoader. 

```
for epoch in range(epochs):
    shuffled_dataset.set_epoch(epoch)
    for example in shuffled_dataset:
        ...
```

### Expected behavior

When the canonical bit of code above is used with `num_workers > 0` and `persistent_workers=True`, the epoch set via `set_epoch()` is propagated to the IterableDataset instances in the worker processes


### Environment info

N/A
GitHub issue: CSV builder raises deprecation warning on verbose parameter
State: closed

CSV builder raises a deprecation warning on `verbose` parameter:
```
FutureWarning: The 'verbose' keyword in pd.read_csv is deprecated and will be removed in a future version.
```

See:
- https://github.com/pandas-dev/pandas/pull/56556
- https://github.com/pandas-dev/pandas/pull/57450