sjwsjw/github-issues huggingface dataset:
GitHub issue: irc_disentangle - Issue with splitting data
State: closed

### Describe the bug

I am trying to access your database through python using "datasets.load_dataset("irc_disentangle")" and I am getting this error message:

ValueError: Instruction "train" corresponds to no data!

### Steps to reproduce the bug

import datasets
ds = datasets.load_dataset('irc_disentangle')
ds

### Expected behavior

The data is supposed to load into ds and be accessable as such:
ds['train'][1050], ds['train'][1055]

### Environment info

I tired Python 3.12 and 3.10
GitHub issue: Extraction protocol for arrow files is not defined
State: closed

### Describe the bug

Passing files with `.arrow` extension into data_files argument, at least when `streaming=True` is very slow.

### Steps to reproduce the bug

 Basically it goes through the `_get_extraction_protocol` method located [here](https://github.com/huggingface/datasets/blob/main/src/datasets/utils/file_utils.py#L820)
The method then looks at some base known extensions where `arrow` is not defined so it proceeds to determine the compression with the magic number method which is slow when dealing with a lot of files which are stored in s3 and by looking at this predefined list, I don't see `arrow` in there either so in the end it return None:

```
MAGIC_NUMBER_TO_COMPRESSION_PROTOCOL = {
    bytes.fromhex("504B0304"): "zip",
    bytes.fromhex("504B0506"): "zip",  # empty archive
    bytes.fromhex("504B0708"): "zip",  # spanned archive
    bytes.fromhex("425A68"): "bz2",
    bytes.fromhex("1F8B"): "gzip",
    bytes.fromhex("FD377A585A00"): "xz",
    bytes.fromhex("04224D18"): "lz4",
    bytes.fromhex("28B52FFD"): "zstd",
}
```

### Expected behavior

My expectation is that `arrow` would be in the known lists so it would return None without going through the magic number method.

### Environment info

datasets 2.19.0
GitHub issue: Add the option of saving in parquet instead of arrow 
State: open

### Feature request

In dataset.save_to_disk('/path/to/save/dataset'),

add the option to save in parquet format

dataset.save_to_disk('/path/to/save/dataset', format="parquet"),

because arrow is not used for Production Big data.... (only parquet)



### Motivation

because arrow is not used for Production Big data.... (only parquet)

### Your contribution

I can do the testing !
GitHub issue: Make CLI convert_to_parquet not raise error if no rights to create script branch
State: closed

Make CLI convert_to_parquet not raise error if no rights to create "script" branch.

Not that before this PR, the error was not critical because it was raised at the end of the script, once all the rest of the steps were already performed.

Fix #6901.

Bug introduced in datasets-2.19.0 by:
- #6809
GitHub issue: HTTPError 403 raised by CLI convert_to_parquet when creating script branch on 3rd party repos
State: closed

CLI convert_to_parquet cannot create "script" branch on 3rd party repos.

It can only create it on repos where the user executing the script has write access.

Otherwise, a 403 Forbidden HTTPError is raised:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 1021, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://huggingface.co/api/datasets/ORG/DATASET/branch/script

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/datasets-cli", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/datasets/commands/datasets_cli.py", line 41, in main
    service.run()
  File "/usr/local/lib/python3.10/dist-packages/datasets/commands/convert_to_parquet.py", line 92, in run
    create_branch(dataset_id, branch="script", repo_type="dataset", token=token, exist_ok=True)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/hf_api.py", line 5503, in create_branch
    hf_raise_for_status(response)
  File "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_errors.py", line 367, in hf_raise_for_status
    raise HfHubHTTPError(message, response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError:  (Request ID: Root=1-6645ee0d-4db1ed8a1fbe04956be15897;139a6e23-df7d-4f62-b5ba-adb6d8e6e696)

403 Forbidden: Forbidden: cannot write to script.
Cannot access content at: https://huggingface.co/api/datasets/ORG/DATASET/branch/script.
If you are trying to create or update content,make sure you have a token with the `write` role.
```
GitHub issue: [WebDataset] KeyError with user-defined `Features` when a field is missing in an example
State: closed

reported at https://huggingface.co/datasets/ProGamerGov/synthetic-dataset-1m-dalle3-high-quality-captions/discussions/1

```
File "/src/services/worker/.venv/lib/python3.9/site-packages/datasets/packaged_modules/webdataset/webdataset.py", line 109, in _generate_examples
    example[field_name] = {"path": example["__key__"] + "." + field_name, "bytes": example[field_name]}
```
GitHub issue: List of dictionary features get standardized
State: open

### Describe the bug

Hi, i’m trying to create a HF dataset from a list using Dataset.from_list.

Each sample in the list is a dict with the same keys (which will be my features). The values for each feature are a list of dictionaries, and each such dictionary has a different set of keys. However, the datasets library standardizes all dictionaries under a feature and adds all possible keys (with None value) from all the dictionaries under that feature.

How can I keep the same set of keys as in the original list for each dictionary under a feature?

### Steps to reproduce the bug

```
from datasets import Dataset

# Define a function to generate a sample with "tools" feature
def generate_sample():
    # Generate random sample data
    sample_data = {
        "text": "Sample text",
        "feature_1": []
    }
    
    # Add feature_1 with random keys for this sample
    feature_1 = [{"key1": "value1"}, {"key2": "value2"}]  # Example feature_1 with random keys
    sample_data["feature_1"].extend(feature_1)
    
    return sample_data

# Generate multiple samples
num_samples = 10
samples = [generate_sample() for _ in range(num_samples)]

# Create a Hugging Face Dataset
dataset = Dataset.from_list(samples)
dataset[0]
```

```{'text': 'Sample text', 'feature_1': [{'key1': 'value1', 'key2': None}, {'key1': None, 'key2': 'value2'}]}```

### Expected behavior

```{'text': 'Sample text', 'feature_1': [{'key1': 'value1'}, {'key2': 'value2'}]}```

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-5.15.0-1040-nvidia-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.23.0
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0
GitHub issue: Fix YAML error in README files appearing on GitHub
State: closed

Fix YAML error in README files appearing on GitHub.

See error message:
![Screenshot from 2024-05-14 06-58-02](https://github.com/huggingface/datasets/assets/8515462/7984cc4e-96ee-4e83-99a4-4c0c5791fa05)


Fix #6897.
GitHub issue: datasets template guide :: issue in documentation YAML
State: closed

### Describe the bug

There is a YAML error at the top of the page, and I don't think it's supposed to be there 

### Steps to reproduce the bug

1. Browse to [this tutorial document](https://github.com/huggingface/datasets/blob/main/templates/README_guide.md)
2. Observe a big red error at the top
3. The rest of the document remains functional

### Expected behavior

I think the YAML block should be displayed or ignored.

### Environment info

N/A
GitHub issue: Add support for categorical/dictionary types
State: closed

Arrow has a very useful dictionary/categorical type (https://arrow.apache.org/docs/python/generated/pyarrow.dictionary.html). This data type has significant speed, memory and disk benefits over pa.string() when there are only a few unique text strings in a column.

Unfortunately, huggingface datasets currently does not support this type. So huggingface datasets cannot natively read many parquet files that use this datatype .This PR adds support for Huggingface Datasets to read categorical/dictionary data.

Note: This PR functions by simply converting those dictionary/categorical types to strings. This means that huggingface datasets cannot take advantage of the compute benefits of categoricals, but it significantly simplifies logic. At this time, I do not think it makes sense to optimize categorical support within huggingface datasets and that we should only try to optimize later, if necessary.

Closes #5706
GitHub issue: Unable to load JSON saved using `to_json`
State: closed

### Describe the bug

Datasets stored in the JSON format cannot be loaded using `json.load()`

### Steps to reproduce the bug

```
import json
from datasets import load_dataset

dataset = load_dataset("squad")
train_dataset, test_dataset = dataset["train"], dataset["validation"]
test_dataset.to_json("full_dataset.json")

# This works
loaded_test = load_dataset("json", data_files="full_dataset.json")

# This fails
loaded_test = json.load(open("full_dataset.json", "r"))
```

### Expected behavior

The JSON should be correctly formatted when writing so that it can be loaded using `json.load()`.

### Environment info

Colab: https://colab.research.google.com/drive/1st1iStFUVgu9ZPvnzSzL4vDeYWDwYpUm?usp=sharing
GitHub issue: add `with_transform` and/or `set_transform` to IterableDataset
State: open

### Feature request

when working with a really large dataset it would save us a lot of time (and compute resources) to use either with_transform or the set_transform from the Dataset class instead of waiting for the entire dataset to map

### Motivation

don't want to wait for a really long dataset to map, this would give IterableDataset an extra advantage over the Dataset class.
reducing time and resources

### Your contribution

I am a little busy with my job search lately, but would post about this feature in my social media.
Apologies again (dad going to kick me out soon), if I ever have some free time I will contribute to making this a reality, but that's going to be hard
      / (┬┬﹏┬┬)\
GitHub issue: fix bug #6877
State: closed

fix bug #6877 due to maybe f becomes invaild after yield process
the results are below:


Resolving data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:01<00:00, 420.41it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 26148.48it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 409731.44it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 289720.84it/s]
Resolving data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 26663.42it/s]
Resolving data files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 434056.21it/s]
Downloading data: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:00<00:00, 13222.33files/s]
Downloading data: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [00:04<00:00, 180.67files/s]
Downloading data: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 828/828 [01:35<00:00,  8.70files/s]
Generating train split: 1571592 examples [00:08, 176736.09 examples/s]
Generating test split: 85533 examples [00:01, 48224.56 examples/s]
Generating validation split: 86246 examples [00:01, 50164.16 examples/s]

Fix https://github.com/huggingface/datasets/issues/6877.

CC: @natolambert
GitHub issue: FAISS load to None
State: open

### Describe the bug

I've use FAISS with Datasets and save to FAISS.

Then load to save FAISS then no error, then ds to None

```python
ds.load_faiss_index('embeddings', 'my_index.faiss')
```

### Steps to reproduce the bug

# 1.
```python
ds_with_embeddings = ds.map(lambda example: {'embeddings': model(transforms(example['image']).unsqueeze(0)).squeeze()}, batch_size=64)

ds_with_embeddings.add_faiss_index(column='embeddings')

ds_with_embeddings.save_faiss_index('embeddings', 'index.faiss')
```

# 2.
```python
ds.load_faiss_index('embeddings', 'my_index.faiss')
```

### Expected behavior

Add column in Datasets.

### Environment info

Google Colab, SageMaker Notebook
GitHub issue: load_dataset with data_dir and cache_dir set fail with not supported
State: open

### Describe the bug

with python 3.11 I execute:
```py
from transformers import Wav2Vec2Processor, Data2VecAudioModel
import torch
from torch import nn
from datasets import load_dataset, concatenate_datasets

# load demo audio and set processor
dataset_clean = load_dataset("librispeech_asr", "clean", split="validation", data_dir="data", cache_dir="cache")
```
This fails in the last line with 
```log
Found cached dataset librispeech_asr (file:///Users/as/Documents/Project/git/audio2vec/cache/librispeech_asr/clean-data_dir=data/2.1.0/cff5df6e7955c80a67f80e27e7e655de71c689e2d2364bece785b972acb37fe7)
Traceback (most recent call last):
  File "/Users/as/Documents/Project/git/audio2vec/src/music2vec-v1.py", line 7, in <module>
    dataset_clean = load_dataset("librispeech_asr", "clean", split="validation", data_dir="data", cache_dir="cache")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/as/anaconda3/lib/python3.11/site-packages/datasets/load.py", line 1810, in load_dataset
    ds = builder_instance.as_dataset(split=split, verification_mode=verification_mode, in_memory=keep_in_memory)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/as/anaconda3/lib/python3.11/site-packages/datasets/builder.py", line 1113, in as_dataset
    raise NotImplementedError(f"Loading a dataset cached in a {type(self._fs).__name__} is not supported.")
NotImplementedError: Loading a dataset cached in a LocalFileSystem is not supported.
```

### Steps to reproduce the bug

I setup an venv with requirements.txt
```txt
transformers==4.40.2
torch==2.2.2
datasets==2.16.0
fsspec==2023.9.2
```

pip freeze is:
```
aiohttp==3.9.5
aiosignal==1.3.1
attrs==23.2.0
certifi==2024.2.2
charset-normalizer==3.3.2
datasets==2.16.0
dill==0.3.7
filelock==3.14.0
frozenlist==1.4.1
fsspec==2023.9.2
huggingface-hub==0.23.0
idna==3.7
Jinja2==3.1.4
MarkupSafe==2.1.5
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.15
networkx==3.3
numpy==1.26.4
packaging==24.0
pandas==2.2.2
pyarrow==16.0.0
pyarrow-hotfix==0.6
python-dateutil==2.9.0.post0
pytz==2024.1
PyYAML==6.0.1
regex==2024.4.28
requests==2.31.0
safetensors==0.4.3
six==1.16.0
sympy==1.12
tokenizers==0.19.1
torch==2.2.2
tqdm==4.66.4
transformers==4.40.2
typing_extensions==4.11.0
tzdata==2024.1
urllib3==2.2.1
xxhash==3.4.1
yarl==1.9.4

```

I execute this on a M1 Mac.


### Expected behavior

I don't understand the error message. Why is "local" caching not supported. Would it possible to give some additional hint with the error message how to solve this issue?


### Environment info

source ....
python -u example.py
GitHub issue: CI is broken after jax-0.4.27 release: AttributeError: 'jaxlib.xla_extension.DeviceList' object has no attribute 'device'
State: closed

After jax-0.4.27 release (https://github.com/google/jax/releases/tag/jax-v0.4.27), our CI is broken with the error:
```Python traceback
AttributeError: 'jaxlib.xla_extension.DeviceList' object has no attribute 'device'. Did you mean: 'devices'?
```

See: https://github.com/huggingface/datasets/actions/runs/8997488610/job/24715736153
```Python traceback
___________________ FormatterTest.test_jax_formatter_device ____________________
[gw1] linux -- Python 3.10.14 /opt/hostedtoolcache/Python/3.10.14/x64/bin/python

self = <tests.test_formatting.FormatterTest testMethod=test_jax_formatter_device>

    @require_jax
    def test_jax_formatter_device(self):
        import jax
    
        from datasets.formatting import JaxFormatter
    
        pa_table = self._create_dummy_table()
        device = jax.devices()[0]
        formatter = JaxFormatter(device=str(device))
        row = formatter.format_row(pa_table)
>       assert row["a"].device() == device
E       AttributeError: 'jaxlib.xla_extension.DeviceList' object has no attribute 'device'. Did you mean: 'devices'?

tests/test_formatting.py:630: AttributeError
```
GitHub issue: Require Pillow >= 9.4.0 to avoid AttributeError when loading image dataset
State: closed

Require Pillow >= 9.4.0 to avoid AttributeError when loading image dataset.

The `PIL.Image.ExifTags` that we use in our code was implemented in Pillow-9.4.0: https://github.com/python-pillow/Pillow/commit/24a5405a9f7ea22f28f9c98b3e407292ea5ee1d3

The bug #6881 was introduced in datasets-2.19.0 by this PR:
- #6739

Fix #6881.
GitHub issue: AttributeError: module 'PIL.Image' has no attribute 'ExifTags'
State: closed

When trying to load an image dataset in an old Python environment (with Pillow-8.4.0), an error is raised:
```Python traceback
AttributeError: module 'PIL.Image' has no attribute 'ExifTags'
```
The error traceback:
```Python traceback
~/huggingface/datasets/src/datasets/iterable_dataset.py in __iter__(self)
   1391                 # `IterableDataset` automatically fills missing columns with None.
   1392                 # This is done with `_apply_feature_types_on_example`.
-> 1393                 example = _apply_feature_types_on_example(
   1394                     example, self.features, token_per_repo_id=self._token_per_repo_id
   1395                 )

~/huggingface/datasets/src/datasets/iterable_dataset.py in _apply_feature_types_on_example(example, features, token_per_repo_id)
   1080     encoded_example = features.encode_example(example)
   1081     # Decode example for Audio feature, e.g.
-> 1082     decoded_example = features.decode_example(encoded_example, token_per_repo_id=token_per_repo_id)
   1083     return decoded_example
   1084 

~/huggingface/datasets/src/datasets/features/features.py in decode_example(self, example, token_per_repo_id)
   1974 
-> 1975         return {
   1976             column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id)
   1977             if self._column_requires_decoding[column_name]

~/huggingface/datasets/src/datasets/features/features.py in <dictcomp>(.0)
   1974 
   1975         return {
-> 1976             column_name: decode_nested_example(feature, value, token_per_repo_id=token_per_repo_id)
   1977             if self._column_requires_decoding[column_name]
   1978             else value

~/huggingface/datasets/src/datasets/features/features.py in decode_nested_example(schema, obj, token_per_repo_id)
   1339         # we pass the token to read and decode files from private repositories in streaming mode
   1340         if obj is not None and schema.decode:
-> 1341             return schema.decode_example(obj, token_per_repo_id=token_per_repo_id)
   1342     return obj
   1343 

~/huggingface/datasets/src/datasets/features/image.py in decode_example(self, value, token_per_repo_id)
    187             image = PIL.Image.open(BytesIO(bytes_))
    188         image.load()  # to avoid "Too many open files" errors
--> 189         if image.getexif().get(PIL.Image.ExifTags.Base.Orientation) is not None:
    190             image = PIL.ImageOps.exif_transpose(image)
    191         if self.mode and self.mode != image.mode:

~/huggingface/datasets/venv/lib/python3.9/site-packages/PIL/Image.py in __getattr__(name)
     75                 )
     76                 return categories[name]
---> 77         raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
     78 
     79 

AttributeError: module 'PIL.Image' has no attribute 'ExifTags'
```

### Environment info

Since datasets 2.19.0
GitHub issue: Batched mapping does not raise an error if values for an existing column are empty
State: open

### Describe the bug

Using `Dataset.map(fn, batched=True)` allows resizing the dataset by returning a dict of lists, all of which must be the same size. If they are not the same size, an error like `pyarrow.lib.ArrowInvalid: Column 1 named x expected length 1 but got length 0` is raised.

This is not the case if the function returns an empty list for an existing column in the dataset. In that case, the dataset is silently resized to 0 rows.

### Steps to reproduce the bug

MWE:

```
import datasets
data = datasets.Dataset.from_dict({"test": [1]})

def mapping_fn(examples):
    return {"test": [], "y": [1]}

data = data.map(mapping_fn, batched=True)
print(len(data))
```

Note  that when returning `"x": []`, the error is raised correctly, also when returning `"test": [1,2]`.

### Expected behavior

Expected an exception: `pyarrow.lib.ArrowInvalid: Column 1 named test expected length 1 but got length 0` or `pyarrow.lib.ArrowInvalid: Column 2 named y expected length 0 but got length 1`.

Any exception would be acceptable.

### Environment info

- `datasets` version: 2.19.1
- Platform: Linux-5.4.0-153-generic-x86_64-with-glibc2.31
- Python version: 3.11.8
- `huggingface_hub` version: 0.22.2
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0
GitHub issue: Unpin hfh
State: closed

Needed to use those in dataset-viewer:

- dev version of hfh https://github.com/huggingface/dataset-viewer/pull/2781: don't span the hub with /paths-info requests
- dev version of datasets at https://github.com/huggingface/datasets/pull/6875: don't write too big logs in the viewer

close https://github.com/huggingface/datasets/issues/6863
GitHub issue: datasets.BuilderConfig does not work.
State: closed

### Describe the bug

I custom a BuilderConfig and GeneratorBasedBuilder.
Here is the code for BuilderConfig
```
class UIEConfig(datasets.BuilderConfig):

    def __init__(
            self,
            *args,
            data_dir=None,
            instruction_file=None,
            instruction_strategy=None,
            task_config_dir=None,
            num_examples=None,
            max_num_instances_per_task=None,
            max_num_instances_per_eval_task=None,
            over_sampling=None,
            **kwargs
    ):
        super().__init__(*args, **kwargs)
        self.data_dir = data_dir
        self.num_examples = num_examples
        self.over_sampling = over_sampling
        self.instructions = self._parse_instruction(instruction_file)
        self.task_configs = self._parse_task_config(task_config_dir)
        self.instruction_strategy = instruction_strategy
        self.max_num_instances_per_task = max_num_instances_per_task
        self.max_num_instances_per_eval_task = max_num_instances_per_eval_task
```
Besides, here is the code for GeneratorBasedBuilder.
```
class UIEInstructions(datasets.GeneratorBasedBuilder):
    VERSION = datasets.Version("2.0.0")
    BUILDER_CONFIG_CLASS = UIEConfig
    BUILDER_CONFIGS = [
        UIEConfig(name="default", description="Default config for NaturalInstructions")
    ]
    DEFAULT_CONFIG_NAME = "default"
```

Here is the load_dataset
```
raw_datasets = load_dataset(
        os.path.join(CURRENT_DIR, "uie_dataset.py"),
        data_dir=data_args.data_dir,
        task_config_dir=data_args.task_config_dir,
        instruction_file=data_args.instruction_file,
        instruction_strategy=data_args.instruction_strategy,
        cache_dir=data_cache_dir,  # for debug, change dataset size, otherwise open it
        max_num_instances_per_task=data_args.max_num_instances_per_task,
        max_num_instances_per_eval_task=data_args.max_num_instances_per_eval_task,
        num_examples=data_args.num_examples,
        over_sampling=data_args.over_sampling
    )
```
Finally, I met the error.
```
BuilderConfig UIEConfig(name='default', version=0.0.0, data_dir=None, data_files=None, description='Default config for NaturalInstructions') doesn't have a 'task_config_dir' key.
```

I debugged the code, but I find the parameters added by me may not work.

### Steps to reproduce the bug

https://github.com/BeyonderXX/InstructUIE/blob/master/src/uie_dataset.py

### Expected behavior

```
BuilderConfig UIEConfig(name='default', version=0.0.0, data_dir=None, data_files=None, description='Default config for NaturalInstructions') doesn't have a 'task_config_dir' key.
```

### Environment info

torch                     2.3.0+cu118
transformers              4.40.1
python 3.8
GitHub issue: Improve performance of JSON loader
State: closed

As reported by @natolambert, loading regular JSON files with `datasets` shows poor performance.

The cause is that we use the `json` Python standard library instead of other faster libraries. See my old comment: https://github.com/huggingface/datasets/pull/2638#pullrequestreview-706983714
> There are benchmarks that compare different JSON packages, with the Standard Library one among the worst performant: 
> - https://github.com/ultrajson/ultrajson#benchmarks
> - https://github.com/ijl/orjson#performance

I remember having a discussion about this and it was decided that it was better not to include an additional dependency on a 3rd-party library.

However:
- We already depend on `pandas` and `pandas` depends on `ujson`: so we have an indirect dependency on `ujson`
- Even if the above were not the case, we always could include `ujson` as an optional extra dependency, and check at runtime if it is installed to decide which library to use, either json or ujson
GitHub issue: Example on Semantic segmentation contains bug
State: open

### Describe the bug

https://huggingface.co/docs/datasets/en/semantic_segmentation shows wrong example with torchvision transforms.
Specifically, as one can see in screenshot below, the object boundaries have weird colors. 

<img width="689" alt="image" src="https://github.com/huggingface/datasets/assets/4803565/59aa0e2c-2e3e-415b-9d42-2314044c5aee">


Original example with `albumentations` is correct 
<img width="705" alt="image" src="https://github.com/huggingface/datasets/assets/4803565/27dbd725-cea5-4e48-ba59-7050c3ce17b3">


That is because `torch vision.transforms.Resize` interpolates with bilinear everything which is wrong when used for segmentation labels - you just cannot mix them. Overall, `torchvision.transforms` is designed for classification only and cannot be used to images and masks together, unless you write two separate branches of augmentations.


The correct way would be to use `v2` version of transforms and convert the segmentation labels to https://pytorch.org/vision/main/generated/torchvision.tv_tensors.Mask.html#torchvision.tv_tensors.Mask object




### Steps to reproduce the bug

Go to the website.
<img width="689" alt="image" src="https://github.com/huggingface/datasets/assets/4803565/ea1276d0-d69a-48cf-b9c2-cd61217815ef">

https://huggingface.co/docs/datasets/en/semantic_segmentation

### Expected behavior

Results, similar to `albumentation`. Or remove the torch vision part altogether. Or use `kornia` instead.

### Environment info

Irrelevant
GitHub issue: Dataset 'rewardsignal/reddit_writing_prompts' doesn't exist on the Hub
State: closed

### Describe the bug

The dataset `rewardsignal/reddit_writing_prompts` is missing in Huggingface Hub.

### Steps to reproduce the bug

```
from datasets import load_dataset

prompt_response_dataset = load_dataset("rewardsignal/reddit_writing_prompts", data_files="prompt_responses_full.csv", split='train[:80%]')
```

### Expected behavior

DatasetNotFoundError: Dataset 'rewardsignal/reddit_writing_prompts' doesn't exist on the Hub or cannot be accessed

### Environment info

Nothing to do with versions
GitHub issue: Fix load_dataset for data_files with protocols other than HF
State: closed

Fixes huggingface/datasets/issues/6598

I've added a new test case and a solution. Before applying the solution the test case was failing with the same error described in the linked issue.

MRE:
```
pip install "datasets[s3]"
python -c "from datasets import load_dataset; load_dataset('csv', data_files={'train': 's3://noaa-gsod-pds/2024/A5125600451.csv'})"
```
GitHub issue: CI fails after huggingface_hub-0.23.0 release: FutureWarning: "resume_download"
State: closed

CI fails after latest huggingface_hub-0.23.0 release: https://github.com/huggingface/huggingface_hub/releases/tag/v0.23.0

```
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_bertscore - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_frugalscore - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_metric_common.py::LocalMetricTest::test_load_metric_perplexity - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_fingerprint.py::TokenizersHashTest::test_hash_tokenizer_with_cache - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
FAILED tests/test_arrow_dataset.py::MiscellaneousDatasetTest::test_set_format_encode - FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
```
GitHub issue: Support folder-based datasets with large metadata.jsonl
State: open

I tried creating an `imagefolder` dataset with a 714MB `metadata.jsonl` but got the error below.  This pull request fixes the problem by increasing the block size like the message suggests.
```
>>> from datasets import load_dataset
>>> dataset = load_dataset("imagefolder", data_dir="data-for-upload")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/datasets/load.py", line 2609, in load_dataset
    builder_instance.download_and_prepare(
      ...
  File "/path/to/datasets/packaged_modules/folder_based_builder/folder_based_builder.py", line 245, in _read_metadata
    return paj.read_json(f)
  File "pyarrow/_json.pyx", line 308, in pyarrow._json.read_json
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```
GitHub issue: Segmentation fault
State: closed

### Describe the bug

Using various version for datasets, I'm no more longer able to load that dataset without a segmentation fault. 
Several others files are also concerned. 

### Steps to reproduce the bug

# Create a new venv 
python3 -m venv venv_test 
source venv_test/bin/activate

# Install the latest version 
pip install datasets

# Load that dataset
python3 -q -X faulthandler -c "from datasets import load_dataset;  load_dataset('EuropeanParliament/Eurovoc', '1998-09')"

### Expected behavior

Data must be loaded 

### Environment info

datasets==2.19.0
Python 3.11.7
Darwin  22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:51:50 PDT 2023; root:xnu-8796.121.2~5/RELEASE_X86_64 x86_64
GitHub issue: Fix line-endings in tests on Windows
State: closed

EDIT:
~~Fix test_delete_from_hub on Windows by passing explicit encoding.~~
Fix test_delete_from_hub and test_xgetsize_private by uploading the README file content directly (encoding the string), instead of writing a local file and uploading it.

Note that local files created on Windows will have "\r\n" line endings, instead of "\n".
These are no longer transformed to "\n" by the Hub.

Fix #6856.
GitHub issue: CI fails on Windows for test_delete_from_hub and test_xgetsize_private due to new-line character
State: closed

CI fails on Windows for test_delete_from_hub after the merge of:
- #6820

This is weird because the CI was green in the PR branch before merging to main.

```
FAILED tests/test_hub.py::test_delete_from_hub - AssertionError: assert [CommitOperat...\r\n---\r\n')] == [CommitOperat...in/*\n---\n')]
  
  At index 1 diff: CommitOperationAdd(path_in_repo='README.md', path_or_fileobj=b'---\r\nconfigs:\r\n- config_name: cats\r\n  data_files:\r\n  - split: train\r\n    path: cats/train/*\r\n---\r\n') != CommitOperationAdd(path_in_repo='README.md', path_or_fileobj=b'---\nconfigs:\n- config_name: cats\n  data_files:\n  - split: train\n    path: cats/train/*\n---\n')
  
  Full diff:
    [
        CommitOperationDelete(
            path_in_repo='dogs/train/0000.csv',
            is_folder=False,
        ),
        CommitOperationAdd(
            path_in_repo='README.md',
  -         path_or_fileobj=b'---\nconfigs:\n- config_name: cats\n  data_files:\n '
  ?                                                                       --------
  +         path_or_fileobj=b'---\r\nconfigs:\r\n- config_name: cats\r\n  data_f'
  ?                               ++          ++                     ++
  -                         b' - split: train\n    path: cats/train/*\n---\n',
  ?                                                                   ^^^^^^ -
  +                         b'iles:\r\n  - split: train\r\n    path: cats/train/*\r'
  ?                           ++++++++++                ++                        ^
  +                         b'\n---\r\n',
        ),
    ]
```
GitHub issue: Wrong example of usage when config name is missing for community script-datasets
State: closed

As reported by @Wauplin, when loading a community dataset with script, there is a bug in the example of usage of the error message if the dataset has multiple configs (and no default config) and the user does not pass any config. For example:
```python
>>> ds = load_dataset("google/fleurs")
ValueError: Config name is missing.
Please pick one among the available configs: ['af_za', 'am_et', 'ar_eg', 'as_in', 'ast_es', 'az_az', 'be_by', 'bg_bg', 'bn_in', 'bs_ba', 'ca_es', 'ceb_ph', 'ckb_iq', 'cmn_hans_cn', 'cs_cz', 'cy_gb', 'da_dk', 'de_de', 'el_gr', 'en_us', 'es_419', 'et_ee', 'fa_ir', 'ff_sn', 'fi_fi', 'fil_ph', 'fr_fr', 'ga_ie', 'gl_es', 'gu_in', 'ha_ng', 'he_il', 'hi_in', 'hr_hr', 'hu_hu', 'hy_am', 'id_id', 'ig_ng', 'is_is', 'it_it', 'ja_jp', 'jv_id', 'ka_ge', 'kam_ke', 'kea_cv', 'kk_kz', 'km_kh', 'kn_in', 'ko_kr', 'ky_kg', 'lb_lu', 'lg_ug', 'ln_cd', 'lo_la', 'lt_lt', 'luo_ke', 'lv_lv', 'mi_nz', 'mk_mk', 'ml_in', 'mn_mn', 'mr_in', 'ms_my', 'mt_mt', 'my_mm', 'nb_no', 'ne_np', 'nl_nl', 'nso_za', 'ny_mw', 'oc_fr', 'om_et', 'or_in', 'pa_in', 'pl_pl', 'ps_af', 'pt_br', 'ro_ro', 'ru_ru', 'sd_in', 'sk_sk', 'sl_si', 'sn_zw', 'so_so', 'sr_rs', 'sv_se', 'sw_ke', 'ta_in', 'te_in', 'tg_tj', 'th_th', 'tr_tr', 'uk_ua', 'umb_ao', 'ur_pk', 'uz_uz', 'vi_vn', 'wo_sn', 'xh_za', 'yo_ng', 'yue_hant_hk', 'zu_za', 'all']
Example of usage:
	`load_dataset('fleurs', 'af_za')`
```

Note the example of usage in the error message suggests loading "fleurs" instead of "google/fleurs".
GitHub issue: Support soft links for load_datasets imagefolder
State: open

### Feature request

Load_dataset from a folder of images doesn't seem to support soft links. It would be nice if it did, especially during methods development where image folders are being curated.

### Motivation

Images are coming from a complex variety of sources and we'd like to be able to soft link directly from the originating folders as opposed to copying. Having a copy of the file ensures that there may be issues with image versioning as well as having double the amount of required disk space.

### Your contribution

N/A
GitHub issue: Write token isn't working while pushing to datasets
State: closed

### Describe the bug

<img width="1001" alt="Screenshot 2024-05-01 at 3 37 06 AM" src="https://github.com/huggingface/datasets/assets/130903099/00fcf12c-fcc1-4749-8592-d263d4efcbcc">

As you can see I logged in to my account and the write token is valid.

But I can't upload on my main account and I am getting that error. It was okay on my test account at first try.

(I refreshed the token, tried a new token but still doesn't work)

### Steps to reproduce the bug

1. I loaded a dataset.

2. I logged in using both cli and huggingface_hub

3. I pushed to my down dataset
(It went well without any issues on my test account)

### Expected behavior

It should have gone smoothly and this is not even my first time uploading to huggingface datasets

### Environment info

colab, dataset (tried multiple versions)
GitHub issue: load_dataset('emotion')    UnicodeDecodeError
State: open

### Describe the bug

**emotions = load_dataset('emotion')**

_UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte_

### Steps to reproduce the bug

load_dataset('emotion') 

### Expected behavior

succese

### Environment info

py3.10
transformers  4.41.0.dev0
datasets     2.19.0
GitHub issue: Problem loading voxpopuli dataset 
State: closed

### Describe the bug

```
Exception has occurred: FileNotFoundError
Couldn't find file at https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/{'en': 'data/en/asr_train.tsv'}
```

Error in logic for link url creation. The link should be  https://huggingface.co/datasets/facebook/voxpopuli/resolve/main/data/en/asr_train.tsv 

Basically there should be links directly under ```metadata["train"]```, not under ```metadata["train"][self.config.languages[0]]```

same for audio urls

### Steps to reproduce the bug

```
from datasets import load_dataset

dataset = load_dataset("facebook/voxpopuli","en")
```

### Expected behavior

Dataset should be loaded successfully.

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-5.15.0-1041-aws-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.12.2
GitHub issue: [Streaming] Only load requested splits without resolving files for the other splits
State: open

e.g. [thangvip](https://huggingface.co/thangvip)/[cosmopedia_vi_math](https://huggingface.co/datasets/thangvip/cosmopedia_vi_math) has 300 splits and it takes a very long time to load only one split.

This is due to `load_dataset()` resolving the files of all the splits even if only one is needed.

In `dataset-viewer` the splits are loaded in different jobs so it results in 300 jobs that resolve 300 splits -> 90k calls to `/paths-info`
GitHub issue: Unimaginable super slow iteration
State: closed

### Describe the bug

Assuming there is a dataset with 52000 sentences, each with a length of 500, it takes 20 seconds to extract a sentence from the dataset……？Is there something wrong with my iteration?

### Steps to reproduce the bug

```python
import datasets
import time
import random

num_rows = 52000
num_cols = 500

random_input = [[random.randint(1, 100) for _ in range(num_cols)] for _ in range(num_rows)]
random_output = [[random.randint(1, 100) for _ in range(num_cols)] for _ in range(num_rows)]


s=time.time()
d={'random_input':random_input,'random_output':random_output}
dataset=datasets.Dataset.from_dict(d)
print('from dict',time.time()-s)
print(dataset)


for i in range(len(dataset)):
    aa=time.time()
    a,b=dataset['random_input'][i],dataset['random_output'][i]
    print(time.time()-aa)

```

corresponding output
```bash
from dict 9.215498685836792
Dataset({
    features: ['random_input', 'random_output'],
    num_rows: 52000
})
19.129778146743774
19.329464197158813
19.27668261528015
19.28557538986206
19.247620582580566
19.624247074127197
19.28673791885376
19.301053047180176
19.290496110916138
19.291821718215942
19.357765197753906

```

### Expected behavior

Under normal circumstances, iteration should be very rapid as it does not involve the main tasks other than getting items

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-3.10.0-1160.71.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.13
- `huggingface_hub` version: 0.21.4
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0
GitHub issue: Datasets with files with colon : in filenames cannot be used on Windows
State: open

### Describe the bug

Datasets (such as https://huggingface.co/datasets/MLCommons/peoples_speech) cannot be used on Windows due to the fact that windows does not allow colons ":" in filenames. These should be converted into alternative strings.



### Steps to reproduce the bug

1. Attempt to run load_dataset on MLCommons/peoples_speech

### Expected behavior

Does not crash during extraction

### Environment info

Windows 11, NTFS filesystem, Python 3.12
GitHub issue: Cannot use cached dataset without Internet connection (or when servers are down)
State: open

### Describe the bug

I want to be able to use cached dataset from HuggingFace even when I have no Internet connection (or when HuggingFace servers are down, or my company has network issues).  
The problem why I can't use it:
`data_files` argument from `datasets.load_dataset()` function get it updates from the server before calculating hash for caching. As a result, when I run the same code with and without Internet I get different dataset configuration directory name.

### Steps to reproduce the bug

```
import datasets

c4_dataset = datasets.load_dataset(
    path="allenai/c4",
    data_files={"train": "en/c4-train.00000-of-01024.json.gz"},
    split="train",
    cache_dir="/datesets/cache",
    download_mode="reuse_cache_if_exists",
    token=False,
)
```
1. Run this code with the Internet.
2. Run the same code without the Internet.

### Expected behavior

When running without the Internet connection, the loader should be able to get dataset from cache

### Environment info

- `datasets` version: 2.19.0
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.10.13
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.12.2
GitHub issue: ExpectedMoreSplits error on load_dataset when upgrading to 2.19.0
State: open

### Describe the bug

Hi there, thanks for the great library! We have been using it a lot in torchtune and it's been a huge help for us. 

Regarding the bug: the same call to `load_dataset` errors with `ExpectedMoreSplits` in 2.19.0 after working fine in 2.18.0. Full details given in the repro below.

### Steps to reproduce the bug

On 2.18.0, things work fine:
```
# First clear the locally cached dataset
rm -r ~/.cache/huggingface/datasets/lvwerra___stack-exchange-paired
pip install "datasets==2.18.0"
python3
>>> from datasets import load_dataset
>>> dataset = load_dataset('lvwerra/stack-exchange-paired', split='train', data_dir='data/rl')
```

On 2.19.0, they do not:
```
# First clear the locally cached dataset
rm -r ~/.cache/huggingface/datasets/lvwerra___stack-exchange-paired
pip install "datasets==2.19.0"
python3
>>> from datasets import load_dataset
>>> dataset = load_dataset('lvwerra/stack-exchange-paired', split='train', data_dir='data/rl')
```

The stack trace I see from the 2.19.0 version of load_dataset can be seen [here](https://gist.github.com/ebsmothers/f9b1f1949bee7030a8d7bb8a491550d2).

(Maybe unsurprising but) notably if I do not delete the cache first I am able to load the dataset successfully. So based on this I suspect the cause is somewhere in the download logic.

### Expected behavior

Download the dataset successfully :)

### Environment info

- `datasets` version: 2.19.0
- Platform: Linux-5.12.0-0_fbk16_zion_7661_geb00762ce6d2-x86_64-with-glibc2.34
- Python version: 3.11.9
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.3.1
GitHub issue: largelisttype not supported (.from_polars())
State: closed

### Describe the bug

The following code fails because LargeListType is not supported.
This is especially a problem for .from_polars since polars uses LargeListType.


### Steps to reproduce the bug

```python
import datasets
import polars as pl

df = pl.DataFrame({"list": [[]]})
datasets.Dataset.from_polars(df)
```

### Expected behavior

Convert LargeListType to list.

### Environment info

- `datasets` version: 2.19.1.dev0
- Platform: Linux-6.8.7-200.fc39.x86_64-x86_64-with-glibc2.38
- Python version: 3.12.2
- `huggingface_hub` version: 0.22.2
- PyArrow version: 16.0.0
- Pandas version: 2.1.4
- `fsspec` version: 2024.3.1
GitHub issue: Super slow iteration with trivial custom transform
State: open

### Describe the bug

Dataset is 10X slower when applying trivial transforms:

```
import time
import numpy as np
from datasets import Dataset, Features, Array2D

a = np.zeros((800, 800))
a = np.stack([a] * 1000)
features = Features({"a": Array2D(shape=(800, 800), dtype="uint8")})

ds1 = Dataset.from_dict({"a": a}, features=features).with_format('numpy')

def transform(batch):
	return batch

ds2 = ds1.with_transform(transform)

%time sum(1 for _ in ds1)
%time sum(1 for _ in ds2)
```

```
CPU times: user 472 ms, sys: 319 ms, total: 791 ms
Wall time: 794 ms
CPU times: user 9.32 s, sys: 443 ms, total: 9.76 s
Wall time: 9.78 s
```

In my real code I'm using set_transform to apply some post-processing on-the-fly for the 2d array, but it significantly slows down the dataset even if the transform itself is trivial.

Related issue: https://github.com/huggingface/datasets/issues/5841

### Steps to reproduce the bug

Use code in the description to reproduce.

### Expected behavior

Trivial custom transform in the example should not slowdown the dataset iteration.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.0-79-generic-x86_64-with-glibc2.35
- Python version: 3.11.4
- `huggingface_hub` version: 0.20.2
- PyArrow version: 15.0.0
- Pandas version: 1.5.3
- `fsspec` version: 2023.12.2
GitHub issue: Support downloading specific splits in `load_dataset`
State: open

This PR builds on https://github.com/huggingface/datasets/pull/6639 to support downloading only the specified splits in `load_dataset`. For this to work, a builder's `_split_generators` need to be able to accept the requested splits (as a list) via a `splits` argument to avoid processing the non-requested ones. Also, the builder has to define a `_available_splits` method that lists all the possible `splits` values.

Close https://github.com/huggingface/datasets/issues/4101, close https://github.com/huggingface/datasets/issues/2538 (I'm probably missing some)

Should also make it possible to address https://github.com/huggingface/datasets/issues/6793