sjwsjw/github-issues huggingface dataset:
GitHub issue: Support for sparse arrays with the Arrow Sparse Tensor format?
State: open

### Feature request

AI in biology is becoming a big thing. One thing that would be a huge benefit to the field that Huggingface Datasets doesn't currently have is native support for **sparse arrays**. 


Arrow has support for sparse tensors. 
https://arrow.apache.org/docs/format/Other.html#sparse-tensor


It would be a big deal if Hugging Face Datasets supported sparse tensors as a feature type, natively. 


### Motivation

This is important for example in the field of transcriptomics (modeling and understanding gene expression), because a large fraction of the genes are not expressed (zero). More generally, in science, sparse arrays are very common, so adding support for them would be very benefitial, it would make just using Hugging Face Dataset objects a lot more straightforward and clean.


### Your contribution

We can discuss this further once the team comments of what they think about the feature, and if there were previous attempts at making it work, and understanding their evaluation of how hard it would be. My intuition is that it should be fairly straightforward, as the Arrow backend already supports it.
GitHub issue: vllm批量推理报错
State: open

### Describe the bug

![Image](https://github.com/user-attachments/assets/3d958e43-28dc-4467-9333-5990c7af3b3f)

### Steps to reproduce the bug

![Image](https://github.com/user-attachments/assets/3067eeca-a54d-4956-b0fd-3fc5ea93dabb)

### Expected behavior

![Image](https://github.com/user-attachments/assets/77d32936-488f-4572-9365-bfb4170e555b)

### Environment info

![Image](https://github.com/user-attachments/assets/230335c4-825f-4db1-b07d-4776ef63ead8)
GitHub issue: Excessive RAM Usage After Dataset Concatenation concatenate_datasets
State: open

### Describe the bug

When loading a dataset from disk, concatenating it, and starting the training process, the RAM usage progressively increases until the kernel terminates the process due to excessive memory consumption.

https://github.com/huggingface/datasets/issues/2276

### Steps to reproduce the bug

```python
from datasets import  DatasetDict, concatenate_datasets

dataset = DatasetDict.load_from_disk("data")

...
...

combined_dataset = concatenate_datasets(
        [dataset[split] for split in dataset]
    )

#start SentenceTransformer training
```

### Expected behavior

I would not expect RAM utilization to increase after concatenation. Removing the concatenation step resolves the issue

### Environment info

sentence-transformers==3.1.1
datasets==3.2.0

python3.10
GitHub issue: Inconsistent Behavior Between `load_dataset` and `load_from_disk` When Loading Sharded Datasets
State: open

### Description

I encountered an inconsistency in behavior between `load_dataset` and `load_from_disk` when loading sharded datasets. Here is a minimal example to reproduce the issue:

#### Code 1: Using `load_dataset`
```python
from datasets import Dataset, load_dataset

# First save with max_shard_size=10
Dataset.from_dict({"id": range(1000)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Second save with max_shard_size=10
Dataset.from_dict({"id": range(500)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Load the DatasetDict
loaded_datasetdict = load_dataset("my_sharded_datasetdict")
print(loaded_datasetdict)
```
**Output**:
- `train` has 1350 samples.
- `test` has 150 samples.

#### Code 2: Using `load_from_disk`
```python
from datasets import Dataset, load_from_disk

# First save with max_shard_size=10
Dataset.from_dict({"id": range(1000)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Second save with max_shard_size=10
Dataset.from_dict({"id": range(500)}).train_test_split(test_size=0.1).save_to_disk("my_sharded_datasetdict", max_shard_size=10)

# Load the DatasetDict
loaded_datasetdict = load_from_disk("my_sharded_datasetdict")
print(loaded_datasetdict)
```
**Output**:
- `train` has 450 samples.
- `test` has 50 samples.

### Expected Behavior
I expected both `load_dataset` and `load_from_disk` to load the same dataset, as they are pointing to the same directory. However, the results differ significantly:
- `load_dataset` seems to merge all shards, resulting in a combined dataset.
- `load_from_disk` only loads the last saved dataset, ignoring previous shards.

### Questions
1. Is this behavior intentional? If so, could you clarify the difference between `load_dataset` and `load_from_disk` in the documentation?
2. If this is not intentional, could this be considered a bug?
3. What is the recommended way to handle cases where multiple datasets are saved to the same directory?


Thank you for your time and effort in maintaining this great library! I look forward to your feedback.
GitHub issue: Support faster processing using pandas or polars functions in `IterableDataset.map()`
State: closed

Following the polars integration :)

Allow super fast processing using pandas or polars functions in `IterableDataset.map()` by adding support to pandas and polars formatting in `IterableDataset`

```python
import polars as pl
from datasets import Dataset

ds = Dataset.from_dict({"i": range(10)}).to_iterable_dataset()
ds = ds.with_format("polars")
ds = ds.map(lambda df: df.with_columns(pl.col("i").add(1).alias("i+1")), batched=True)
ds = ds.with_format(None)
print(next(iter(ds)))
# {'i': 0, 'i+1': 1}
```

It leverages arrow's zero-copy features from/to pandas and polars.

related to https://github.com/huggingface/datasets/issues/3444 #6762
GitHub issue: Importing dataset gives unhelpful error message when filenames in metadata.csv are not found in the directory
State: open

### Describe the bug

While importing an audiofolder dataset, where the names of the audiofiles don't correspond to the filenames in the metadata.csv, we get an unclear error message that is not helpful for the debugging, i.e. 
```
ValueError: Instruction "train" corresponds to no data!
```

### Steps to reproduce the bug

Assume an audiofolder with audiofiles, filename1.mp3, filename2.mp3 etc and a file metadata.csv which contains the columns file_name and sentence. The file_names are formatted like filename1.mp3, filename2.mp3 etc. 

Load the audio
```
from datasets import load_dataset
load_dataset("audiofolder", data_dir='/path/to/audiofolder')
```
When the file_names in the csv are not in sync with the filenames in the audiofolder, then we get an Error message:
```
File /opt/conda/lib/python3.12/site-packages/datasets/arrow_reader.py:251, in BaseReader.read(self, name, instructions, split_infos, in_memory)
    249 if not files:
    250     msg = f'Instruction "{instructions}" corresponds to no data!'
--> 251     raise ValueError(msg)
    252 return self.read_files(files=files, original_instructions=instructions, in_memory=in_memory)

ValueError: Instruction "train" corresponds to no data!
```
load_dataset has a default value for the argument split = 'train'. 


### Expected behavior

It would be better to get an error report something like:
```
The metadata.csv file has different filenames than the files in the datadirectory. 
```
It would have saved me 4 hours of debugging. 

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-427.40.1.el9_4.x86_64-x86_64-with-glibc2.39
- Python version: 3.12.8
- `huggingface_hub` version: 0.27.0
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
GitHub issue: Dataset.from_dict() can't handle large dict
State: open

### Describe the bug

I have 26,000,000 3-tuples. When I use Dataset.from_dict() to load, neither. py nor Jupiter notebook can run successfully. This is my code:
```
    # len(example_data) is 26,000,000, 'diff' is a text
    diff1_list = [example_data[i].texts[0] for i in range(len(example_data))]
    diff2_list = [example_data[i].texts[1] for i in range(len(example_data))]
    label_list = [example_data[i].label for i in range(len(example_data))]

    embedding_dataset = Dataset.from_dict({
        "diff1": diff1_list,
        "diff2": diff2_list,
        "label": label_list
    })
```

### Steps to reproduce the bug

1. Initialize a large 3-tuple, e.g. 26,000,000
2. Use Dataset.from_dict() to load

### Expected behavior

Dataset.from_dict() run successfully

### Environment info

sentence-transformers                   3.3.1
GitHub issue: A parameter is specified but not used in datasets.arrow_dataset.Dataset.from_pandas()
State: open

### Describe the bug

I am interested in creating train, test and eval splits from a pandas Dataframe, therefore I was looking at the possibilities I can follow. I noticed the split parameter and was hopeful to use it in order to generate the 3 at once, however, while trying to understand the code, i noticed that it has no added value (correct me if I am wrong or misunderstood the code). 


from_pandas function code :

```python
    if info is not None and features is not None and info.features != features:
            raise ValueError(
                f"Features specified in `features` and `info.features` can't be different:\n{features}\n{info.features}"
            )
        features = features if features is not None else info.features if info is not None else None
        if info is None:
            info = DatasetInfo()
        info.features = features
        table = InMemoryTable.from_pandas(
            df=df,
            preserve_index=preserve_index,
        )
        if features is not None:
            # more expensive cast than InMemoryTable.from_pandas(..., schema=features.arrow_schema)
            # needed to support the str to Audio conversion for instance
            table = table.cast(features.arrow_schema)
        return cls(table, info=info, split=split)
```

### Steps to reproduce the bug

```python
from datasets import Dataset
# Filling the split parameter with whatever causes no harm at all
data = Dataset.from_pandas(self.raw_data, split='egiojegoierjgoiejgrefiergiuorenvuirgurthgi')
```

### Expected behavior

Would be great if there is no split parameter (if it isn't working), or to add a concrete example of how it can be used.

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.15.0-127-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.27.1
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
GitHub issue: API endpoints for gated dataset access requests
State: closed

### Feature request

I would like a programatic way of requesting access to gated datasets. The current solution to gain access forces me to visit a website and physically click an "agreement" button (as per the [documentation](https://huggingface.co/docs/hub/en/datasets-gated#access-gated-datasets-as-a-user)).

An ideal approach would be HF API download methods that negotiate access on my behalf based on information from my CLI login and/or token. I realise that may be naive given the various types of access semantics available to dataset authors (automatic versus manual approval, for example) and complexities it might add to existing methods, but something along those lines would be nice.

Perhaps using the `*_access_request` methods available to dataset authors can be a precedent; see [`reject_access_request`](https://huggingface.co/docs/huggingface_hub/main/en/package_reference/hf_api#huggingface_hub.HfApi.reject_access_request) for example.

### Motivation

When trying to download files from a gated dataset, I'm met with a `GatedRepoError` and instructed to visit the repository's website to gain access:

```
Cannot access gated repo for url https://huggingface.co/datasets/open-llm-leaderboard/meta-llama__Meta-Llama-3.1-70B-Instruct-details/resolve/main/meta-llama__Meta-Llama-3.1-70B-Instruct/samples_leaderboard_math_precalculus_hard_2024-07-19T18-47-29.522341.jsonl.
Access to dataset open-llm-leaderboard/meta-llama__Meta-Llama-3.1-70B-Instruct-details is restricted and you are not in the authorized list. Visit https://huggingface.co/datasets/open-llm-leaderboard/meta-llama__Meta-Llama-3.1-70B-Instruct-details to ask for access.
```

This makes task automation extremely difficult. For example, I'm interested in studying sample-level responses of models on the LLM leaderboard -- how they answered particular questions on a given evaluation framework. As I come across more and more participants that gate their data, it's becoming unwieldy to continue my work (there over 2,000 participants, so in the worst case that's the number of website visits I'd need to manually undertake).

One approach is use Selenium to react to the `GatedRepoError`, but that seems like overkill; and a potential violation HF terms of service (?).

As mentioned in the previous section, there seems to be an [API for gated dataset owners](https://huggingface.co/docs/hub/en/datasets-gated#via-the-api) to managed access requests, and thus some appetite for allowing automated management of gating. This feature request is to extend that to dataset users.

### Your contribution

Whether I can help depends on a few things; one being the complexity of the underlying gated access design. If this feature request is accepted I am open to being involved in discussions and testing, and even development under the right time-outcome tradeoff.
GitHub issue: ImportError: To support decoding images, please install 'Pillow'.
State: open

### Describe the bug

Following this tutorial locally using a macboko and VSCode: https://huggingface.co/docs/diffusers/en/tutorials/basic_training
This line of code: for i, image in enumerate(dataset[:4]["image"]):
throws: ImportError: To support decoding images, please install 'Pillow'.

Pillow is installed.

### Steps to reproduce the bug

Run the tutorial


### Expected behavior

Images should be rendered

### Environment info

MacBook, VSCode
GitHub issue: HuggingFace CLI dataset download raises error
State: closed

### Describe the bug

Trying to download Hugging Face datasets using Hugging Face CLI raises error. This error only started after December 27th, 2024.  For example:
```
huggingface-cli download --repo-type dataset gboleda/wikicorpus
Traceback (most recent call last):
  File "/home/ubuntu/test_venv/bin/huggingface-cli", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/test_venv/lib/python3.10/site-packages/huggingface_hub/commands/huggingface_cli.py", line 51, in main
    service.run()
  File "/home/ubuntu/test_venv/lib/python3.10/site-packages/huggingface_hub/commands/download.py", line 146, in run
    print(self._download())  # Print path to downloaded files
  File "/home/ubuntu/test_venv/lib/python3.10/site-packages/huggingface_hub/commands/download.py", line 180, in _download
    return snapshot_download(
  File "/home/ubuntu/test_venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/ubuntu/test_venv/lib/python3.10/site-packages/huggingface_hub/_snapshot_download.py", line 164, in snapshot_download
    repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision, token=token)
  File "/home/ubuntu/test_venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/ubuntu/test_venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2491, in repo_info
    return method(
  File "/home/ubuntu/test_venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
  File "/home/ubuntu/test_venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 2366, in dataset_info
    return DatasetInfo(**data)
  File "/home/ubuntu/test_venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 799, in __init__
    self.tags = kwargs.pop("tags")
KeyError: 'tags'
```

### Steps to reproduce the bug

```
1. huggingface-cli download --repo-type dataset gboleda/wikicorpus
```


### Expected behavior

There should be no error.

### Environment info


- `datasets` version: 2.19.1
- Platform: Linux-6.8.0-1015-aws-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.23.5
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.3.1
GitHub issue: Fix lock permission
State: open

All files except lock file have proper permission obeying `ACL` property if it is set.

If the cache directory has `ACL` property, it should be respected instead of just using `umask` for permission.
To fix it, just create a lock file and pass the created `mode`.

By creating a lock file with `touch()` before `FileLock` create it with `mode`,
- if `ACL` is not set, same as before
- if `ACL` is set, `ACL` is respected

If it is acceptable, it should be also applied to [`huggingface_hub`](https://github.com/huggingface/huggingface_hub/blob/2702ec2a2bd0124cc1fddfd72ccb1297b2478148/src/huggingface_hub/utils/_fixes.py#L95) I guess.
GitHub issue: How about adding a feature to pass the key when performing map on DatasetDict?
State: closed

### Feature request

Add a feature to pass the key of the DatasetDict when performing map

### Motivation

I often preprocess using map on DatasetDict. 
Sometimes, I need to preprocess train and valid data differently depending on the task. 
So, I thought it would be nice to pass the key (like train, valid) when performing map on DatasetDict. 

What do you think?

### Your contribution

I can submit a pull request to add the feature to pass the key of the DatasetDict when performing map.
GitHub issue: Not available datasets[audio] on python 3.13
State: open

### Describe the bug

This is the error I got, it seems numba package does not support python 3.13

PS C:\Users\sergi\Documents> pip install datasets[audio]
Defaulting to user installation because normal site-packages is not writeable
Collecting datasets[audio]
  Using cached datasets-3.2.0-py3-none-any.whl.metadata (20 kB)
... (OTHER PACKAGES)
Collecting numba>=0.51.0 (from librosa->datasets[audio])
  Downloading numba-0.60.0.tar.gz (2.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.7/2.7 MB 44.1 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [24 lines of output]
      Traceback (most recent call last):
        File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.496.0_x64__qbz5n2kfra8p0\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
          ~~~~^^
        File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.496.0_x64__qbz5n2kfra8p0\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.496.0_x64__qbz5n2kfra8p0\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "C:\Users\sergi\AppData\Local\Temp\pip-build-env-yauns_qh\overlay\Lib\site-packages\setuptools\build_meta.py", line 334, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
                 ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\sergi\AppData\Local\Temp\pip-build-env-yauns_qh\overlay\Lib\site-packages\setuptools\build_meta.py", line 304, in _get_build_requires
          self.run_setup()
          ~~~~~~~~~~~~~~^^
      RuntimeError: Cannot install on Python version 3.13.1; only versions >=3.9,<3.13 are supported.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

### Steps to reproduce the bug

1. install python >=3.13
2. !pip install datasets[audio]

### Expected behavior

I needed datasets[audio] in the python 3.13

### Environment info

python 3.13.1
GitHub issue: A module that was compiled using NumPy 1.x cannot be run in NumPy 2.0.2 as it may crash. To support both 1.x and 2.x versions of NumPy, modules must be compiled with NumPy 2.0. Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
State: closed

### Describe the bug

Following this tutorial: https://huggingface.co/docs/diffusers/en/tutorials/basic_training and running it locally using VSCode on my MacBook.  The first line in the tutorial fails:  from datasets import load_dataset
dataset = load_dataset('huggan/smithsonian_butterflies_subset', split="train"). with this error: 
A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.
If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2. and ImportError: numpy.core.multiarray failed to import.   

Does from datasets import load_dataset really use NumPy 1.x?  

### Steps to reproduce the bug

Open VSCode.  create a new venv.  Create a new ipynb file.  Import pip install diffusers[training] try to run this line of code: from datasets import load_dataset

### Expected behavior

data is loaded

### Environment info

ran this: datasets-cli env
and got A module that was compiled using NumPy 1.x cannot be run in
NumPy 2.0.2 as it may crash. To support both 1.x and 2.x
versions of NumPy, modules must be compiled with NumPy 2.0.
Some module may need to rebuild instead e.g. with 'pybind11>=2.12'.

If you are a user of the module, the easiest solution will be to
downgrade to 'numpy<2' or try to upgrade the affected module.
We expect that some modules will need time to support NumPy 2.
GitHub issue: Converting Arrow to WebDataset TAR Format for Offline Use
State: closed

### Feature request

Hi, 

I've downloaded an Arrow-formatted dataset offline using the hugggingface's datasets library by:

```
import json
from datasets import load_dataset

dataset = load_dataset("pixparse/cc3m-wds")
dataset.save_to_disk("./cc3m_1") 
```

now I need to convert it to WebDataset's TAR format for offline data ingestion. 
Is there a straightforward method to achieve this conversion without an internet connection? Can I simply convert it by 
```
tar -cvf 
```

btw, when I tried:
```
import webdataset as wds
from huggingface_hub import get_token
from torch.utils.data import DataLoader

hf_token = get_token()
url = "https://huggingface.co/datasets/timm/imagenet-12k-wds/resolve/main/imagenet12k-train-{{0000..1023}}.tar"
url = f"pipe:curl -s -L {url} -H 'Authorization:Bearer {hf_token}'"
dataset = wds.WebDataset(url).decode()
dataset.save_to_disk("./cc3m_webdataset") 
```
error occured:
```
AttributeError: 'WebDataset' object has no attribute 'save_to_disk'
```

Thanks a lot!

### Motivation

Converting Arrow to WebDataset TAR Format

### Your contribution

No clue yet
GitHub issue: Different behaviour of IterableDataset.map vs Dataset.map with remove_columns
State: closed

### Describe the bug

The following code

```python
import datasets as hf
ds1 = hf.Dataset.from_list([{'i': i} for i in [0,1]])
#ds1 = ds1.to_iterable_dataset()
ds2 = ds1.map(
    lambda i: {'i': i+1},
    input_columns = ['i'],
    remove_columns = ['i']
)   
list(ds2)
```

produces

```python
[{'i': 1}, {'i': 2}]
```

as expected. If the line that converts `ds1` to iterable is uncommented so that the `ds2` is a map of an `IterableDataset`, the result is 

```python
[{},{}]
```

I expected the output to be the same as before. It seems that in the second case the removed column is not added back into the output.

The issue seems to be [here](https://github.com/huggingface/datasets/blob/6c6a82a573f946c4a81069f56446caed15cee9c2/src/datasets/iterable_dataset.py#L1093): the columns are removed after the mapping which is not what we want (or what the [documentation says](https://github.com/huggingface/datasets/blob/6c6a82a573f946c4a81069f56446caed15cee9c2/src/datasets/iterable_dataset.py#L2370)) because we want the columns removed from the transformed example but then added if the map produced them.

This is `datasets==3.2.0` and `python==3.10`



### Steps to reproduce the bug

see above

### Expected behavior

see above

### Environment info

see above
GitHub issue: [Bug] Inconsistent behavior of data_files and data_dir in load_dataset method.
State: closed

### Describe the bug

Inconsistent operation of data_files and data_dir in load_dataset method.


### Steps to reproduce the bug

# First
I have three files, named 'train.json', 'val.json', 'test.json'.
Each one has a simple dict `{text:'aaa'}`.
Their path are `/data/train.json`,  `/data/val.json`,  `/data/test.json`
I load dataset with `data_files` argument:
```py
files = [os.path.join('./data',file) for file in os.listdir('./data')]
ds = load_dataset(
            path='json',
            data_files=files,)
```
And I get:
```py
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 3
    })
})
```
However, If I load dataset with `data_dir` argument:
```py
ds = load_dataset(
            path='json',
            data_dir='./data',)
```
And I get:
```py
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 1
    })
    validation: Dataset({
        features: ['text'],
        num_rows: 1
    })
    test: Dataset({
        features: ['text'],
        num_rows: 1
    })
})
```
Two results are not the same. Their behaviors are not equal, even if the statement [here](https://github.com/huggingface/datasets/blob/d0c152a979d91cc34b605c0298aebc650ab7dd27/src/datasets/load.py#L1790) said that their behaviors are equal. 


# Second
If some filename include 'test' while others do not, `load_dataset` only return `test` dataset and others files are **abandoned**.
Given two files named `test.json` and `1.json`
Each one has a simple dict `{text:'aaa'}`.
I load the dataset using:
```py
ds = load_dataset(
            path='json',
            data_dir='./data',)
```
Only `test` is returned, `1.json` is missing:
```py
DatasetDict({
    test: Dataset({
        features: ['text'],
        num_rows: 1
    })
})
```
Things do not change even I manually set `split='train'`

 



### Expected behavior

1. Fix the above bugs.
2. Although the document says that load_dataset method will `Find which file goes into which split (e.g. train/test) based on file and directory names or on the YAML configuration`, I hope I can manually decide whether to do so. Sometimes users may accidentally put a `test` string in the filename but they just want a single `train` dataset. If the number of files in `data_dir` is huge, it's not easy to find out what cause the second situation metioned above.

### Environment info

datasets==3.2.0
Ubuntu18.84
GitHub issue: One or several metadata.jsonl were found, but not in the same directory or in a parent directory of
State: open

### Describe the bug

ImageFolder with metadata.jsonl error. I downloaded liuhaotian/LLaVA-CC3M-Pretrain-595K locally from Hugging Face. According to the tutorial in https://huggingface.co/docs/datasets/image_dataset#image-captioning, only put images.zip and metadata.jsonl containing information in the same folder. However, after loading, an error was reported: One or several metadata.jsonl were found, but not in the same directory or in a parent directory of.

The data in my jsonl file is as follows:

> {"id": "GCC_train_002448550", "file_name": "GCC_train_002448550.jpg", "conversations": [{"from": "human", "value": "<image>\nProvide a brief description of the given image."}, {"from": "gpt", "value": "a view of a city , where the flyover was proposed to reduce the increasing traffic on thursday ."}]}


### Steps to reproduce the bug

from datasets import load_dataset
image = load_dataset("imagefolder",data_dir='data/opensource_data')

### Expected behavior

success

### Environment info

datasets==3.2.0
GitHub issue: Clarify documentation or Create DatasetCard
State: open

### Feature request

I noticed that you can use a Model Card instead of a Dataset Card when pushing a dataset to the Hub, but this isn’t clearly mentioned in [the docs.](https://huggingface.co/docs/datasets/dataset_card)

- Update the docs to clarify that a Model Card can work for datasets too.
- It might be worth creating a dedicated DatasetCard module, similar to the ModelCard module, for consistency and better support.

Not sure if this belongs here or on the [Hub repo](https://github.com/huggingface/huggingface_hub), but thought I’d bring it up!

### Motivation

I just spent an hour like on [this issue](https://github.com/huggingface/trl/pull/2491) trying to create a `DatasetCard` for a script.

### Your contribution

might later
GitHub issue: Too many open files: '/root/.cache/huggingface/token'
State: open

### Describe the bug

I ran this code:
```
from datasets import load_dataset
dataset = load_dataset("common-canvas/commoncatalog-cc-by", cache_dir="/datadrive/datasets/cc", num_proc=1000)
```

And got this error.
Before it was some other file though (lie something...incomplete)

runnting
```
ulimit -n 8192
```
did not help at all.

### Steps to reproduce the bug

Run the code i sent

### Expected behavior

Should be no errors 

### Environment info

linux, jupyter lab.
GitHub issue: TypeError: Value.__init__() missing 1 required positional argument: 'dtype'
State: open

### Describe the bug

ds = load_dataset(
    "./xxx.py",
    name="default",
    split="train",
)

The datasets does not support debugging locally anymore...

### Steps to reproduce the bug

```
from datasets import load_dataset

ds = load_dataset(
    "./repo.py",
    name="default",
    split="train",
)

for item in ds:
    print(item)
```

It works fine for "username/repo", but it does not work for "./repo.py" when debugging locally...
Running above code template will report TypeError: Value.__init__() missing 1 required positional argument: 'dtype'

### Expected behavior

fix this bug

### Environment info

python 3.10 datasets==2.21
GitHub issue: .map() is not caching and ram goes OOM
State: open

### Describe the bug

Im trying to run a fairly simple map that is converting a dataset into numpy arrays. however, it just piles up on memory and doesnt write to disk. Ive tried multiple cache techniques such as specifying the cache dir, setting max mem, +++ but none seem to work. What am I missing here?

### Steps to reproduce the bug

```
from pydub import AudioSegment
import io
import base64
import numpy as np
import os
CACHE_PATH = "/mnt/extdisk/cache" # "/root/.cache/huggingface/"# 
os.environ["HF_HOME"] = CACHE_PATH
import datasets
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# Create a handler for Jupyter notebook
handler = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)

#datasets.config.IN_MEMORY_MAX_SIZE= 1000#*(2**30) #50 gb
print(datasets.config.HF_CACHE_HOME)
print(datasets.config.HF_DATASETS_CACHE)
# Decode the base64 string into bytes
def convert_mp3_to_audio_segment(example):
    """
    example = ds['train'][0]
    """
    try:
        audio_data_bytes = base64.b64decode(example['audio'])
        # Use pydub to load the MP3 audio from the decoded bytes
        audio_segment = AudioSegment.from_file(io.BytesIO(audio_data_bytes), format="mp3")
        # Resample to 24_000
        audio_segment = audio_segment.set_frame_rate(24_000)
        audio = {'sampling_rate' : audio_segment.frame_rate,
        'array' : np.array(audio_segment.get_array_of_samples(), dtype="float")}
        del audio_segment
        duration = len(audio['array']) / audio['sampling_rate']
    except Exception as e:
        logger.warning(f"Failed to convert audio for {example['id']}. Error: {e}")
        audio = {'sampling_rate' : 0,
        'array' : np.array([]), duration : 0}
    return {'audio' : audio, 'duration' : duration}

ds = datasets.load_dataset("NbAiLab/nb_distil_speech_noconcat_stortinget", cache_dir=CACHE_PATH, keep_in_memory=False)

#%%
num_proc=32
ds_processed = (
    ds
    #.select(range(10))
    .map(convert_mp3_to_audio_segment, num_proc=num_proc, desc="Converting mp3 to audio segment") #, cache_file_name=f"{CACHE_PATH}/stortinget_audio" # , cache_file_name="test"
)
```

### Expected behavior

the map should write to disk

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-6.8.0-45-generic-x86_64-with-glibc2.39
- Python version: 3.12.7
- `huggingface_hub` version: 0.26.3
- PyArrow version: 18.1.0
- Pandas version: 2.2.3
- `fsspec` version: 2024.9.0
GitHub issue: Remove upper bound for fsspec
State: open

### Describe the bug

As also raised by @cyyever in https://github.com/huggingface/datasets/pull/7296 and @NeilGirdhar in https://github.com/huggingface/datasets/commit/d5468836fe94e8be1ae093397dd43d4a2503b926#commitcomment-140952162 , `datasets` has a problematic version constraint on `fsspec`.

In our case this causes (unnecessary?) troubles due to a race condition bug in that version of the corresponding `gcsfs` plugin, that causes deadlocks: https://github.com/fsspec/gcsfs/pull/643

We just use a version override to ignore the constraint from `datasets`, but imho the version constraint could just be removed in the first place?

The last few PRs bumping the upper bound were basically uneventful:

* https://github.com/huggingface/datasets/pull/7219
* https://github.com/huggingface/datasets/pull/6921
* https://github.com/huggingface/datasets/pull/6747

### Steps to reproduce the bug

-

### Expected behavior

Installing `fsspec>=2024.10.0` along `datasets` should be possible without overwriting constraints.

### Environment info

All recent datasets versions
GitHub issue: Unexpected cache behaviour using load_dataset
State: closed

### Describe the bug

Following the (Cache management)[https://huggingface.co/docs/datasets/en/cache] docu and previous behaviour from datasets version 2.18.0, one is able to change the cache directory. Previously, all downloaded/extracted/etc files were found in this folder. As i have recently update to the latest version this is not the case anymore. Downloaded files are stored in `~/.cache/huggingface/hub`.
Providing the `cache_dir` argument in `load_dataset` the cache directory is created and there are some files but the bulk is still in `~/.cache/huggingface/hub`.

I believe this could be solved by adding the cache_dir argument [here](https://github.com/huggingface/datasets/blob/fdda5585ab18ea1292547f36c969d12c408ab842/src/datasets/utils/file_utils.py#L188)


### Steps to reproduce the bug

For example using https://huggingface.co/datasets/ashraq/esc50:
```python
from datasets import load_dataset
ds = load_dataset("ashraq/esc50", "default", cache_dir="~/custom/cache/path/esc50")
```


### Expected behavior

I would expect the bulk of files related to the dataset to be stored somewhere in `~/custom/cache/path/esc50`, but it seems they are in `~/.cache/huggingface/hub/datasets--ashraq--esc50`. 

### Environment info

- `datasets` version: 3.2.0
- Platform: Linux-5.14.0-503.15.1.el9_5.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.14
- `huggingface_hub` version: 0.26.5
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.6.1
GitHub issue: ArrowInvalid: JSON parse error: Column() changed from object to array in row 0
State: open

### Describe the bug

Encountering an error while loading the ```liuhaotian/LLaVA-Instruct-150K dataset```.

### Steps to reproduce the bug

```
from datasets import load_dataset
fw =load_dataset("liuhaotian/LLaVA-Instruct-150K")
```
Error:
```
ArrowInvalid                              Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/datasets/packaged_modules/json/json.py](https://localhost:8080/#) in _generate_tables(self, files)
    136                                 try:
--> 137                                     pa_table = paj.read_json(
    138                                         io.BytesIO(batch), read_options=paj.ReadOptions(block_size=block_size)

20 frames
ArrowInvalid: JSON parse error: Column() changed from object to array in row 0

During handling of the above exception, another exception occurred:

ArrowTypeError                            Traceback (most recent call last)
ArrowTypeError: ("Expected bytes, got a 'int' object", 'Conversion failed for column id with type object')

The above exception was the direct cause of the following exception:

DatasetGenerationError                    Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/datasets/builder.py](https://localhost:8080/#) in _prepare_split_single(self, gen_kwargs, fpath, file_format, max_shard_size, job_id)
   1895             if isinstance(e, DatasetGenerationError):
   1896                 raise
-> 1897             raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1898 
   1899         yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```
### Expected behavior

I have tried loading the dataset both on my own server and on Colab, and encountered errors in both instances.

### Environment info

```
- `datasets` version: 3.2.0
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- `huggingface_hub` version: 0.26.3
- PyArrow version: 17.0.0
- Pandas version: 2.2.2
- `fsspec` version: 2024.9.0
```
GitHub issue: ImportError: cannot import name 'set_caching_enabled' from 'datasets' 
State: open

### Describe the bug

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/local/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/home/Medusa/axolotl/src/axolotl/cli/__init__.py", line 23, in <module>
    from axolotl.train import TrainDatasetMeta
  File "/home/Medusa/axolotl/src/axolotl/train.py", line 23, in <module>
    from axolotl.utils.trainer import setup_trainer
  File "/home/Medusa/axolotl/src/axolotl/utils/trainer.py", line 13, in <module>
    from datasets import set_caching_enabled
ImportError: cannot import name 'set_caching_enabled' from 'datasets' (/usr/local/lib/python3.10/site-packages/datasets/__init__.py)

### Steps to reproduce the bug

1、axolotl
2、accelerate launch -m axolotl.cli.train examples/medusa/qwen_lora_stage1.yml

### Expected behavior

enable datasets

### Environment info

python3.10
GitHub issue: Introduce support for PDFs
State: open

### Feature request

The idea (discussed in the Discord server with @lhoestq ) is to have a Pdf type like Image/Audio/Video. For example [Video](https://github.com/huggingface/datasets/blob/main/src/datasets/features/video.py) was recently added and contains how to decode a video file encoded in a dictionary like {"path": ..., "bytes": ...} as a VideoReader using decord. We want to do the same with pdf and get a [pypdfium2.PdfDocument](https://pypdfium2.readthedocs.io/en/stable/_modules/pypdfium2/_helpers/document.html#PdfDocument).

### Motivation

In many cases PDFs contain very valuable information beyond text (e.g. images, figures). Support for PDFs would help create datasets where all the information is preserved.

### Your contribution

I can start the implementation of the Pdf type :)
GitHub issue: Cannot create a dataset with relative audio path
State: open

### Describe the bug

Hello! I want to create a dataset of parquet files, with audios stored as separate .mp3 files. However, it says "No such file or directory" (see the reproducing code).

### Steps to reproduce the bug

Creating a dataset
```
from pathlib import Path
from datasets import Dataset, load_dataset, Audio

Path('my_dataset/audio').mkdir(parents=True, exist_ok=True)
Path('my_dataset/audio/file.mp3').touch(exist_ok=True)
Dataset.from_list(
    [{'audio': {'path': 'audio/file.mp3'}}]
).to_parquet('my_dataset/data.parquet')
```

Result:
```
# my_dataset
# ├── audio
# │   └── file.mp3
# └── data.parquet
```

Trying to load the dataset
```
dataset = (
    load_dataset('my_dataset', split='train')
    .cast_column('audio', Audio(sampling_rate=16_000))
)
dataset[0]

>>> FileNotFoundError: [Errno 2] No such file or directory: 'audio/file.mp3'
```

### Expected behavior

I expect the dataset to load correctly.

I've found 2 workarounds, but they are not very good:
1. I can specify an absolute path to the audio, however, when I move the folder or upload to HF it will stop working.
2. I can set `'path': 'file.mp3'`, and load with `load_dataset('my_dataset', data_dir='audio')` - it seems to work, but does this mean that anyone from Hugging Face who wants to use this dataset should also pass the `data_dir` argument, otherwise it won't work?

### Environment info

datasets 3.1.0, Ubuntu 24.04.1
GitHub issue: [Audio Features - DO NOT MERGE] PoC for adding an offset+sliced reading to audio file.
State: open

This is a proof of concept for #7310 . The idea is to enable the access to others column of the dataset row when loading an audio file into a table. This is to allow sliced reading. As stated in the issue, many people have very long audio files and use start and stop slicing in this audio file.

Right now, this code work as a PoC on my dataset. However, this is **just to illustrate** the idea. Many things are messed up, the first being that the shards have wildly varying sizes. 

Could be of interest to @lhoestq and @sanchit-gandhi ? 

Happy to test better ideas locally.
GitHub issue: How to get the original dataset name with username?
State: open

### Feature request

The issue is related to ray data https://github.com/ray-project/ray/issues/49008 which it requires to check if the dataset is the original one just after `load_dataset` and parquet files are already available on hf hub.

The solution used now is to get the dataset name, config and split, then `load_dataset` again and check the fingerprint. But it's unable to get the correct dataset name if it contains username. So how to get the dataset name with username prefix, or is there another way to query if a dataset is the original one with parquet available?

@lhoestq 

### Motivation

https://github.com/ray-project/ray/issues/49008

### Your contribution

Would like to fix that.
GitHub issue: Enable the Audio Feature to decode / read with an offset + duration
State: open

### Feature request

For most large speech dataset, we do not wish to generate hundreds of millions of small audio samples. Instead, it is quite common to provide larger audio files with frame offset (soundfile start and stop arguments). We should be able to pass these arguments to Audio() (column ID corresponding in the dataset row).

### Motivation

I am currently generating a fairly big dataset to .parquet(). Unfortunately, it does not work because all existing functions load the whole .wav file corresponding to the row. All my attempts at bypassing this failed. We should be able to put in the Table only the bytes corresponding to what soundfile reads with an offset (and subset of the audio file). 

### Your contribution

I can totally test whatever code on my large dataset creation script.
GitHub issue: Allow manual configuration of Dataset Viewer for datasets not created with the `datasets` library
State: open

#### **Problem Description**
Currently, the Hugging Face Dataset Viewer automatically interprets dataset fields for datasets created with the `datasets` library. However, for datasets pushed directly via `git`, the Viewer:
- Defaults to generic columns like `label` with `null` values if no explicit mapping is provided.
- Does not allow dataset creators to configure field mappings or suppress default fields unless the dataset is recreated and pushed using the `datasets` library.

This creates a limitation for creators who:
- Use custom workflows to prepare datasets (e.g., manifest files with audio-transcription mappings).
- Push large datasets directly via `git` and cannot easily restructure them to conform to the `datasets` library format.

#### **Proposed Solution**
Introduce a feature that allows dataset creators to manually configure the Dataset Viewer behavior for datasets not created with the `datasets` library. This could be achieved by:
1. **Using the YAML Metadata in `README.md`:**
   - Add support for defining the dataset's field mappings directly in the `README.md` YAML section.
   - Example:
   
     ```yaml
     viewer:
       fields:
         - name: "audio"
           type: "audio_path" / "text"
           source: "manifest['audio']"
         - name: "bambara_transcription"
           type: "text"
           source: "manifest['bambara']"
         - name: "french_translation"
           type: "text"
           source: "manifest['french']"
     ```
With manifest being a csv or json like format file in the repository so that the viewer understands that it should look for the values of each field in that file. 

#### **Benefits**
- Improves flexibility for dataset creators who push datasets via `git`.
- Enhances dataset discoverability and usability on the Hugging Face Hub by allowing creators to present meaningful field mappings without restructuring their data.
- Reduces overhead for creators of large or complex datasets.

#### **Examples of Use Case**
- An audio dataset with transcriptions in multiple languages stored in a `manifest.json` file, where the user wants the Viewer to:
  - Display the `audio` column and Explicitly map features that he defined such as `bambara_transcription` and `french_translation` from the manifest.
GitHub issue: Faster parquet streaming + filters with predicate pushdown
State: closed

ParquetFragment.to_batches uses a buffered stream to read parquet data, which makes streaming faster (x2 on my laptop).

I also added the `filters` config parameter to support filtering with predicate pushdown, e.g.

```python
from datasets import load_dataset

filters = [('problem_source', '==', 'math')]
ds = load_dataset("nvidia/OpenMathInstruct-2", streaming=True, filters=filters)
first_example = next(iter(ds["train"]))
print(first_example["problem_source"])
# 'math'
```

cc @allisonwang-db this is a nice plus for usage in spark