rIsHu009/github-issues-updated huggingface dataset:
Title: Fix issue with case sensitivity when loading dataset from local cache
Author: Sumsky21 (NONE)
State: open
Created: 2024-03-28T14:52:35Z

When a dataset with upper-cases in its name is first loaded using `load_dataset()`, the local cache directory is created with all lowercase letters. 

However, upon subsequent loads, the current version attempts to locate the cache directory using the dataset's original name, which includes uppercase letters. This discrepancy can lead to confusion and, particularly in offline mode, results in errors. 

### Reproduce
```bash
~$ python            
Python 3.9.19 (main, Mar 21 2024, 17:11:28) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> dataset = load_dataset("locuslab/TOFU", "full")
>>> quit()

~$ export HF_DATASETS_OFFLINE=1 
~$ python                      
Python 3.9.19 (main, Mar 21 2024, 17:11:28) 
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_dataset
>>> dataset = load_dataset("locuslab/TOFU", "full")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "xxxxxx/anaconda3/envs/llm/lib/python3.9/site-packages/datasets/load.py", line 2556, in load_dataset
    builder_instance = load_dataset_builder(
  File "xxxxxx/anaconda3/envs/llm/lib/python3.9/site-packages/datasets/load.py", line 2228, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "xxxxxx/anaconda3/envs/llm/lib/python3.9/site-packages/datasets/load.py", line 1871, in dataset_module_factory
    raise ConnectionError(f"Couldn't reach the Hugging Face Hub for dataset '{path}': {e1}") from None
ConnectionError: Couldn't reach the Hugging Face Hub for dataset 'locuslab/TOFU': Offline mode is enabled.
>>> 
```

I fix this issue by lowering the dataset name (`.lower()`) when generating cache_dir.

Comments:
I also need this feature for ["Cnam-LMSSC/vibravox "](https://huggingface.co/datasets/Cnam-LMSSC/vibravox)


EDIT: Upgrading to `2.19.0` fixed my problem thanks to [this PR](https://github.com/huggingface/datasets/pull/6754)
Title: Persistent multi-process Pool
Author: fostiropoulos (NONE)
State: open
Created: 2024-03-26T17:35:25Z
Labels: enhancement

### Feature request

Running .map and filter functions with `num_procs` consecutively instantiates several multiprocessing pools iteratively. 

As instantiating a Pool is very resource intensive it can be a bottleneck to performing iteratively filtering. 

My ideas:
1. There should be an option to declare `persistent_workers` similar to pytorch DataLoader. Downside would be that would be complex to determine the correct resource allocation and deallocation of the pool. i.e. the dataset can outlive the utility of the pool. 
2. Provide a pool as an argument. Downside would be the expertise required by the user. Upside, is that there is better resource management.

### Motivation

Is really slow to iteratively perform map and filter operations on a dataset.

### Your contribution

If approved I could integrate it. I would need to know what method would be most suitable to implement from the two options above.

Comments:
Title: Passing `sample_by` to `load_dataset` when loading text data does not work
Author: ntoxeg (NONE)
State: closed
Created: 2024-03-26T14:55:33Z

### Describe the bug

I have a dataset that consists of a bunch of text files, each representing an example. There is an undocumented `sample_by` argument for the `TextConfig` class that is used by `Text` to decide whether to split files into lines, paragraphs or take them whole. Passing `sample_by=“document”` to `load_dataset` results in files getting split into lines regardless. I have edited `src/datasets/packaged_modules/text/text.py` for myself to switch the default and it works fine.

As a side note, the `if-else` for `sample_by` will silently load an empty dataset if someone makes a typo in the argument, which is not ideal.

### Steps to reproduce the bug

1. Prepare data as a bunch of files in a directory.
2. Load that data via `load_dataset(“text”, data_files=<data_dir>/<files_glob>, …, sample_by=“document”)`.
3. Inspect the resultant dataset — every item should have the form of `{“text”: <a line from a file>}`.

### Expected behavior

`load_dataset(“text”, data_files=<data_dir>/<files_glob>, …, sample_by=“document”)` should result in a dataset with items of the form `{“text”: <one document>}`.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.0-1046-nvidia-x86_64-with-glibc2.35
- Python version: 3.11.8
- `huggingface_hub` version: 0.21.4
- PyArrow version: 15.0.2
- Pandas version: 2.2.1
- `fsspec` version: 2024.2.0

Comments:
Thanks for reporting! We are working on a fix.
Title: Test disabling transformers containers in docs CI
Author: Wauplin (CONTRIBUTOR)
State: open
Created: 2024-03-25T17:16:11Z

Related to https://github.com/huggingface/doc-builder/pull/487 and [internal slack thread](https://huggingface.slack.com/archives/C04F8N7FQNL/p1711384899462349?thread_ts=1711041424.720769&cid=C04F8N7FQNL). There is now a `custom_container` option when building docs in CI. When set to `""` (instead of `"huggingface/transformers-doc-builder"` by default), we don't run the CI inside a container, therefore saving ~2min of download time. The plan is to test disabling the transformers container on a few "big" repo and if everything works correctly, we will stop making it the default container. More details on https://github.com/huggingface/doc-builder/pull/487.

cc @mishig25

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6757). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

On slack it was mentioned that it was actually slower for `datasets`, should we close this one or am I missing something ?

@lhoestq I converted to draft. Want to make some more tests and will let you know
Title: Support SQLite files?
Author: severo (COLLABORATOR)
State: closed
Created: 2024-03-25T11:48:05Z
Labels: enhancement

### Feature request

Support loading a dataset from a SQLite file

https://huggingface.co/datasets/severo/test_iris_sqlite/tree/main

### Motivation

SQLite is a popular file format.

### Your contribution

See discussion on slack: https://huggingface.slack.com/archives/C04L6P8KNQ5/p1702481859117909 (internal)

In particular: a SQLite file can contain multiple tables, which might be matched to multiple configs. Maybe the detail of splits and configs should be defined in the README YAML, or use the same format as for ZIP files: `Iris.sqlite::Iris`. 

See dataset here: https://huggingface.co/datasets/severo/test_iris_sqlite

Note: should we also support DuckDB files?

Comments:
You can use `Dataset.from_sql(path_to_sql_file)` already. Though we haven't added the Sql dataset builder to the `_PACKAGED_DATASETS_MODULES` list or in `_EXTENSION_TO_MODULE` to map `.sqlite` to the Sql dataset builder

This would allow to load a dataset repository with a `.sqlite` file using `load_dataset` and enable the Dataset Viewer

Considering `Dataset.from_sql`'s (extremely) low usage, I don't think many users are interested in using this format for their datasets. Also, SQLite files are hard/impossible to stream efficiently and require custom logic to define splits/subsets, so IMO we shouldn't encourage people to use SQLite on the Hub.

@severo Do you have some real-world examples of datasets published in this format?

No. Indeed, it seems better to explicitly not support sqlite
Title: Small typo on the documentation
Author: fostiropoulos (NONE)
State: closed
Created: 2024-03-24T21:47:52Z
Labels: good first issue

### Describe the bug

There is a small typo on https://github.com/huggingface/datasets/blob/d5468836fe94e8be1ae093397dd43d4a2503b926/src/datasets/dataset_dict.py#L938

It should be `caching is enabled`. 

### Steps to reproduce the bug

Please visit 

https://github.com/huggingface/datasets/blob/d5468836fe94e8be1ae093397dd43d4a2503b926/src/datasets/dataset_dict.py#L938


### Expected behavior

`caching is enabled`

### Environment info

- `datasets` version: 2.17.1
- Platform: Linux-5.15.0-101-generic-x86_64-with-glibc2.35
- Python version: 3.11.7
- `huggingface_hub` version: 0.20.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2023.10.0

Comments:
Thanks for reporting @fostiropoulos! I've edited your comment to fix the link to the problematic line.


@mariosasko can i take this up?

#self-assign
Title: Precision being changed from float16 to float32 unexpectedly
Author: gcervantes8 (NONE)
State: open
Created: 2024-03-23T20:53:56Z

### Describe the bug

I'm loading a HuggingFace Dataset for images. 

I'm running a preprocessing (map operation) step that runs a few operations, one of them being conversion to float16.  The Dataset features also say that the 'img' is of type float16.  Whenever I take an image from that HuggingFace Dataset instance, the type turns out to be float32.  



### Steps to reproduce the bug

```python
import torchvision.transforms.v2 as transforms
from datasets import load_dataset

dataset = load_dataset('cifar10', split='test')
dataset = dataset.with_format("torch")

data_transform = transforms.Compose([transforms.Resize((32, 32)), 
                                     transforms.ToDtype(torch.float16, scale=True),
                                     transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5]),
                                     ])
def _preprocess(examples):
    # Permutes from (BS x H x W x C) to (BS x C x H x W)
    images = torch.permute(examples['img'], (0, 3, 2, 1))
    examples['img'] = data_transform(images)
    return examples

dataset = dataset.map(_preprocess, batched=True, batch_size=8)
```

Now at this point the dataset.features are showing float16 which is great because that's what I want. 

```python
print(data_loader.features['img'])

Sequence(feature=Sequence(feature=Sequence(feature=Value(dtype='float16', id=None), length=-1, id=None), length=-1, id=None), length=-1, id=None)
```

But when I try to sample an image from this dataloader; I'm getting a float32 image, when I'm expecting float16:
```python
print(next(iter(data_loader))['img'].dtype)

torch.float32
```

### Expected behavior

I'm expecting the images loaded after the transformation to stay in float16.

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
- Python version: 3.10.9
- `huggingface_hub` version: 0.21.4
- PyArrow version: 14.0.2
- Pandas version: 2.0.3
- `fsspec` version: 2023.10.0

Comments:
This is because of the formatter (`torch` in this case).
It defaults to `float32`.

You can load it in `float16` using `dataset.set_format("torch", dtype=torch.float16)`.
Title: Use 'with' operator for some download functions
Author: Moisan (NONE)
State: closed
Created: 2024-03-23T16:32:08Z

Some functions in `streaming_download_manager.py` are not closing the file they open which lead to `Unclosed file` warnings in our code. This fixes a few of them.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6751). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I was mistaken on the intent of those functions, closing the PR.
Title: Strange slicing behavior
Author: Luciennnnnnn (NONE)
State: open
Created: 2024-03-22T01:49:13Z

### Describe the bug

I have loaded a dataset, and then slice first 300 samples using `:` ops, however, the resulting dataset is not expected, as the output below:

```bash
len(dataset)=1050324
len(dataset[:300])=2
len(dataset[0:300])=2
len(dataset.select(range(300)))=300
```

### Steps to reproduce the bug

load a dataset then:
```bash
    dataset = load_from_disk(args.train_data_dir)

    print(f"{len(dataset)=}", flush=True)
    print(f"{len(dataset[:300])=}", flush=True)
    print(f"{len(dataset[0:300])=}", flush=True)
    print(f"{len(dataset.select(range(300)))=}", flush=True)
```

### Expected behavior

```bash
len(dataset)=1050324
len(dataset[:300])=300
len(dataset[0:300])=300
len(dataset.select(range(300)))=300
```

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- `huggingface_hub` version: 0.20.2
- PyArrow version: 10.0.1
- Pandas version: 1.5.3
- `fsspec` version: 2023.10.0

Comments:
As explained in the [docs](https://huggingface.co/docs/datasets/v2.18.0/en/access#slicing), slicing a `Dataset` returns a dictionary that maps its column names to their values. So, `len(dataset[:300])=2` is expected, assuming your dataset has 2 columns (the returned dict has 2 keys, but each value in the dict has 300 items).
`
Title: Scraping the whole of github including private repos is bad; kindly stop
Author: ghost (NONE)
State: closed
Created: 2024-03-20T20:54:06Z
Labels: enhancement

### Feature request

https://github.com/bigcode-project/opt-out-v2 - opt out is not consent. kindly quit this ridiculous nonsense.

### Motivation

[EDITED: insults not tolerated]

### Your contribution

[EDITED: insults not tolerated]

Comments:
It's not twitter here
Title: Option to disable file locking
Author: VRehnberg (NONE)
State: open
Created: 2024-03-20T15:59:45Z
Labels: enhancement

### Feature request

Commands such as `load_dataset` creates file locks with `filelock.FileLock`. It would be good if there was a way to disable this.

### Motivation

File locking doesn't work on all file-systems (in my case NFS mounted Weka). If the `cache_dir` only had small files then it would be possible to point to local disk and the problem would be solved. However, as cache_dir is both where the small info files are written and the processed datasets are put this isn't a feasible solution.

Considering https://github.com/huggingface/datasets/issues/6395 I still do think this is something that belongs in HuggingFace. The possibility to control packages separately is valuable. It might be that a user has their dataset on a file-system that doesn't support file-locking while they are using file locking on local disk to control some other type of access.

### Your contribution

My suggested solution:
```
diff --git a/src/datasets/utils/_filelock.py b/src/datasets/utils/_filelock.py
index 19620e6e..58f41a02 100644
--- a/src/datasets/utils/_filelock.py
+++ b/src/datasets/utils/_filelock.py
@@ -18,11 +18,15 @@
 import os
 
 from filelock import FileLock as FileLock_
-from filelock import UnixFileLock
+from filelock import SoftFileLock, UnixFileLock
 from filelock import __version__ as _filelock_version
 from packaging import version
 
 
+if os.getenv('HF_USE_SOFTFILELOCK', 'false').lower() in ('true', '1'):
+    FileLock_ = SoftFileLock
+
+
 class FileLock(FileLock_):
     """
     A `filelock.FileLock` initializer that handles long paths.
```

Comments:
Title: Support for loading geotiff files as a part of the ImageFolder
Author: sunny1401 (NONE)
State: closed
Created: 2024-03-18T20:00:39Z
Labels: enhancement

### Feature request

Request for adding rasterio support to load geotiff as a part of ImageFolder, instead of using PIL

### Motivation

As of now, there are many datasets in HuggingFace Hub which are predominantly focussed towards RemoteSensing or are from RemoteSensing. The current ImageFolder (if I have understood correctly) uses PIL. This is not really optimized because mostly these datasets have images with many channels and additional metadata. Using PIL makes one loose it unless we provide a custom script. Hence, maybe an API could be added to have this in common?

### Your contribution

If the issue is accepted - i can contribute the code, because I would like to have it automated and generalised.

Comments:
Title: Dict feature is non-nullable while nested dict feature is
Author: polinaeterna (CONTRIBUTOR)
State: closed
Created: 2024-03-18T14:31:47Z
Labels: bug

When i try to create a `Dataset` object with None values inside a dict column, like this: 
```python
from datasets import Dataset, Features, Value

Dataset.from_dict(
    {
        "dict": [{"a": 0, "b": 0}, None],
    }, features=Features(
        {"dict": {"a": Value("int16"), "b": Value("int16")}}
    )
)
```
i get `ValueError: Got None but expected a dictionary instead`.


At the same time, having None in _nested_ dict feature works, for example, this doesn't throw any errors:

```python
from datasets import Dataset, Features, Value, Sequence

dataset = Dataset.from_dict(
    {
        "list_dict": [[{"a": 0, "b": 0}], None],
        "sequence_dict": [[{"a": 0, "b": 0}], None],
    }, features=Features({
        "list_dict": [{"a": Value("int16"), "b": Value("int16")}],
        "sequence_dict": Sequence({"a": Value("int16"), "b": Value("int16")}),
    })
)
```
Other types of features also seem to be nullable (but I haven't checked all of them).

Version of `datasets` is the latest atm (2.18.0)


Is this an expected behavior or a bug?

Comments:
It looks like a bug, by default every feature should be nullable.

I've linked a PR with a fix :)

@mariosasko awesome thank you!
Title: Mosaic Streaming (MDS) Support
Author: siddk (NONE)
State: open
Created: 2024-03-16T18:42:04Z
Labels: enhancement

### Feature request

I'm a huge fan of the current HF Datasets `webdataset` integration (especially the built-in streaming support). However, I'd love to upload some robotics and multimodal datasets I've processed for use with [Mosaic Streaming](https://docs.mosaicml.com/projects/streaming/en/stable/), specifically their [MDS Format](https://docs.mosaicml.com/projects/streaming/en/stable/fundamentals/dataset_format.html#mds). 

Because the shard files have similar semantics to WebDataset, I'm hoping that adding such support won't be too much trouble?

### Motivation

One of the downsides with WebDataset is a lack of out-of-the-box determinism (especially for large-scale training and reproducibility), easy job resumption, and the ability to quickly debug / visualize individual examples. 

Mosaic Streaming provides a [great interface for this out of the box](https://docs.mosaicml.com/projects/streaming/en/stable/#key-features), so I'd love to see it supported in HF Datasets.

### Your contribution

Happy to help test things / provide example data. Can potentially submit a PR if maintainers could point me to the necessary WebDataset logic / steps for adding a new streaming format!

Comments:
Hi ! that would be great :) Though note that `datasets` doesn't implement format-specific resuming when streaming, so in general I think it's better if users can use the mosaic-streaming library to read their MDS datasets. I wonder if they support `hf://` paths though...

Anyway for those interested, the code for WebDataset is a single file here: https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/webdataset/webdataset.py.

It implements `_split_generators` that downloads files and returns the lists of splits (train/validation/test) and `_split_generators` to generate examples (dicts) from the downloaded files. Streaming is automatically supported by making download steps lazy and by extending `open()` to work with remote URLs.
Title: EmptyDatasetError when loading dataset downloaded with HuggingFace cli
Author: StwayneXG (NONE)
State: open
Created: 2024-03-14T16:41:27Z

### Describe the bug

I am using a cluster that does not have access to the internet when given a job. I tried downloading the dataset using the huggingface-cli command and then loading it with load_dataset but I get an error:

```raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None```

The dataset I'm using is "lmsys/chatbot_arena_conversations". The folder structure is

- README.md
- data
  - train-00000-of-00001-cced8514c7ed782a.parquet



### Steps to reproduce the bug

1. Download dataset using HuggingFace CLI: ```huggingface-cli download lmsys/chatbot_arena_conversations --local-dir ./lmsys/chatbot_arena_conversations```
2. In Python
```
from datasets import load_dataset
load_dataset("lmsys/chatbot_arena_conversations")
```

### Expected behavior

Should return a Dataset Dict in the form of
```
DatasetDict({
    train: Dataset({
        features: [...],
        num_rows: 33,000
    })
})
```
### Environment info

Python 3.11.5
Datasets 2.18.0
Transformers 4.38.2
Pytorch 2.2.0
Pyarrow 15.0.1
Rocky Linux release 8.9 (Green Obsidian)

Comments:
Hi! `datasets` is not compatible with `huggingface_hub`'s cache structure, hence the error.

You can track https://github.com/huggingface/datasets/issues/5080 to get notified when this is implemented.
Title: Unexpected behavior when using load_dataset with streaming=True in a for loop
Author: uApiv (NONE)
State: closed
Created: 2024-03-12T23:26:43Z

### Describe the bug

### My Code
```
from datasets import load_dataset
res=[]
for i in [0,1]:
    di=load_dataset(
            "json", 
            data_files='path_to.json', 
            split='train',
            streaming=True, 
        ).map(lambda x: {"source": i})

    res.append(di)

for e in res[0]:
    print(e)
```

### Unexpected Behavior
Data in `res[0]` has `source=1`. However the expected value is 0.

### FYI
I further switch `streaming` to `False`. And the output value is as expected (0). So there may exist bugs in setting `streaming=True` in a for loop.

### Environment
Python 3.8.0
datasets==2.18.0
transformers==4.28.1

### Steps to reproduce the bug

1. Create a Json file with any content.
2. Run the provided code.
3. Switch `streaming` to `False` and run again to see the expected behavior.

### Expected behavior

The expected behavior is the data are mapped with its corresponding value in the for loop.

### Environment info

Python 3.8.0

datasets==2.18.0
transformers==4.28.1

Ubuntu 20.04

Comments:
This is normal behavior in python when using `lambda`: the `i` defined in your `lambda` refers to the global variable `i` in your loop, and `i` equals to `1` when you run your `for e in res[0]` line.

You should pass `fn_kwargs` that will be passed to your `lambda` instead of using the global variable:

```python
from datasets import load_dataset

res=[]
for i in [0,1]:
    di = load_dataset(
            "json", 
            data_files='path_to.json', 
            split='train',
            streaming=True, 
    ).map(lambda x, source: {"source": source}, fn_kwargs={"source": i})

    res.append(di)

for e in res[0]:
    print(e)
```

This doesn't happen in non-streaming since in that case `map` is executed while the variable `i` has the right value. In streaming mode, `map` is executed on-the-fly when you iterate on the dataset.

Thank you very much for your answer. I think this issue can be closed now.
Title: Request for a comparison of huggingface datasets compared with other data format especially webdataset
Author: Luciennnnnnn (NONE)
State: open
Created: 2024-03-08T08:23:01Z
Labels: enhancement

### Feature request

Request for a comparison of huggingface datasets compared with other data format especially webdataset

### Motivation

I see huggingface datasets uses Apache Arrow as its backend, it seems to be great, but I'm curious about how it is good compared with other dataset format, like webdataset, what's the pros/cons of them.

### Your contribution

More information

Comments:
Title: Dataset with loading script does not work in renamed repos
Author: BramVanroy (CONTRIBUTOR)
State: open
Created: 2024-03-07T17:38:38Z

### Describe the bug

My data repository was first called `BramVanroy/hplt-mono-v1-2` but I then renamed to use underscores instead of dashes. However, it seems that `datasets` retrieves the old repo name when it checks whether the repo contains data loading scripts in this line.

https://github.com/huggingface/datasets/blob/6fb6c834f008996c994b0a86c3808d0a33d44525/src/datasets/load.py#L1845

When I print `filename` it returns `hplt-mono-v1-2.py` but the files in the repo are of course `['.gitattributes', 'README.md', 'hplt_mono_v1_2.py']`. So the `filename` is the original reponame instead of the renamed one.

I am not sure if this is a caching issue or not or how I can resolve it.

### Steps to reproduce the bug

```
from datasets import load_dataset


ds = load_dataset(
    "BramVanroy/hplt-mono-v1-2",
    "ky",
    trust_remote_code=True
)
```

### Expected behavior

That the most recent repo name is used when `filename` is generated.

### Environment info

- `datasets` version: 2.16.1
- Platform: Linux-5.14.0-284.25.1.el9_2.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.1
- Pandas version: 2.1.3
- `fsspec` version: 2023.10.0

Comments:
Title: Add details in docstring
Author: severo (COLLABORATOR)
State: closed
Created: 2024-03-07T17:02:07Z

see https://github.com/huggingface/datasets-server/pull/2554#discussion_r1516516867

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_6722). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Title: Hi,do you know how to load the dataset from local file now?
Author: Gera001 (NONE)
State: open
Created: 2024-03-07T13:58:40Z

Hi, if I want to load the dataset from local file, then how to specify the configuration name?

_Originally posted by @WHU-gentle in https://github.com/huggingface/datasets/issues/2976#issuecomment-1333455222_

Comments:

@Gera001
# Loading Dataset from Local Files Using 🤗Hugging Face.

To load a dataset from local files using the Hugging Face datasets library, you can use the `load_dataset` function.

```
from datasets import load_dataset
dataset = load_dataset('csv', data_files={'train': 'path/to/train.csv',
                                         'test': 'path/to/test.csv'})
```

Reference to [HF Datasets docs for loading from local](https://huggingface.co/docs/datasets/en/loading#csv).       

@albertvillanova
this issue can be closed here.

like this: from datasets import load_from_disk
dataset = load_from_disk(data_path)


@ge00009 
> like this: from datasets import load_from_disk dataset = load_from_disk(data_path)

Loads a dataset that was previously saved using `save_to_disk()`.

Reference link:
https://huggingface.co/docs/datasets/en/package_reference/loading_methods#datasets.load_from_disk.example
Title: Is there any way to solve hanging of IterableDataset using split by node + filtering during inference
Author: ssharpe42 (NONE)
State: open
Created: 2024-03-05T15:55:13Z

### Describe the bug

I am using an iterable dataset in a multi-node setup, trying to do training/inference while filtering the data on the fly. I usually do not use `split_dataset_by_node` but it is very slow using the IterableDatasetShard in `accelerate` and `transformers`. When I filter after applying `split_dataset_by_node`, it results in shards that are not equal sizes due to unequal samples filtered from each one. 

The distributed process hangs when trying to accomplish this. Is there any way to resolve this or is it impossible to implement?



### Steps to reproduce the bug

Here is a toy example of what I am trying to do that reproduces the behavior

```
# torchrun --nproc-per-node 2 file.py


import os

import pandas as pd
import torch
from accelerate import Accelerator
from datasets import Features, Value, load_dataset
from datasets.distributed import split_dataset_by_node
from torch.utils.data import DataLoader

accelerator = Accelerator(device_placement=True, dispatch_batches=False)
if accelerator.is_main_process:
    if not os.path.exists("scratch_data"):
        os.mkdir("scratch_data")

    n_shards = 4
    for i in range(n_shards):
        df = pd.DataFrame({"id": list(range(10 * i, 10 * (i + 1)))})
        df.to_parquet(f"scratch_data/shard_{i}.parquet")


world_size = accelerator.num_processes
local_rank = accelerator.process_index


def collate_fn(examples):
    input_ids = []
    for example in examples:
        input_ids.append(example["id"])
    return torch.LongTensor(input_ids)


dataset = load_dataset(
    "parquet", data_dir="scratch_data", split="train", streaming=True
)
dataset = (
    split_dataset_by_node(dataset, rank=local_rank, world_size=world_size)
    .filter(lambda x: x["id"] < 35)
    .shuffle(seed=42, buffer_size=100)
)

batch_size = 2
train_dataloader = DataLoader(
    dataset,
    batch_size=batch_size,
    collate_fn=collate_fn,
    num_workers=2
)  

for x in train_dataloader:
    x = x.to(accelerator.device)
    print({"rank": local_rank, "id": x})
   
    y = accelerator.gather_for_metrics(x)
    if accelerator.is_main_process:
        print("gathered", y)


```

### Expected behavior

Is there any way to continue training/inference on the GPUs that have remaining data left without waiting for the others? Is it impossible to filter when 

### Environment info

- `datasets` version: 2.18.0
- Platform: Linux-5.10.209-198.812.amzn2.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.13
- `huggingface_hub` version: 0.21.3
- PyArrow version: 15.0.0
- Pandas version: 2.2.1
- `fsspec` version: 2023.6.0

Comments:
Title: Non-deterministic `Dataset.builder_name` value
Author: harupy (NONE)
State: closed
Created: 2024-03-05T09:23:21Z

### Describe the bug

I'm not sure if this is a bug, but `print(ds.builder_name)` in the following code sometimes prints out `rotten_tomatoes` instead of `parquet`:

```python
import datasets

for _ in range(100):
    ds = datasets.load_dataset("rotten_tomatoes", split="train")
    print(ds.builder_name)  # prints out "rotten_tomatoes" sometimes instead of "parquet"
```

Output:

```
...
parquet
parquet
parquet
rotten_tomatoes
parquet
parquet
parquet
...
```

Here's a reproduction using GitHub Actions:
https://github.com/mlflow/mlflow/actions/runs/8153247984/job/22284263613?pr=11329#step:12:241

One of our tests is flaky because `builder_name` is not deterministic.

### Steps to reproduce the bug

1. Run the code above.

### Expected behavior

Always prints out `parquet`?

### Environment info

```
Copy-and-paste the text below in your GitHub issue.
- `datasets` version: 2.18.0
- Platform: Linux-6.5.0-1015-azure-x86_64-with-glibc2.34
- Python version: 3.8.18
- `huggingface_hub` version: 0.21.3
- PyArrow version: 15.0.0
- Pandas version: 2.0.3
- `fsspec` version: 2024.2.0
```

Comments:
When `rotten_tomatoes` is printed out, the following warning message is also printed out:

```
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
```

Hi ! This behavior happens because the dataset was originakky created using a dataset script [rotten_tomatoes.py](https://huggingface.co/datasets/rotten_tomatoes/blob/26f40d324d7b281d8b3fb1c47f30f8b9957f206b/rotten_tomatoes.py) and because we added features recently allowing to download the dataset directly from Parquet files (parquet builder) without running the dataset script (rotten_tomatoes). The flakiness must come from the availability of the Parquet files (we automatically export them in the refs/convert/parquet branch and we recently had to move some files).

Anyway the easy fix on our side is to remove the dataset script completely, let me open a PR at https://huggingface.co/datasets/rotten_tomatoes

EDIT: opened https://huggingface.co/datasets/rotten_tomatoes/discussions/6, feel free to comment there if you're ok with that change

@lhoestq Thanks for the comment, explanation, and patch!

> we automatically export them in the refs/convert/parquet branch

When this operation is in progress, the parquet files become temporarily unavailable?

> When this operation is in progress, the parquet files become temporarily unavailable?

Yes correct. I just merged the patch btw :)

@lhoestq Thanks for merging the PR! I think this issue can be closed.
Title: Push samples to dataset on hub without having the dataset locally
Author: jbdel (NONE)
State: closed
Created: 2024-02-29T19:17:12Z
Labels: enhancement

### Feature request

Say I have the following code:

```
from datasets import Dataset
import pandas as pd
new_data = {
    "column_1": ["value1", "value2"],
    "column_2": ["value3", "value4"],
}
df_new = pd.DataFrame(new_data)
dataset_new = Dataset.from_pandas(df_new)
# add these samples to a remote dataset
```
It would be great to have a way to push dataset_new to a remote dataset that respects the same schema. This way one would not have to do the following:
```
from datasets import load_dataset
dataset = load_dataset('username/dataset_name', use_auth_token='your_hf_token_here')
updated_dataset = dataset['train'].concatenate(dataset_new)
updated_dataset.push_to_hub('username/dataset_name', use_auth_token='your_hf_token_here')
```





### Motivation

No need to download the dataset.

### Your contribution

Maybe this feature already exists, didnt see it though. I do not have the expertise to do this.

Comments:
Hi ! For now I would recommend creating a new Parquet file using `dataset_new.to_parquet()` and upload it to HF using `huggingface_hub` every time you get a new batch of data. You can name the Parquet files `0000.parquet`, `0001.parquet`, etc.

Though maybe make sure to not upload one file per sample since that would be inefficient. You can buffer your data and upload when you have enough new samples for example

This is excellent, thanks!
Title: remove_columns is not in-place but the doc shows it is in-place
Author: shelfofclub (NONE)
State: closed
Created: 2024-02-28T12:36:22Z

### Describe the bug

The doc of `datasets` v2.17.0/v2.17.1 shows that `remove_columns` is in-place. [link](https://huggingface.co/docs/datasets/v2.17.1/en/package_reference/main_classes#datasets.DatasetDict.remove_columns)

In the text classification example of transformers v4.38.1, the columns are not removed.
https://github.com/huggingface/transformers/blob/a0857740c0e6127485c11476650314df3accc2b6/examples/pytorch/text-classification/run_classification.py#L421

### Steps to reproduce the bug

https://github.com/huggingface/transformers/blob/a0857740c0e6127485c11476650314df3accc2b6/examples/pytorch/text-classification/run_classification.py#L421

### Expected behavior

Actually remove the columns.

### Environment info

1. datasets v2.17.0
2. transformers v4.38.1

Comments:
Good catch! I've opened a PR with a fix in the `transformers` repo.

@mariosasko Thanks!

Will the doc of `datasets` be updated?

I find some possible mistakes in doc about whether `remove_columns` is in-place.
1. [You can also remove a column using map() with remove_columns but the present method is in-place (doesn’t copy the data to a new dataset) and is thus faster.](https://huggingface.co/docs/datasets/v2.17.1/en/package_reference/main_classes#datasets.Dataset.remove_columns)
2. [You can also remove a column using Dataset.map() with remove_columns but the present method is in-place (doesn’t copy the data to a new dataset) and is thus faster.](https://huggingface.co/docs/datasets/v2.17.1/en/package_reference/main_classes#datasets.DatasetDict.remove_columns)
3. [🤗 Datasets also has a remove_columns() function which is faster because it doesn’t copy the data of the remaining columns.](https://huggingface.co/docs/datasets/v2.17.1/en/process#map)

I've linked a PR that will fix the usage in the `datasets` docs.
Title: Support JSON file with an array of strings
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-02-26T12:35:11Z
Labels: enhancement

Support loading a dataset from a JSON file with an array of strings.

See: https://huggingface.co/datasets/CausalLM/Refined-Anime-Text/discussions/1

Comments:
https://huggingface.co/datasets/CausalLM/Refined-Anime-Text/discussions/1 has been fixed, but how can we check if there are other datasets with the same error, in datasets-server's database? I don't know how to get the list of erroneous cache entries, since we only copied `Error code:   JobManagerCrashedError`, but not the traceback in `details`... Do you remember the error message, or the underlying exception, we had?
Title: __add__ for Dataset, IterableDataset
Author: oh-gnues-iohc (NONE)
State: open
Created: 2024-02-26T01:46:55Z

It's too cumbersome to write this command every time we perform a dataset merging operation. ```pythonfrom datasets import concatenate_datasets``` We have added a simple `__add__` magic method to each class using `concatenate_datasets.`

```python
from datasets import load_dataset

bookcorpus = load_dataset("bookcorpus", split="train")
wiki = load_dataset("wikimedia/wikipedia", "20231101.ab", split="train")
wiki = wiki.remove_columns([col for col in wiki.column_names if col != "text"])  # only keep the 'text' column

bookcorpus + wiki

#Dataset({
#   features: ['text'],
#   num_rows: 74004228
#})
#Dataset({
#    features: ['text'],
#    num_rows: 6152
#})
#Dataset({
#   features: ['text'],
#    num_rows: 74010380
#})
```

Comments:
Hi! You can find a reason why we are against this feature in https://github.com/huggingface/datasets/issues/3449. 

> It's too cumbersome to write this command every time we perform a dataset merging operation

Explicit is better than implicit, so this isn't a good enough reason. 

Thanks for the effort nonetheless :)!
Title: Enhancement: Enable loading TSV files in load_dataset()
Author: harsh1504660 (NONE)
State: closed
Created: 2024-02-24T11:38:59Z

Fix #6691

Comments:
Hi @harsh1504660,

Thanks for your work, but this functionality already exists. See my comment in the corresponding issue: https://github.com/huggingface/datasets/issues/6691#issuecomment-1963449923

Next time you would like to contribute, I would suggest you take on an issue that is previously validated by one of the maintainers. Thanks anyway.
Title: load_dataset() does not support tsv
Author: dipsivenkatesh (NONE)
State: closed
Created: 2024-02-24T05:56:04Z
Labels: enhancement

### Feature request

the load_dataset() for local functions support file types like csv, json etc but not of type tsv (tab separated values).

### Motivation

cant easily load files of type tsv, have to convert them to another type like csv then load

### Your contribution

Can try by raising a PR with a little help, currently went through the code but didn't fully understand

Comments:
#self-assign

Hi @dipsivenkatesh,

Please note that this functionality is already implemented. Our CSV builder uses `pandas.read_csv` under the hood, and you can pass the parameter `delimiter="\t"` to read TSV files.

See the list of CSV config parameters in our docs: https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.packaged_modules.csv.CsvConfig
Title: Question: Is there any way for uploading a large image dataset?
Author: zhjohnchan (NONE)
State: open
Created: 2024-02-21T22:07:21Z

I am uploading an image dataset like this:
```
dataset = load_dataset(
    "json",
    data_files={"train": "data/custom_dataset/train.json", "validation": "data/custom_dataset/val.json"},
)
dataset = dataset.cast_column("images", Sequence(Image()))
dataset.push_to_hub("StanfordAIMI/custom_dataset", max_shard_size="1GB")
```
where it takes a long time in the `Map` process. Do you think I can use multi-processing to map all the image data to the memory first? For the `Map()` function, I can set `num_proc`. But for `push_to_hub` and `cast_column`, I can not find it.

Thanks in advance!

Best,

Comments:
```
import pandas as pd
from datasets import Dataset, Image

# Read the CSV file
data = pd.read_csv("XXXX.csv")

# Create a Hugging Face Dataset
dataset = Dataset.from_pandas(data)
dataset = dataset.cast_column("file_name", Image())

# Upload to Hugging Face Hub (make sure authentication is set up)
dataset.push_to_hub("XXXXX"")
```

stuck in "Casting the dataset
![截屏2024-05-02 11 44 50](https://github.com/huggingface/datasets/assets/48406770/dc012dc5-16f6-4fd5-9e02-1b705c552c5b)
"
Title: Node.js 16 GitHub Actions are deprecated
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-02-19T09:47:37Z
Labels: maintenance

`Node.js` 16 GitHub Actions are deprecated. See: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/

We should update them to Node 20.

See warnings in our CI, e.g.: https://github.com/huggingface/datasets/actions/runs/7957295009?pr=6678
> Node.js 16 actions are deprecated. Please update the following actions to use Node.js 20: actions/checkout@v3, actions/setup-python@v4. For more information see: https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/.

Comments:
Title: Can't Read List of JSON Files Properly
Author: lordsoffallen (NONE)
State: open
Created: 2024-02-17T22:58:15Z

### Describe the bug

Trying to read a bunch of JSON files into Dataset class but default approach doesn't work. I don't get why it works when I read it one by one but not when I pass as a list :man_shrugging: 

The code fails with 

```
ArrowInvalid: JSON parse error: Invalid value. in row 0  
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte 
DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

This doesn't work

```
from datasets import Dataset

# dir contains 100 json files.
Dataset.from_json("/PUT SOME PATH HERE/*")
```

This works:
```
from datasets import concatenate_datasets

ls_ds = []

for file in list_of_json_files:
    ls_ds.append(Dataset.from_json(file))

ds = concatenate_datasets(ls_ds)

```





### Expected behavior

I expect this to read json files properly as error is not clear

### Environment info

- `datasets` version: 2.17.0
- Platform: Linux-6.5.0-15-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 15.0.0
- Pandas version: 2.2.0
- `fsspec` version: 2023.10.0

Comments:
Found the issue, if there are other files in the directory, it gets caught into this `*` so essentially it should be `*.json`. Could we possibly to check for list of files to make sure the pattern matches json files and raise error if not?

I don't think we should filter for `*.json` as this might silently remove desired files for many users. And this could be a major breaking change for many organizations.

You could do the globbing yourself which would keep the code clean.

```python
from glob import glob

Dataset.from_json(glob('folder/*.json'))
```

I think it should still be fine to log a warning message in case the folder contains different files? I also don't get why would this be breaking as in the end using `from_FILE_TYPE` should be able to read a specific file type only. Maybe some other use case I am not aware of but since globbing or this case not mentioned anywhere in the doc, I spent quite a bit of time trying to figure out where the issue was. Just making sure it's clear for users.
Title: Allow image model (color conversion) to be specified as part of datasets Image() decode
Author: rwightman (NONE)
State: closed
Created: 2024-02-16T23:43:20Z
Labels: enhancement

### Feature request

Typical torchvision / torch Datasets in image applications apply color conversion in the Dataset portion of the code as part of image decode, separately from the image transform stack. This is true for PIL.Image where convert is usually called in dataset, for native torchvision https://pytorch.org/vision/main/generated/torchvision.io.decode_jpeg.html, and similarly in tensorflow.data pipelines decode_jpeg or https://www.tensorflow.org/api_docs/python/tf/io/decode_and_crop_jpeg have a channels arg that allows controlling the image mode in the decode step.

datasets currently requires this pattern (from [examples](https://huggingface.co/docs/datasets/main/en/image_process)):
```
from torchvision.transforms import Compose, ColorJitter, ToTensor

jitter = Compose(
    [
         ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.7),
         ToTensor(),
    ]
)

def transforms(examples):
    examples["pixel_values"] = [jitter(image.convert("RGB")) for image in examples["image"]]
    return examples
```




### Motivation

It would be nice to be able to handle `image.convert("RGB")` (or other modes) in the decode step, before applying torchvision transforms, this would reduce differences in code when handling pipelines that can handle torchvision, webdatset, or hf datasets with fewer code differences and without needing to handle image mode argument passing in two different stages of the pipelines...

### Your contribution

Can do a PR with guidance on how mode should be passed / set on the dataset.

Comments:
It would be a great addition indeed :)

This can be implemented the same way we have `sampling_rate` for Audio(): we just add a new parameter to the Image() type and take this parameter into account in `Image.decode_example`

EDIT: adding an example of how it can be used:

```python
ds = ds.cast_column("image", Image(mode=...))
```
Title:  Depprcated Overview.ipynb Link to new Quickstart Notebook invalid
Author: Codeblockz (CONTRIBUTOR)
State: closed
Created: 2024-02-16T22:51:35Z

### Describe the bug

For the dreprecated notebook found  [here](https://github.com/huggingface/datasets/blob/main/notebooks/Overview.ipynb). The link to the new notebook is broken. 

### Steps to reproduce the bug

Click the [Quickstart notebook](https://github.com/huggingface/notebooks/blob/main/datasets_doc/quickstart.ipynb) link in the notebook.

### Expected behavior

I believe is it suposed to link [here](https://github.com/huggingface/notebooks/blob/main/datasets_doc/en/quickstart.ipynb). That is mentioned in the readme.

### Environment info

Colab

Comments:
Good catch! Feel free to open a PR to fix the link.
Title: IterableDataset `set_epoch` is ignored when DataLoader `persistent_workers=True`
Author: rwightman (NONE)
State: closed
Created: 2024-02-16T21:38:12Z
Labels: bug

streaming

### Describe the bug

When persistent workers are enabled, the epoch that's set via the IterableDataset instance held by the training process is ignored by the workers as they are disconnected across processes.

PyTorch samplers for non-iterable datasets have a mechanism to sync this, datasets.IterableDataset does not.

In my own use of IterableDatasets I usually track the epoch count which crosses process boundaries in a multiprocessing.Value 

### Steps to reproduce the bug

Use a streaming dataset (Iterable) w/ the recommended pattern below and `persistent_workers=True` in the torch DataLoader. 

```
for epoch in range(epochs):
    shuffled_dataset.set_epoch(epoch)
    for example in shuffled_dataset:
        ...
```

### Expected behavior

When the canonical bit of code above is used with `num_workers > 0` and `persistent_workers=True`, the epoch set via `set_epoch()` is propagated to the IterableDataset instances in the worker processes


### Environment info

N/A

Comments:
Title: CSV builder raises deprecation warning on verbose parameter
Author: albertvillanova (MEMBER)
State: closed
Created: 2024-02-16T14:23:46Z

CSV builder raises a deprecation warning on `verbose` parameter:
```
FutureWarning: The 'verbose' keyword in pd.read_csv is deprecated and will be removed in a future version.
```

See:
- https://github.com/pandas-dev/pandas/pull/56556
- https://github.com/pandas-dev/pandas/pull/57450

Comments: