sunny2309/githug-issues huggingface dataset:
Tokenization with multiple workers is too slow

I am trying to tokenize a few million documents with multiple workers but the tokenization process is taking forever. 

Code snippet:

```
raw_datasets.map(
            encode_function,
            batched=False,
            num_proc=args.preprocessing_num_workers,
            load_from_cache_file=not args.overwrite_cache,
            remove_columns=[name for name in raw_datasets["train"].column_names if name not in ["input_ids", "labels", "attention_mask"]],
            desc="Tokenizing data",
        )
```

Details:

```
transformers==4.28.0.dev0
datasets==4.28.0.dev0
preprocessing_num_workers==48
```
tokenizer == decapoda-research/llama-7b-hf
Support buffer shuffle for to_tf_dataset

### Feature request

I'm using to_tf_dataset to convert a large dataset to tf.data.Dataset and use Keras fit to train model. 
Currently, to_tf_dataset only supports full size shuffle, which can be very slow on large dataset. 
tf.data.Dataset support buffer shuffle by default. 
shuffle(
    buffer_size, seed=None, reshuffle_each_iteration=None, name=None
)

### Motivation

I'm very frustrated to find the loading with shuffling large dataset is very slow. It seems impossible to shuffle before training Keras with big dataset. 

### Your contribution

NA
Overwrite legacy default config name in `dataset_infos.json` in packaged datasets

Currently if we push data as default config with `.push_to_hub` to a repo that has a legacy `dataset_infos.json` file containing a legacy default config name like `{username}--{dataset_name}`, new key `"default"` is added to `dataset_infos.json` along with the legacy one. I think the legacy one should be dropped in this case.

Also, in `load.py` I suggest to check if a legacy config name is indeed a legacy config name because after this fix it might not be the case (this check was first introduced in https://github.com/huggingface/datasets/pull/6218)
Conversion from RGB to BGR in Object Detection tutorial

The [tutorial](https://huggingface.co/docs/datasets/main/en/object_detection) mentions the necessity of conversion the input image from BGR to RGB

> albumentations expects the image to be in BGR format, not RGB, so you’ll have to convert the image before applying the transform.

[Link to tutorial](https://github.com/huggingface/datasets/blob/0a068dbf3b446417ffd89d32857608394ec699e6/docs/source/object_detection.mdx#L77)

However, relevant albumentations' tutorials [on channels conversion](https://albumentations.ai/docs/examples/example/#read-the-image-from-the-disk-and-convert-it-from-the-bgr-color-space-to-the-rgb-color-space) and  [on boxes](https://albumentations.ai/docs/examples/example_bboxes/) imply that it's not really true no more.

I suggest removing this outdated conversion from the tutorial.
Ignore `dataset_info.json` in data files resolution

`save_to_disk` creates this file, but also [`HugginFaceDatasetSever`](https://github.com/gradio-app/gradio/blob/26fef8c7f85a006c7e25cdbed1792df19c512d02/gradio/flagging.py#L214), so this is needed to avoid issues such as [this one](https://discord.com/channels/879548962464493619/1149295819938349107/1149295819938349107).
Support saving datasets with custom formatting

Requested in https://discuss.huggingface.co/t/using-set-transform-on-a-dataset-leads-to-an-exception/53036.

I am not sure if supporting this is the best idea for the following reasons:

>For this to work, we would have to pickle a custom transform, which means the transform and the objects it references need to be serializable. Also, deserializing these bytes would make `load_from_disk` unsafe, so I'm not sure this is a good idea. 

@lhoestq WDYT?
Rename old push_to_hub configs to "default" in dataset_infos

Fix

```python
from datasets import load_dataset_builder

b = load_dataset_builder("lambdalabs/pokemon-blip-captions", "default")
print(b.info)
```

which should return

```
DatasetInfo(
  features={'image': Image(decode=True, id=None), 'text': Value(dtype='string', id=None)}, 
  dataset_name='pokemon-blip-captions',
  config_name='default', 
  version=0.0.0,
  splits={'train': SplitInfo(name='train', num_bytes=119417410.0, num_examples=833, shard_lengths=None, dataset_name='pokemon-blip-captions')},
  download_checksums=None,
  download_size=99672355,
  dataset_size=119417410.0,
  size_in_bytes=219089765.0,
  ...
)
```

instead of and empty dataset info.

The dataset has a dataset_infos.json file with a deprecated config name "lambdalabs--pokemon-blip-captions". We switched those config names to "default" in 2.14, so the builder.info should take this into account.
`Dataset.to_dict()` ignore `decode=True` with Image feature

### Describe the bug

`Dataset.to_dict` seems to ignore the decoding instruction passed in features.

### Steps to reproduce the bug

```python
import datasets
import numpy as np
from PIL import Image

img = np.random.randint(0, 256, (5, 5, 3), dtype=np.uint8)
img = Image.fromarray(img)

features = datasets.Features({"image": datasets.Image(decode=True)})
dataset = datasets.Dataset.from_dict({"image": [img]}, features=features)
print({key: dataset[key] for key in dataset.column_names})
# {'image': [<PIL.PngImagePlugin.PngImageFile image mode=RGB size=5x5 at 0x7EFBC80E15B0>]}
print(dataset.to_dict())
# {'image': [{'bytes': b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x00\x05\x00\x00\x00\x05\x08\x02\x00\x00\x00\x02\r\xb1\xb2\x00\x00\x00[IDATx\x9c\x01P\x00\xaf\xff\x01\x13\x1b<7\xe7\xe0\xdc^6\xed\x04\xc7M\xd2\x9f\x00X\x1b\xb0?\x1ba\x15\xc5 o\xd0\x80\xbe\x19/\x01\xec\x95\x1f\x9f\xffj\xfa1\xa7\xc4X\xea\xbe\xa4g\x00\xc4\x15\xdeC\xc7 \xbbaqe\xc8\xb9\xa9q\xe7\x00,?M\xc0)\xdaD`}\xb1\xdci\x1e\xafC\xa9]%.@\xa6\xf0\xb3\x00\x00\x00\x00IEND\xaeB`\x82', 'path': None}]}
```




### Expected behavior

I would expect `{key: dataset[key] for key in dataset.column_names}` and `dataset.to_dict()` to be equivalent. If the previous behavior is expected, then it should be stated [in the doc](https://huggingface.co/docs/datasets/v2.14.4/en/package_reference/main_classes#datasets.Dataset.to_dict).

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-6.2.0-31-generic-x86_64-with-glibc2.35
- Python version: 3.9.12
- Huggingface_hub version: 0.15.1
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
- Pillow 9.5.0
- numpy 1.25.2
Tilde (~) is not supported for data_files

### Describe the bug

Attempting to `load_dataset` from a path starting with `~` (as a shorthand for the user's home directory) seems not to be fully working - at least as far as the `parquet` dataset builder is concerned.

(the same file can be loaded correctly if providing its absolute path instead)

I think that this is very similar to https://github.com/huggingface/datasets/issues/5757, but for `data_files` rather than `data_dir`

### Steps to reproduce the bug

```python
import datasets

# save a parquet file at ~/path/to/data.parquet

data_files = "~/path/to/data.parquet"
dataset = datasets.load_dataset("parquet", data_files=data_files)
```
```
Downloading data files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 12671.61it/s]
Extracting data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 22671.91it/s]
Generating train split: 0 examples [00:00, ? examples/s]
Traceback (most recent call last):
  File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1949, in _prepare_split_single
    num_examples, num_bytes = writer.finalize()
                              ^^^^^^^^^^^^^^^^^
  File ".venv/lib/python3.11/site-packages/datasets/arrow_writer.py", line 598, in finalize
    raise SchemaInferenceError("Please pass `features` or at least one example when writing data")
datasets.arrow_writer.SchemaInferenceError: Please pass `features` or at least one example when writing data

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File ".venv/lib/python3.11/site-packages/datasets/load.py", line 2133, in load_dataset
    builder_instance.download_and_prepare(
  File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 954, in download_and_prepare
    self._download_and_prepare(
  File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1049, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1813, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File ".venv/lib/python3.11/site-packages/datasets/builder.py", line 1958, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset

```

### Expected behavior

Can use `~` shorthand in paths when loading local (parquet) datasets.

### Environment info

`datasets                               2.14.3`
Support loading from a DVC remote repository

### Feature request

Adding support for loading a file from a DVC repository, tracked remotely on a SCM.

### Motivation

DVC is a popular version control system to version and manage datasets. The files are stored on a remote object storage platform, but they are tracked using Git. Integration with DVC is possible through the `DVCFileSystem`.

I have a Gitlab repository where multiple files are tracked using DVC and stored in a GCP bucket. I would like to be able to load these files using `datasets` directly using an URL. My goal is to write a generic code that abstracts the storage layer, such that my users will only have to pass in an `fsspec`-compliant URL and the corresponding files will be loaded.

### Your contribution

I managed to instantiate a `DVCFileSystem` pointing to a Gitlab repo from a `fsspec` chained URL in [this pull request](https://github.com/iterative/dvc/pull/9903) to DVC. 

```python
from fsspec.core import url_to_fs

fs, _ = url_to_fs("dvc::https://gitlab.com/repository/group/my-repo")
```

From now I'm not sure how to continue, it seems that `datasets` expects the URL to be fully qualified like so: `dvc::https://gitlab.com/repository/group/my-repo/my-folder/my-file.json` but this fails because `DVCFileSystem` expects the URL to point to the root of an SCM repo. Is there a way to make this work with `datasets`?
avoid downgrading jax version

### Feature request

Whenever I `pip install datasets[jax]` it downgrades jax to version 0.3.25. I seem to be able to install this library first then upgrade jax back to version 0.4.13.

### Motivation

It would be nice to not overwrite currently installed version of jax if possible.

### Your contribution

I would be willing to beta test. Or maybe write some code if I could get pointed in the right direction, I'm not super familiar with this codebase.
Fix to_json ValueError and remove pandas pin

This PR fixes the root cause of the issue:
- #6197

This PR also removes the temporary pin of `pandas` introduced by:
- #6200


Note that for orient in ['records', 'values'], index value is ignored but
- in `pandas` < 2.1.0, a ValueError is raised if not index and orient not in ['split', 'table']
  - for orient = 'records', we need index = True
  - default index value is True
- in `pandas` = 2.1.0, a ValueError is raised if index is True and orient in ['records', 'values']
  - for orient = 'records', we need index = False or None
  - default index value is None

This PR fixes the issue by not passing index and thus using default index value (valid for all pandas versions), unless orient is 'split' or 'table' (where we pass index = False, as it was done before this fix).
Use load_dataset for local json files, but it not works

### Describe the bug

when I use load_dataset to load my local datasets，it always goes to Hugging Face to download the data instead of loading the local dataset.

### Steps to reproduce the bug

`raw_datasets = load_dataset(
        ‘json’,
        data_files=data_files)`


### Expected behavior

![image](https://github.com/huggingface/datasets/assets/50519434/add3747f-6481-4da7-b374-8f81c5a6472c)


### Environment info

python version 3.8.5
datasets version 2.12
os version unbuntu 18.04
Preserve split order in DataFilesDict

After investigation, I have found that this copy forces the splits to be sorted alphabetically: https://github.com/huggingface/datasets/blob/029227a116c14720afca71b9b22e78eb2a1c09a6/src/datasets/builder.py#L556

This PR removes the alphabetically sort of `DataFilesDict` keys.
- Note that for a `dict`, the order of keys is relevant when hashing:
  ```python
  hash1 = Hasher.hash({'train': 'train.csv', 'test': 'test.csv'})
  hash2 = Hasher.hash({'test': 'test.csv', 'train': 'train.csv'})
  assert hash1 != hash2
  ```
- The `DataFilesDict` is a subclass of `dict`, thus the order should be relevant as well
  ```python
  hash1 = Hasher.hash(DataFilesDict({'train': 'train.csv', 'test': 'test.csv'}))
  hash2 = Hasher.hash(DataFilesDict({'test': 'test.csv', 'train': 'train.csv'}))
  assert hash1 != hash2
  ```

Fix #6196.
ValueError: 'index=True' is only valid when 'orient' is 'split', 'table', 'index', or 'columns'

### Describe the bug

Saving a dataset `.to_json()` fails with a `ValueError` since the latest `pandas` [release](https://pandas.pydata.org/docs/dev/whatsnew/v2.1.0.html) (`2.1.0`)

In their latest release we have:

> Improved error handling when using [DataFrame.to_json()](https://pandas.pydata.org/docs/dev/reference/api/pandas.DataFrame.to_json.html#pandas.DataFrame.to_json) with incompatible index and orient arguments ([GH 52143](https://github.com/pandas-dev/pandas/issues/52143))

i.e. an error is now raised for invalid combinations of `index` and `orient`.

This means that unfortunately the custom logic at this line might sometimes lead to contradictions:

https://github.com/huggingface/datasets/blob/029227a116c14720afca71b9b22e78eb2a1c09a6/src/datasets/io/json.py#L96

e.g. for the default case `orient=records` leads to `index=True`, which now raises a `ValueError`

### Steps to reproduce the bug

```python
import datasets


if __name__ == '__main__':

    dataset = datasets.Dataset.from_dict({"A": [1, 2, 3], "B": [4, 5, 6]})
    dataset.to_json("dataset.json")
```

```shell
>>>
ValueError: 'index=True' is only valid when 'orient' is 'split', 'table', 'index', or 'columns'.
```

### Expected behavior

The dataset is successfully saved as `.json`

### Environment info

`python >= 3.9`
`pandas >= 2.1.0`
Force to reuse cache at given path

### Describe the bug

I have run the official example of MLM like:

```bash
  python run_mlm.py \
      --model_name_or_path roberta-base \
      --dataset_name togethercomputer/RedPajama-Data-1T \
      --dataset_config_name arxiv \
      --per_device_train_batch_size 10 \
      --preprocessing_num_workers 20 \
      --validation_split_percentage 0 \
      --cache_dir /project/huggingface_cache/datasets \
      --line_by_line \
      --do_train \
      --pad_to_max_length \
      --output_dir /project/huggingface_cache/test-mlm
```
it successfully runs and at my cache folder has `cache-1982fea76aa54a13_00001_of_00020.arrow`.....  `cache-1982fea76aa54a13_00020_of_00020.arrow ` as tokenization cache of `map` method. And the cache works fine every time I run the command above.

However, when I switched to jupyter notebook (since I do not want to load datasets every time when I changed other parameters not related to the dataloading). It is not recognizing the cache files and starts to re-run the entire tokenization process. 

I changed my code to 
```python
tokenized_datasets = raw_datasets["train"].map(
                tokenize_function,
                batched=True,
                num_proc=data_args.preprocessing_num_workers,
                remove_columns=[text_column_name],
                load_from_cache_file=True,
                desc="Running tokenizer on dataset line_by_line",
                # cache_file_names= {"train": "cache-1982fea76aa54a13.arrow"}
                cache_file_name="cache-1982fea76aa54a13.arrow",
                new_fingerprint="1982fea76aa54a13"
            )
```
it still does not recognize the previously cached files and trying to re-run the tokenization process.

### Steps to reproduce the bug

use jupyter notebook for dataset map function.

### Expected behavior

the map function accepts the given cache_file_name and new_fingerprint then load the previously cached files.

### Environment info

- `datasets` version: 2.14.4.dev0
- Platform: Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.8
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
Support custom fingerprinting with `Dataset.from_generator`

### Feature request

When using `Dataset.from_generator`, the generator is hashed when building the fingerprint. Similar to `.map`, it would be interesting to let the user bypass this hashing by accepting a `fingerprint` argument to `.from_generator`.

### Motivation

Using the `.from_generator` constructor with a non-picklable generator fails. By accepting a `fingerprint` argument to `.from_generator`, the user would have the opportunity to manually fingerprint the dataset and thus bypass the crash.

### Your contribution

If validated, I can try to submit a PR for this.
Dataset loading script method does not work with .pyc file

### Describe the bug

The huggingface dataset library specifically looks for ‘.py’ file while loading the dataset using loading script approach and it does not work with ‘.pyc’ file.
While deploying in production, it becomes an issue when we are restricted to use only .pyc files. Is there any work around for this ?

### Steps to reproduce the bug

1. Create a dataset  loading script to read the custom data.
2. compile the code to make sure that .pyc file is created 
3. Delete the loading script and re-run the code. Usually, python should make use of complied .pyc files. However, in this case, the dataset library errors out with the message that it's unable to find the data loader loading script.

### Expected behavior

The code should make use of .pyc file and run without any error.

### Environment info

NA
`Invalid user token` even when correct user token is passed!

### Describe the bug

I'm working on a dataset which comprises other datasets on the hub.
URL: https://huggingface.co/datasets/open-asr-leaderboard/datasets-test-only

Note: Some of the sub-datasets in this metadataset require explicit access.

All the other datasets work fine, except, `common_voice`.

### Steps to reproduce the bug

https://github.com/Vaibhavs10/scratchpad/blob/main/cv_datasets_bug_repro.ipynb

### Expected behavior

It should work if the provided access token is valid (as it does for all the other datasets)

### Environment info

datasets version -> 2.14.4
[Feature Request] Check the length of batch before writing so that empty batch is allowed

### Use Case
I use `dataset.map(process_fn, batched=True)` to process the dataset, with data **augmentations or filtering**. However, when all examples within a batch is filtered out, i.e. **an empty batch is returned**, the following error will be thrown:
  ```
  ValueError: Schema and number of arrays unequal
  ```
This is because the empty batch does not comply with the schema of other batches. I think an empty batch should be allowed to facilitate coding (one does not need to assign an empty list manually for all keys.)

A simple fix is to check the length of `batch` before writing:
```
if len(batch):
  writer.write_batch(batch)
```
instead of 

https://github.com/huggingface/datasets/blob/74d60213dcbd7c99484c62ce1d3dfd90a1df0770/src/datasets/arrow_dataset.py#L3493
Couldn't find a dataset script at /content/tsv/tsv.py or any data file in the same directory

### Describe the bug

```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
[<ipython-input-48-6a7b3e847019>](https://localhost:8080/#) in <cell line: 7>()
      5 }
      6 
----> 7 csv_datasets_reloaded = load_dataset("tsv", data_files=data_files)
      8 csv_datasets_reloaded

2 frames
[/usr/local/lib/python3.10/dist-packages/datasets/load.py](https://localhost:8080/#) in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1489                     raise e1 from None
   1490                 if isinstance(e1, FileNotFoundError):
-> 1491                     raise FileNotFoundError(
   1492                         f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1493                         f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"

FileNotFoundError: Couldn't find a dataset script at /content/tsv/tsv.py or any data file in the same directory. Couldn't find 'tsv' on the Hugging Face Hub either: FileNotFoundError: Dataset 'tsv' doesn't exist on the Hub
```

### Steps to reproduce the bug

```
data_files = {
    "train": "/content/PUBHEALTH/train.tsv",
    "validation": "/content/PUBHEALTH/dev.tsv",
    "test": "/content/PUBHEALTH/test.tsv",
}

tsv_datasets_reloaded = load_dataset("tsv", data_files=data_files)
tsv_datasets_reloaded
```
```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-48-6a7b3e847019> in <cell line: 7>()
      5 }
      6 
----> 7 csv_datasets_reloaded = load_dataset("tsv", data_files=data_files)
      8 csv_datasets_reloaded

2 frames
/usr/local/lib/python3.10/dist-packages/datasets/load.py in dataset_module_factory(path, revision, download_config, download_mode, dynamic_modules_path, data_dir, data_files, **download_kwargs)
   1489                     raise e1 from None
   1490                 if isinstance(e1, FileNotFoundError):
-> 1491                     raise FileNotFoundError(
   1492                         f"Couldn't find a dataset script at {relative_to_absolute_path(combined_path)} or any data file in the same directory. "
   1493                         f"Couldn't find '{path}' on the Hugging Face Hub either: {type(e1).__name__}: {e1}"

FileNotFoundError: Couldn't find a dataset script at /content/tsv/tsv.py or any data file in the same directory. Couldn't find 'tsv' on the Hugging Face Hub either: FileNotFoundError: Dataset 'tsv' doesn't exist on the Hub
```


### Expected behavior

load the data, push to hub

### Environment info

jupyter notebook RTX 3090