sunny2309/githug-issues huggingface dataset:
Fix authentication issues

This PR fixes 3 authentication issues:
- Fix authentication when passing `token`.
- Fix authentication in `Audio.decode_example` and `Image.decode_example`.
- Fix authentication to resolve `data_files` in repositories without script.

This PR also fixes our CI so that we properly test when passing `token` and we do not use the token stored in `HfFolder`.

Fix #6126.

## Details


### Fix authentication when passing `token`

See c0a77dc943de68a17f23f141517028c734c78623

The root issue was caused when the `token` was set in an already instantiated `DownloadConfig` and thus not propagated to `self._storage_options`:

  ```python
  download_config.token = token
  ```

As this usage pattern is very common, the fix consists in overriding `DownloadConfig.__setattr__`.

This fixes authentication issues in the following functions:
- `load_dataset` and `load_dataset_builder`
- `Dataset.push_to_hub` and `Dataset.push_to_hub`
- `inspect.get_dataset_config_info`, `inspect.get_dataset_infos` and `inspect.get_dataset_split_names`

### Fix authentication in `Audio.decode_example` and `Image.decode_example`. 

See: 58e62af004b6b8b84dcfd897a4bc71637cfa6c3f

The `token` was not set because the `repo_id` was wrongly tried to be parsed from an HTTP URL (`"http://..."`), instead of an HFFileSystem URL (`"hf://"`)

### Fix authentication to resolve `data_files` in repositories without script

See: e4684fc1032321abf0d494b0c130ea7c82ebda80

This is fixed by passing `download_config` to the function `create_builder_configs_from_metadata_configs`
Datasets crashing runs due to KeyError

### Describe the bug

Hi all,

I have been running into a pretty persistent issue recently when trying to load datasets. 
```python
    train_dataset = load_dataset(
        'llama-2-7b-tokenized', 
        split = 'train'
    )
```
I receive a KeyError which crashes the runs.
```
Traceback (most recent call last):
    main()

    train_dataset = load_dataset(
                    ^^^^^^^^^^^^^
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
    raise e1 from None

    ).get_module()
      ^^^^^^^^^^^^
    else get_data_patterns(base_path, download_config=self.download_config)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return _get_data_files_patterns(resolver)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    data_files = pattern_resolver(pattern)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
    fs, _, _ = get_fs_token_paths(pattern, storage_options=storage_options)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    paths = [f for f in sorted(fs.glob(paths)) if not fs.isdir(f)]
                               ^^^^^^^^^^^^^^

    allpaths = self.find(root, maxdepth=depth, withdirs=True, detail=True, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    for _, dirs, files in self.walk(path, maxdepth, detail=True, **kwargs):

    listing = self.ls(path, detail=True, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    "last_modified": parse_datetime(tree_item["lastCommit"]["date"]),
                                    ~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'lastCommit'
```
Any help would be greatly appreciated.

Thank you,

Enrico

### Steps to reproduce the bug

Load the dataset from the Huggingface hub.
```python
    train_dataset = load_dataset(
        'llama-2-7b-tokenized', 
        split = 'train'
    )
```

### Expected behavior

Loads the dataset.

### Environment info

datasets-2.14.3
CUDA 11.8
Python 3.11
Upload README via `push_to_hub`

### Feature request

`push_to_hub` now allows users to upload datasets programmatically. However, based on the latest doc, we still need to open the dataset page to add readme file manually.

However, I do discover snippets to intialize a README for every `push_to_hub`:
```
dataset_card = (
    DatasetCard(
        "---\n"
        + str(dataset_card_data)
        + "\n---\n"
        + f'# Dataset Card for "{repo_id.split("/")[-1]}"\n\n[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)'
    )
    if dataset_card is None
    else dataset_card
)

HfApi(endpoint=config.HF_ENDPOINT).upload_file(
    path_or_fileobj=str(dataset_card).encode(),
    path_in_repo="README.md",
    repo_id=repo_id,
    token=token,
    repo_type="dataset",
    revision=branch,
)
```

So, if we can enable `push_to_hub` to upload a readme file by ourselves instead of using the auto generated ones, it can save ton of time, and will definitely alleviate the current "lack-of-dataset-card" situation.

### Motivation

as elabrated above.

### Your contribution

I might be able to make a pr.
Lookahead streaming support?

### Feature request

From what I understand, streaming dataset currently pulls the data, and process the data as it is requested.
This can introduce significant latency delays when data is loaded into the training process, needing to wait for each segment.

While the delays might be dataset specific (or even mapping instruction/tokenizer specific)

Is it possible to introduce a `streaming_lookahead` parameter, which is used for predictable workloads (even shuffled dataset with fixed seed). As we can predict in advance what the next few datasamples will be. And fetch them while the current set is being trained.

With enough CPU & bandwidth to keep up with the training process, and a sufficiently large lookahead, this will reduce the various latency involved while waiting for the dataset to be ready between batches.

### Motivation

Faster streaming performance, while training over extra large TB sized datasets

### Your contribution

I currently use HF dataset, with pytorch lightning trainer for RWKV project, and would be able to help test this feature if supported.
[Docs] The "Process" how-to guide lacks description of `select_columns` function

### Feature request

The [how to process dataset guide](https://huggingface.co/docs/datasets/main/en/process) currently does not mention the [`select_columns`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.select_columns) function. It would be nice to include it in the guide.

### Motivation

This function is a commonly requested feature (see this [forum thread](https://discuss.huggingface.co/t/how-to-create-a-new-dataset-from-another-dataset-and-select-specific-columns-and-the-data-along-with-the-column/15120) and #5468 #5474). However, it has not been included in the guide since its implementation by PR #5480.

Mentioning it in the guide would help future users discover this added feature.

### Your contribution

I could submit a PR to add a brief description of the function to said guide.
Cache not being used when loading commonvoice 8.0.0

### Describe the bug

I have commonvoice 8.0.0 downloaded in `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/b2f8b72f8f30b2e98c41ccf855954d9e35a5fa498c43332df198534ff9797a4a`.  The folder contains all the arrow files etc, and was used as the cached version last time I touched the ec2 instance I'm working on.  Now, with the same command that downloaded it initially:

```
dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")
```

it tries to redownload the dataset to `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/05bdc7940b0a336ceeaeef13470c89522c29a8e4494cbeece64fb472a87acb32`

### Steps to reproduce the bug

Steps to reproduce the behavior:

1.  ```dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")```
2. dataset is updated by maintainers
3. ```dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")```

### Expected behavior

I expect that it uses the already downloaded data in `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/b2f8b72f8f30b2e98c41ccf855954d9e35a5fa498c43332df198534ff9797a4a`.

Not sure what's happening in 2. but if, say it's an issue with the dataset referenced by "mozilla-foundation/common_voice_8_0" being modified by the maintainers, how would I force datasets to point to the original version I downloaded?

EDIT:  It was indeed that the maintainers had updated the dataset (v 8.0.0).  However I still cant load the dataset from disk instead of redownloading, with for example:

```
load_dataset(".cache/huggingface/datasets/downloads/extracted/<hash>/cv-corpus-8.0-2022-01-19/en/", "en")

> ...
> File [~/miniconda3/envs/aa_torch2/lib/python3.10/site-packages/datasets/table.py:1938](.../ python3.10/site-packages/datasets/table.py:1938), in cast_array_to_feature(array, feature, allow_number_to_str)
   1937 elif not isinstance(feature, (Sequence, dict, list, tuple)):
-> 1938     return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
...
   1794         e = e.__context__
-> 1795     raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1797 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```


### Environment info

datasets==2.7.0
python==3.10.8
OS:  AWS Linux
load_dataset() fails with streamlit caching inside docker

### Describe the bug

When calling `load_dataset` in a streamlit application running within a docker container, get a failure with the error message:

EmptyDatasetError: The directory at hf://datasets/fetch-rewards/inc-rings-2000@bea27cf60842b3641eae418f38864a2ec4cde684 doesn't contain any data files
Traceback:
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script
    exec(code, module.__dict__)
File "/home/user/app/app.py", line 62, in <module>
    dashboard()
File "/home/user/app/app.py", line 47, in dashboard
    feat_dict, path_gml = load_data(hf_repo, model_gml_dict[selected_model], hf_token)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 211, in wrapper
    return cached_func(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 240, in __call__
    return self._get_or_create_cached_value(args, kwargs)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 266, in _get_or_create_cached_value
    return self._handle_cache_miss(cache, value_key, func_args, func_kwargs)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 320, in _handle_cache_miss
    computed_value = self._info.func(*func_args, **func_kwargs)
File "/home/user/app/hf_interface.py", line 16, in load_data
    hf_dataset = load_dataset(repo_id, use_auth_token=hf_token)
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 2109, in load_dataset
    builder_instance = load_dataset_builder(
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1795, in load_dataset_builder
    dataset_module = dataset_module_factory(
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1486, in dataset_module_factory
    raise e1 from None
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1476, in dataset_module_factory
    ).get_module()
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1032, in get_module
    else get_data_patterns(base_path, download_config=self.download_config)
File "/opt/conda/lib/python3.10/site-packages/datasets/data_files.py", line 458, in get_data_patterns
    raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None

### Steps to reproduce the bug

```python
@st.cache_resource
def load_data(repo_id: str, hf_token=None):
    """Load data from HuggingFace Hub
    """
    
    hf_dataset = load_dataset(repo_id, use_auth_token=hf_token)
    hf_dataset = hf_dataset.map(lambda x: json.loads(x["ground_truth"]), remove_columns=["ground_truth"])

    return hf_dataset
```

### Expected behavior

Expect to load. 
Note: works fine with datasets==2.13.1

### Environment info

datasets==2.14.2, 
Ubuntu bionic-based Docker container.
raise FileNotFoundError("Directory {dataset_path} is neither a `Dataset` directory nor a `DatasetDict` directory." )

### Describe the bug

For researchers in some countries or regions, it is usually the case that the download ability of `load_dataset` is disabled due to the complex network environment. People in these regions often prefer to use git clone or other programming tricks to manually download the files to the disk (for example, [How to elegantly download hf models, zhihu zhuanlan](https://zhuanlan.zhihu.com/p/475260268) proposed a crawlder based solution,  and [Is there any mirror for hf_hub, zhihu answer](https://www.zhihu.com/question/371644077) provided some cloud based solutions, and [How to avoid pitfalls on Hugging face downloading, zhihu zhuanlan] gave some useful suggestions), and then use `load_from_disk` to get the dataset object. 
However, when one finally has the local files on the disk, it is still buggy when trying to load the files into objects. 

### Steps to reproduce the bug

Steps to reproduce the bug:
1. Found CIFAR dataset in hugging face: https://huggingface.co/datasets/cifar100/tree/main
2. Click ":" button to show "Clone repository" option, and then follow the prompts on the box:
 ```bash
  cd my_directory_absolute
  git lfs install
  git clone https://huggingface.co/datasets/cifar100
  ls  my_directory_absolute/cifar100 # confirm that the directory exists and it is OK. 
  ```
3. Write A python file to try to load the dataset
```python
from datasets import load_dataset, load_from_disk
dataset = load_from_disk("my_directory_absolute/cifar100")
```
Notice that according to issue #3700 , it is wrong to use load_dataset("my_directory_absolute/cifar100"), so we must use load_from_disk instead. 

4. Then you will see the error reported:
```log
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[5], line 9
      1 from datasets import load_dataset, load_from_disk
----> 9 dataset = load_from_disk("my_directory_absolute/cifar100")

File [~/miniconda3/envs/ai/lib/python3.10/site-packages/datasets/load.py:2232), in load_from_disk(dataset_path, fs, keep_in_memory, storage_options)
   2230     return DatasetDict.load_from_disk(dataset_path, keep_in_memory=keep_in_memory, storage_options=storage_options)
   2231 else:
-> 2232     raise FileNotFoundError(
   2233         f"Directory {dataset_path} is neither a `Dataset` directory nor a `DatasetDict` directory."
   2234     )

FileNotFoundError: Directory my_directory_absolute/cifar100 is neither a `Dataset` directory nor a `DatasetDict` directory.
```

### Expected behavior

The dataset should be load successfully. 

### Environment info

```bash
datasets-cli env
```
-> results:
```txt

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.14.2
- Platform: Linux-4.18.0-372.32.1.el8_6.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
```
[BUG] Dataset initialized from in-memory data does not create cache.

### Describe the bug

`Dataset` initialized from in-memory data (dictionary in my case, haven't tested with other types) does not create cache when processed with the `map` method, unlike `Dataset` initialized by other methods such as `load_dataset`.

### Steps to reproduce the bug

```python
# below code was run the second time so the map function can be loaded from cache if exists

from datasets import load_dataset, Dataset

dataset = load_dataset("tatsu-lab/alpaca")['train']

dataset = dataset.map(lambda x: {'input': x['input'] + 'hi'}) # some random map
print(len(dataset.cache_files))
# 1

# copy the exact same data but initialize from a dictionary
memory_dataset = Dataset.from_dict({
    'instruction': dataset['instruction'],
    'input': dataset['input'],
    'output': dataset['output'],
    'text': dataset['text']})

memory_dataset = memory_dataset.map(lambda x: {'input': x['input'] + 'hi'}) # exact same map
print(len(memory_dataset.cache_files))
# Map: 100%|██████████| 52002[/52002]
# 0
```

### Expected behavior

The `map` function should create cache regardless of the method the `Dataset` was created.

### Environment info

- `datasets` version: 2.14.2
- Platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
Problems in downloading Amazon reviews from HF

### Describe the bug

I have a script downloading `amazon_reviews_multi`.

When the download starts, I get

```
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 243B [00:00, 1.43MB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.54s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 842.40it/s]
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 243B [00:00, 928kB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.42s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 832.70it/s]
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 243B [00:00, 1.81MB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.40s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 1294.14it/s]
Generating train split:   0%|          | 0/200000 [00:00<?, ? examples/s]
```

the file is clearly too small to contain the requested dataset, in fact it contains en error message:

```
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>AGJWSY3ZADT2QVWE</RequestId><HostId>Gx1O2KXnxtQFqvzDLxyVSTq3+TTJuTnuVFnJL3SP89Yp8UzvYLPTVwd1PpniE4EvQzT3tCaqEJw=</HostId></Error>
```

obviously the script fails:

```
>           raise DatasetGenerationError("An error occurred while generating the dataset") from e
E           datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```



### Steps to reproduce the bug

1. load_dataset("amazon_reviews_multi", name="en", split="train", cache_dir="ADDYOURPATHHERE")

### Expected behavior

I would expect the dataset to be downloaded and processed

### Environment info

* The problem is present with both datasets 2.12.0 and 2.14.2
* python version 3.10.12
Fix deprecation of use_auth_token in file_utils

Fix issues with the deprecation of `use_auth_token`  introduced by:
- #5996

in functions:
- `get_authentication_headers_for_url`
- `request_etag`
- `get_from_cache`

Currently, `TypeError` is raised: https://github.com/huggingface/datasets-server/actions/runs/5711650666/job/15484685570?pr=1588
```
FAILED tests/job_runners/config/test_parquet_and_info.py::test__is_too_big_external_files[None-None-False] - TypeError: get_authentication_headers_for_url() got an unexpected keyword argument 'use_auth_token'

FAILED tests/job_runners/config/test_parquet_and_info.py::test_fill_builder_info[None-False] - libcommon.exceptions.FileSystemError: Could not read the parquet files: get_authentication_headers_for_url() got an unexpected keyword argument 'use_auth_token'
```

Related to:
- #6094
load local json_file as dataset

### Describe the bug

I tried to load local json file as dataset but failed to parsing json file because some columns are 'float' type.

### Steps to reproduce the bug

1. load json file with certain columns are 'float' type. For example `data = load_data("json", data_files=JSON_PATH)`
2. Then, the error will be triggered like `ArrowInvalid: Could not convert '-0.2253' with type str: tried to convert to double

### Expected behavior

Should allow some columns are 'float' type, at least it should convert those columns to str type.
I tried to avoid the error by naively convert the float item to str:
```python
# if col type is not str, we need to convert it to str
mapping = {}
for col in keys:
    if isinstance(dataset[0][col], str):
        mapping[col] = [row.get(col) for row in dataset]
    else:
        mapping[col] = [str(row.get(col)) for row in dataset]
```

### Environment info

- `datasets` version: 2.14.2
- Platform: Linux-5.4.0-52-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.0
- Pandas version: 2.0.1
HF Datasets data access is extremely slow even when in memory

### Describe the bug

Doing a simple `some_dataset[:10]` can take more than a minute.

Profiling it:
<img width="1280" alt="image" src="https://github.com/huggingface/datasets/assets/36224762/e641fb95-ff02-4072-9016-5416a65f75ab">

`some_dataset` is completely in memory with no disk cache.

This is proving fatal to my usage of HF Datasets. Is there a way I can forgo the arrow format and store the dataset as PyTorch tensors so that `_tensorize` is not needed? And is `_consolidate` supposed to take this long?

It's faster to produce the dataset from scratch than to access it from HF Datasets!

### Steps to reproduce the bug

I have uploaded the dataset that causes this problem [here](https://huggingface.co/datasets/NightMachinery/hf_datasets_bug1).

```python
#!/usr/bin/env python3
import sys
import time
import torch
from datasets import load_dataset


def main(dataset_name):
    # Start the timer
    start_time = time.time()

    # Load the dataset from Hugging Face Hub
    dataset = load_dataset(dataset_name)

    # Set the dataset format as torch
    dataset.set_format(type="torch")

    # Perform an identity map
    dataset = dataset.map(lambda example: example, batched=True, batch_size=20)

    # End the timer
    end_time = time.time()

    # Print the time taken
    print(f"Time taken: {end_time - start_time:.2f} seconds")


if __name__ == "__main__":
    dataset_name = "NightMachinery/hf_datasets_bug1"
    print(f"dataset_name: {dataset_name}")
    main(dataset_name)
```

### Expected behavior

_

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
TypeError when loading from GCP bucket

### Describe the bug

Loading a dataset from a GCP bucket raises a type error. This bug was introduced recently (either in 2.14 or 2.14.1), and appeared during a migration from 2.13.1.

### Steps to reproduce the bug

Load any file from a GCP bucket:

```python
import datasets

datasets.load_dataset("json", data_files=["gs://..."])
```

The following exception is raised:

```python
Traceback (most recent call last):
...
packages/datasets/data_files.py", line 335, in resolve_pattern
    protocol_prefix = fs.protocol + "://" if fs.protocol != "file" else ""
TypeError: can only concatenate tuple (not "str") to tuple
```

With a `GoogleFileSystem`, the attribute `fs.protocol` is a tuple `('gs', 'gcs')` and hence cannot be concatenated with a string.


### Expected behavior

The file should be loaded without exception.

### Environment info

- `datasets` version: 2.14.1
- Platform: macOS-13.2.1-x86_64-i386-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3