sjwsjw/github-issues huggingface dataset:
GitHub issue: 'import datasets' throws "invalid syntax error"
State: closed

### Describe the bug

Hi,

I have been trying to import the datasets library but I keep gtting this error.

`Traceback (most recent call last):

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/IPython/core/interactiveshell.py:3508 in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  Cell In[2], line 1
    import datasets

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/__init__.py:22
    from .arrow_dataset import Dataset

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/arrow_dataset.py:67
    from .arrow_writer import ArrowWriter, OptimizedTypedSequence

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/arrow_writer.py:27
    from .features import Features, Image, Value

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/features/__init__.py:17
    from .audio import Audio

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/features/audio.py:11
    from ..download.streaming_download_manager import xopen, xsplitext

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/download/__init__.py:10
    from .streaming_download_manager import StreamingDownloadManager

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/datasets/download/streaming_download_manager.py:18
    from aiohttp.client_exceptions import ClientError

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/__init__.py:7
    from .connector import *  # noqa

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/connector.py:12
    from .client import ClientRequest

  File /opt/local/jupyterhub/lib64/python3.9/site-packages/aiohttp/client.py:144
    yield from asyncio.async(resp.release(), loop=loop)
                       ^
SyntaxError: invalid syntax`

I have simply used these commands:
`import datasets`
and
`from datasets import load_dataset`

### Environment info

The library has been installed a virtual machine on JupyterHub. Although I have used this library multiple times (on the same VM) before, to train/test an ASR or other ML models, I had never encountered this error.
GitHub issue: how to limit the size of memory mapped file?
State: open

### Describe the bug

Huggingface datasets use memory-mapped file to map large datasets in memory for fast access.
However, it seems like huggingface will occupy all the memory for memory-mapped files, which makes a troublesome situation since we cluster will distribute a small portion of memory to me (once it's over the limit, memory cannot be allocated), however, when the dataset checks the total memory, all of the memory will be taken into account which makes huggingface dataset try to allocate more memory than allowed. 
So is there a way to explicitly limit the size of memory mapped file?

### Steps to reproduce the bug

python
>>> from datasets import load_dataset
>>> dataset = load_dataset("c4", "en", streaming=True)

### Expected behavior

In a normal environment, this will not have any problem.
However, when the system allocates a portion of the memory to the program and when the dataset checks the total memory, all of the memory will be taken into account which makes huggingface dataset try to allocate more memory than allowed. 

### Environment info

linux cluster with SGE（Sun Grid Engine）
GitHub issue: PyArrow 13 CI fixes
State: closed

Fixes:
* bumps the PyArrow version check in the `cast_array_to_feature` to avoid the offset bug (still not fixed)
* aligns the Pandas formatting tests with the Numpy ones (the current test fails due to https://github.com/apache/arrow/pull/35656, which requires `.to_pandas(coerce_temporal_nanoseconds=True)` to always return `datetime [ns]` objects)

Fix #6173
GitHub issue: Fix CI for pyarrow 13.0.0
State: closed

pyarrow 13.0.0 just came out

```
FAILED tests/test_formatting.py::ArrowExtractorTest::test_pandas_extractor - AssertionError: Attributes of Series are different

Attribute "dtype" are different
[left]:  datetime64[us, UTC]
[right]: datetime64[ns, UTC]
```

```
FAILED tests/test_table.py::test_cast_sliced_fixed_size_array_to_features - TypeError: Couldn't cast array of type
fixed_size_list<item: int32>[3]
to
Sequence(feature=Value(dtype='int64', id=None), length=3, id=None)
```

e.g. in https://github.com/huggingface/datasets/actions/runs/5952253963/job/16143847230

first error may be related to https://github.com/apache/arrow/issues/33321

second one maybe because `feature.length * len(array) == len(array_values)` is not satisfied anymore somehow ?
GitHub issue: Make Dataset streaming queries retryable
State: open

### Feature request

Streaming datasets, as intended, do not load the entire dataset in memory or disk. However, while querying the next data chunk from the remote, sometimes it is possible that the service is down or there might be other issues that may cause the query to fail. In such a scenario, it would be nice to make these queries retryable (perhaps with a backoff strategy).

### Motivation

I was working on a model and the model checkpoints after every 1000 steps. At step 1800 I got a 504 HTTP status code error from Huggingface hub for my pytorch `dataloader`. Given the size of my model and data, it took around 2 hours to reach 1800 steps and now it will take about an hour to recover the lost 800. It would be better to get a retryable querying strategy.

### Your contribution

It would be better if someone having experience in this area takes this up as this would require some testing.
GitHub issue: feat: Return the name of the currently loaded file
State: open

Added an optional parameter return_file_name in the load_dataset function. When it is set to True, the function will include the name of the file corresponding to the current line as a feature in the returned output.

I added this here https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/json/json.py#L92.

fixes #5806
GitHub issue: Configurations in yaml not working
State: open

### Dataset configurations cannot be created in YAML/README

Hello! I'm trying to follow the docs here in order to create structure in my dataset as added from here (#5331): https://github.com/huggingface/datasets/blob/8b8e6ee067eb74e7965ca2a6768f15f9398cb7c8/docs/source/repository_structure.mdx#L110-L118

I have the exact example in my config file for [my data repo](https://huggingface.co/datasets/tsor13/test):
```
configs:
- config_name: main_data
  data_files: "main_data.csv"
- config_name: additional_data
  data_files: "additional_data.csv"
```

Yet, I'm unable to load different configurations:
```
from datasets import get_dataset_config_names
get_dataset_config_names('tsor13/test', use_auth_token=True)
```
returns a single split, `['tsor13--test']`

Does anyone have any insights?

@polinaeterna thank you for adding this feature, it is super useful. Do you happen to have any ideas?

### Steps to reproduce the bug

from datasets import get_dataset_config_names
get_dataset_config_names('tsor13/test')

### Expected behavior

I would expect there to be two splits, `main_data` and `additional_data`. However, only `['tsor13--test']` test is returned.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.1
GitHub issue: Fix multiprocessing with spawn in iterable datasets
State: closed

The "Spawn" method is preferred when multiprocessing on macOS or Windows systems, instead of the "Fork" method on linux systems.

This causes some methods of Iterable Datasets to break when using a dataloader with more than 0 workers.

I fixed the issue by replacing lambda and local methods which are not pickle-able.

See the example below:

```python
from datasets import load_dataset
from torch.utils.data import DataLoader


if __name__ == "__main__":
    dataset = load_dataset("lhoestq/demo1", split="train")
    dataset = dataset.to_iterable_dataset(num_shards=3)

    dataset = dataset.remove_columns(["package_name"])
    dataset = dataset.rename_columns({
        "review": "review1"
    })
    dataset = dataset.rename_column("date", "date1")
    for sample in DataLoader(dataset, batch_size=None, num_workers=3):
        print(sample)
```

To notice the fix on a linux system, adding these lines should do the trick:

```python
import multiprocessing
multiprocessing.set_start_method('spawn')
```

I also removed what looks like code duplication between rename_colums and rename_column
GitHub issue: Fix: Missing a MetadataConfigs init when the repo has a `datasets_info.json` but no README
State: closed

When I try to push to an arrow repo (can provide the link on Slack), it uploads the files but fails to update the metadata, with
```
  File "app.py", line 123, in add_new_eval
    eval_results[level].push_to_hub(my_repo, token=TOKEN, split=SPLIT)
  File "blabla_my_env_path/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 5501, in push_to_hub
    if not metadata_configs:
UnboundLocalError: local variable 'metadata_configs' referenced before assignment
```

This fixes it.
GitHub issue: Error type: ArrowInvalid Details: Failed to parse string: '[254,254]' as a scalar of type int32
State: open

### Describe the bug

I am getting the following error while I am trying to upload the CSV sheet to train a model. My CSV sheet content is exactly same as shown in the example CSV file in the Auto Train page. Attaching screenshot of error for reference. I have also tried converting the index of the answer that are integer into string by placing inverted commas and also without inverted commas. 
Can anyone please help me out? 
FYI : I am using Chrome browser.

Error type: ArrowInvalid
Details: Failed to parse string: '[254,254]' as a scalar of type int32

![Screenshot 2023-08-19 165827](https://github.com/huggingface/datasets/assets/90616801/95fad96e-7dce-4bb5-9f83-9f1659a32891)


### Steps to reproduce the bug

Kindly let me know how to fix this?

### Expected behavior

Kindly let me know how to fix this?

### Environment info

Kindly let me know how to fix this?
GitHub issue: Add `BoundingBox` feature
State: open

... to make working with object detection datasets easier. Currently, `Sequence(int_or_float, length=4)` can be used to represent this feature optimally (in the storage backend), so I only see this feature being useful if we make it work with the viewer. Also, bounding boxes usually come in 4 different formats (explained [here](https://albumentations.ai/docs/getting_started/bounding_boxes_augmentation/)), so we need to decide which one to support (or maybe all of them).

cc @NielsRogge @severo
GitHub issue: Why not use self._epoch as seed to shuffle in distributed training with IterableDataset
State: closed

### Describe the bug

Currently, distributed training with `IterableDataset` needs to pass fixed seed to shuffle to keep each node use the same seed to avoid overlapping.
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1174-L1177

My question is why not directly use `self._epoch` which is set by `set_epoch` as seed? It's almost the same across nodes.
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1790-L1801

If not using `self._epoch` as shuffling seed, what does this method do to prepare an epoch seeded generator?
https://github.com/huggingface/datasets/blob/a7f8d9019e7cb104eac4106bdc6ec0292f0dc61a/src/datasets/iterable_dataset.py#L1206



### Steps to reproduce the bug

As mentioned above.

### Expected behavior

As mentioned above.

### Environment info

Not related
GitHub issue: FolderBase Dataset automatically resolves under current directory when data_dir is not specified
State: open

### Describe the bug

FolderBase Dataset automatically resolves under current directory when data_dir is not specified.

For example:
```
load_dataset("audiofolder")
```
takes long time to resolve and collect data_files from current directory. But I think it should reach out to this line for error handling https://github.com/huggingface/datasets/blob/cb8c5de5145c7e7eee65391cb7f4d92f0d565d62/src/datasets/packaged_modules/folder_based_builder/folder_based_builder.py#L58-L59

### Steps to reproduce the bug

```
load_dataset("audiofolder")
```

### Expected behavior

Error report

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-78-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3
GitHub issue: Faster sorting for single key items
State: closed

### Feature request

A faster way to sort a dataset which contains a large number of rows.

### Motivation

The current sorting implementations took significantly longer than expected when I was running on a dataset trying to sort by timestamps. 

**Code snippet:**
```python
ds = datasets.load_dataset( "json", **{"data_files": {"train": "path-to-jsonlines"}, "split": "train"}, num_proc=os.cpu_count(), keep_in_memory=True) 
sorted_ds = ds.sort("pubDate", keep_in_memory=True)
```

However, once I switched to a different method which
1. unpacked to a list of tuples
2. sorted tuples by key
3. run `.select` with the sorted list of indices
It was significantly faster (orders of magnitude, especially with M's of rows)

### Your contribution

I'd be happy to implement a crude single key sorting algorithm so that other users can benefit from this trick. Broadly, this would take a `Dataset` and perform;

```python
# ds is a Dataset object
# key_name is the sorting key

class Dataset:
   ...
   def _sort(key_name: str) -> Dataset:
      index_keys = [(i,x) for i,x in enumerate(self[key_name])]
      sorted_rows = sorted(row_pubdate, key=lambda x: x[1])
      sorted_indicies = [x[0] for x in sorted_rows]
      return self.select(sorted_indicies)
```
GitHub issue: Dataset.from_parquet cannot load subset of columns
State: closed

### Describe the bug

When using `Dataset.from_parquet(path_or_paths, columns=[...])` and a subset of columns, loading fails with a variant of the following

```
ValueError: Couldn't cast
a: int64
-- schema metadata --
pandas: '{"index_columns": [], "column_indexes": [], "columns": [{"name":' + 273
to
{'a': Value(dtype='int64', id=None), 'b': Value(dtype='int64', id=None)}
because column names don't match

The above exception was the direct cause of the following exception:
```

Looks to be triggered by https://github.com/huggingface/datasets/blob/c02a44715c036b5261686669727394b1308a3a4b/src/datasets/table.py#L2285-L2286

### Steps to reproduce the bug

```
import pandas as pd
from datasets import Dataset


pd.DataFrame([{"a": 1, "b": 2}]).to_parquet("test.pq")
Dataset.from_parquet("test.pq", columns=["a"])
```

### Expected behavior

A subset of columns should be loaded without error

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.10.0-23-cloud-amd64-x86_64-with-glibc2.2.5
- Python version: 3.8.16
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
GitHub issue: ValueError when running BeamBasedBuilder with GCS path in cache_dir
State: closed

### Describe the bug

When running the BeamBasedBuilder with a GCS path specified in the cache_dir, the following ValueError occurs:

```
ValueError: Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: gcs://my-bucket/huggingface_datasets/my_beam_dataset/default/0.0.0/my_beam_dataset-train [while running 'train/Save to parquet/Write/WriteImpl/InitializeWrite']
```

Same error occurs after running `pip install apache-beam[gcp]` as instructed.

### Steps to reproduce the bug

Put `my_beam_dataset.py`:

```python
import datasets


class MyBeamDataset(datasets.BeamBasedBuilder):
    def _info(self):
        features = datasets.Features({"value": datasets.Value("int64")})
        return datasets.DatasetInfo(features=features)

    def _split_generators(self, dl_manager, pipeline):
        return [datasets.SplitGenerator(name=datasets.Split.TRAIN, gen_kwargs={})]

    def _build_pcollection(self, pipeline):
        import apache_beam as beam

        return pipeline | beam.Create([{"value": i} for i in range(10)])
```

Run:
```bash
datasets-cli run_beam my_beam_dataset.py --cache_dir=gs://my-bucket/huggingface_datasets/ --beam_pipeline_options="runner=DirectRunner"
```

### Expected behavior

Running the BeamBasedBuilder with a GCS cache path without any errors.

### Environment info

- `datasets` version: 2.14.4
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- PyArrow version: 9.0.0
- Pandas version: 2.0.3
GitHub issue: custom load dataset to hub
State: closed

### System Info

kaggle notebook

i transformed dataset:
```
dataset = load_dataset("Dahoas/first-instruct-human-assistant-prompt")
```
to 
formatted_dataset: 
```
Dataset({
    features: ['message_tree_id', 'message_tree_text'],
    num_rows: 33143
})
```
but would like to know how to upload to hub

### Who can help?

@ArthurZucker @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

shared above

### Expected behavior

load dataset to hub
GitHub issue: Misalignment between file format specified in configs metadata YAML and the inferred builder
State: closed

There is a misalignment between the format of the `data_files` specified in the configs metadata YAML (CSV):
```yaml
configs:
  - config_name: default
    data_files:
      - split: train
        path: data.csv
``` 
and the inferred builder (JSON). Note there are multiple JSON files in the repo, but they do not appear in the configs metadata YAML.

See: https://huggingface.co/datasets/freddyaboulton/chatinterface_with_image_csv/discussions/1

CC: @freddyaboulton @polinaeterna
GitHub issue: Offline dataset viewer
State: closed

### Feature request

The dataset viewer feature is very nice. It enables to the user to easily view the dataset. However, when working for private companies we cannot always upload the dataset to the hub. Is there a way to create dataset viewer offline? I.e. to run a code that will open some kind of html or something that makes it easy to view the dataset.

### Motivation

I want to easily view my dataset even when it is hosted locally.

### Your contribution

N.A.
GitHub issue: Ignore CI lint rule violation in Pickler.memoize
State: closed

This PR ignores the violation of the lint rule E721 in `Pickler.memoize`.

The lint rule violation was introduced in this PR:
- #3182

@lhoestq is there a reason you did not use `isinstance` instead?

As a hotfix, we just ignore the violation of the lint rule.

Fix #6136.
GitHub issue: Dataset is slower after calling `to_iterable_dataset`
State: open

### Describe the bug

Can anyone explain why looping over a dataset becomes slower after calling `to_iterable_dataset` to convert to `IterableDataset`

### Steps to reproduce the bug

Any dataset after converting to `IterableDataset`

### Expected behavior

Maybe it should be faster on big dataset? I only test on small dataset

### Environment info

- `datasets` version: 2.14.4
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3
GitHub issue: default config name doesn't work when config kwargs are specified.
State: closed

### Describe the bug

https://github.com/huggingface/datasets/blob/12cfc1196e62847e2e8239fbd727a02cbc86ddec/src/datasets/builder.py#L518-L522

If `config_name` is `None`, `DEFAULT_CONFIG_NAME` should be select. But once users pass `config_kwargs` to their customized `BuilderConfig`, the logic is ignored, and dataset cannot select the default config from multiple configs.

### Steps to reproduce the bug

```python
import datasets

datasets.load_dataset('/dataset/with/multiple/config'')  # Ok
datasets.load_dataset('/dataset/with/multiple/config', some_field_in_config='some')  # Err
```

### Expected behavior

Default config behavior should be consistent.

### Environment info

- `datasets` version: 2.14.3
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.17
- Python version: 3.8.15
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 1.5.3
GitHub issue: Fix authentication issues
State: closed

This PR fixes 3 authentication issues:
- Fix authentication when passing `token`.
- Fix authentication in `Audio.decode_example` and `Image.decode_example`.
- Fix authentication to resolve `data_files` in repositories without script.

This PR also fixes our CI so that we properly test when passing `token` and we do not use the token stored in `HfFolder`.

Fix #6126.

## Details


### Fix authentication when passing `token`

See c0a77dc943de68a17f23f141517028c734c78623

The root issue was caused when the `token` was set in an already instantiated `DownloadConfig` and thus not propagated to `self._storage_options`:

  ```python
  download_config.token = token
  ```

As this usage pattern is very common, the fix consists in overriding `DownloadConfig.__setattr__`.

This fixes authentication issues in the following functions:
- `load_dataset` and `load_dataset_builder`
- `Dataset.push_to_hub` and `Dataset.push_to_hub`
- `inspect.get_dataset_config_info`, `inspect.get_dataset_infos` and `inspect.get_dataset_split_names`

### Fix authentication in `Audio.decode_example` and `Image.decode_example`. 

See: 58e62af004b6b8b84dcfd897a4bc71637cfa6c3f

The `token` was not set because the `repo_id` was wrongly tried to be parsed from an HTTP URL (`"http://..."`), instead of an HFFileSystem URL (`"hf://"`)

### Fix authentication to resolve `data_files` in repositories without script

See: e4684fc1032321abf0d494b0c130ea7c82ebda80

This is fixed by passing `download_config` to the function `create_builder_configs_from_metadata_configs`
GitHub issue: Reinforcement Learning and Robotics are not task categories in HF datasets metadata
State: closed

### Describe the bug

In https://huggingface.co/models there are task categories for RL and robotics but none in https://huggingface.co/datasets

Our lab is currently moving our datasets over to hugging face and would like to be able to add those 2 tags

Moreover we see some older datasets that do have that tag, but we can't seem to add it ourselves.

### Steps to reproduce the bug

1. Create a new dataset on Hugging face
2. Try to type reinforcemement-learning or robotics into the tasks categories, it does not allow you to commit

### Expected behavior

Expected to be able to add RL and robotics as task categories as some previous datasets have these tags

### Environment info

N/A
GitHub issue: Datasets crashing runs due to KeyError
State: closed

### Describe the bug

Hi all,

I have been running into a pretty persistent issue recently when trying to load datasets. 
```python
    train_dataset = load_dataset(
        'llama-2-7b-tokenized', 
        split = 'train'
    )
```
I receive a KeyError which crashes the runs.
```
Traceback (most recent call last):
    main()

    train_dataset = load_dataset(
                    ^^^^^^^^^^^^^
    builder_instance = load_dataset_builder(
                       ^^^^^^^^^^^^^^^^^^^^^
    dataset_module = dataset_module_factory(
                     ^^^^^^^^^^^^^^^^^^^^^^^
    raise e1 from None

    ).get_module()
      ^^^^^^^^^^^^
    else get_data_patterns(base_path, download_config=self.download_config)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    return _get_data_files_patterns(resolver)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    data_files = pattern_resolver(pattern)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
    fs, _, _ = get_fs_token_paths(pattern, storage_options=storage_options)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    paths = [f for f in sorted(fs.glob(paths)) if not fs.isdir(f)]
                               ^^^^^^^^^^^^^^

    allpaths = self.find(root, maxdepth=depth, withdirs=True, detail=True, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    for _, dirs, files in self.walk(path, maxdepth, detail=True, **kwargs):

    listing = self.ls(path, detail=True, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    "last_modified": parse_datetime(tree_item["lastCommit"]["date"]),
                                    ~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'lastCommit'
```
Any help would be greatly appreciated.

Thank you,

Enrico

### Steps to reproduce the bug

Load the dataset from the Huggingface hub.
```python
    train_dataset = load_dataset(
        'llama-2-7b-tokenized', 
        split = 'train'
    )
```

### Expected behavior

Loads the dataset.

### Environment info

datasets-2.14.3
CUDA 11.8
Python 3.11
GitHub issue: Upload README via `push_to_hub`
State: closed

### Feature request

`push_to_hub` now allows users to upload datasets programmatically. However, based on the latest doc, we still need to open the dataset page to add readme file manually.

However, I do discover snippets to intialize a README for every `push_to_hub`:
```
dataset_card = (
    DatasetCard(
        "---\n"
        + str(dataset_card_data)
        + "\n---\n"
        + f'# Dataset Card for "{repo_id.split("/")[-1]}"\n\n[More Information needed](https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)'
    )
    if dataset_card is None
    else dataset_card
)

HfApi(endpoint=config.HF_ENDPOINT).upload_file(
    path_or_fileobj=str(dataset_card).encode(),
    path_in_repo="README.md",
    repo_id=repo_id,
    token=token,
    repo_type="dataset",
    revision=branch,
)
```

So, if we can enable `push_to_hub` to upload a readme file by ourselves instead of using the auto generated ones, it can save ton of time, and will definitely alleviate the current "lack-of-dataset-card" situation.

### Motivation

as elabrated above.

### Your contribution

I might be able to make a pr.
GitHub issue: Lookahead streaming support?
State: open

### Feature request

From what I understand, streaming dataset currently pulls the data, and process the data as it is requested.
This can introduce significant latency delays when data is loaded into the training process, needing to wait for each segment.

While the delays might be dataset specific (or even mapping instruction/tokenizer specific)

Is it possible to introduce a `streaming_lookahead` parameter, which is used for predictable workloads (even shuffled dataset with fixed seed). As we can predict in advance what the next few datasamples will be. And fetch them while the current set is being trained.

With enough CPU & bandwidth to keep up with the training process, and a sufficiently large lookahead, this will reduce the various latency involved while waiting for the dataset to be ready between batches.

### Motivation

Faster streaming performance, while training over extra large TB sized datasets

### Your contribution

I currently use HF dataset, with pytorch lightning trainer for RWKV project, and would be able to help test this feature if supported.
GitHub issue: [Docs] The "Process" how-to guide lacks description of `select_columns` function
State: closed

### Feature request

The [how to process dataset guide](https://huggingface.co/docs/datasets/main/en/process) currently does not mention the [`select_columns`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.select_columns) function. It would be nice to include it in the guide.

### Motivation

This function is a commonly requested feature (see this [forum thread](https://discuss.huggingface.co/t/how-to-create-a-new-dataset-from-another-dataset-and-select-specific-columns-and-the-data-along-with-the-column/15120) and #5468 #5474). However, it has not been included in the guide since its implementation by PR #5480.

Mentioning it in the guide would help future users discover this added feature.

### Your contribution

I could submit a PR to add a brief description of the function to said guide.
GitHub issue: Cache not being used when loading commonvoice 8.0.0
State: closed

### Describe the bug

I have commonvoice 8.0.0 downloaded in `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/b2f8b72f8f30b2e98c41ccf855954d9e35a5fa498c43332df198534ff9797a4a`.  The folder contains all the arrow files etc, and was used as the cached version last time I touched the ec2 instance I'm working on.  Now, with the same command that downloaded it initially:

```
dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")
```

it tries to redownload the dataset to `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/05bdc7940b0a336ceeaeef13470c89522c29a8e4494cbeece64fb472a87acb32`

### Steps to reproduce the bug

Steps to reproduce the behavior:

1.  ```dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")```
2. dataset is updated by maintainers
3. ```dataset = load_dataset("mozilla-foundation/common_voice_8_0", "en", use_auth_token="<mytoken>")```

### Expected behavior

I expect that it uses the already downloaded data in `~/.cache/huggingface/datasets/mozilla-foundation___common_voice_8_0/en/8.0.0/b2f8b72f8f30b2e98c41ccf855954d9e35a5fa498c43332df198534ff9797a4a`.

Not sure what's happening in 2. but if, say it's an issue with the dataset referenced by "mozilla-foundation/common_voice_8_0" being modified by the maintainers, how would I force datasets to point to the original version I downloaded?

EDIT:  It was indeed that the maintainers had updated the dataset (v 8.0.0).  However I still cant load the dataset from disk instead of redownloading, with for example:

```
load_dataset(".cache/huggingface/datasets/downloads/extracted/<hash>/cv-corpus-8.0-2022-01-19/en/", "en")

> ...
> File [~/miniconda3/envs/aa_torch2/lib/python3.10/site-packages/datasets/table.py:1938](.../ python3.10/site-packages/datasets/table.py:1938), in cast_array_to_feature(array, feature, allow_number_to_str)
   1937 elif not isinstance(feature, (Sequence, dict, list, tuple)):
-> 1938     return array_cast(array, feature(), allow_number_to_str=allow_number_to_str)
...
   1794         e = e.__context__
-> 1795     raise DatasetGenerationError("An error occurred while generating the dataset") from e
   1797 yield job_id, True, (total_num_examples, total_num_bytes, writer._features, num_shards, shard_lengths)

DatasetGenerationError: An error occurred while generating the dataset
```


### Environment info

datasets==2.7.0
python==3.10.8
OS:  AWS Linux
GitHub issue: load_dataset() fails with streamlit caching inside docker
State: closed

### Describe the bug

When calling `load_dataset` in a streamlit application running within a docker container, get a failure with the error message:

EmptyDatasetError: The directory at hf://datasets/fetch-rewards/inc-rings-2000@bea27cf60842b3641eae418f38864a2ec4cde684 doesn't contain any data files
Traceback:
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script
    exec(code, module.__dict__)
File "/home/user/app/app.py", line 62, in <module>
    dashboard()
File "/home/user/app/app.py", line 47, in dashboard
    feat_dict, path_gml = load_data(hf_repo, model_gml_dict[selected_model], hf_token)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 211, in wrapper
    return cached_func(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 240, in __call__
    return self._get_or_create_cached_value(args, kwargs)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 266, in _get_or_create_cached_value
    return self._handle_cache_miss(cache, value_key, func_args, func_kwargs)
File "/opt/conda/lib/python3.10/site-packages/streamlit/runtime/caching/cache_utils.py", line 320, in _handle_cache_miss
    computed_value = self._info.func(*func_args, **func_kwargs)
File "/home/user/app/hf_interface.py", line 16, in load_data
    hf_dataset = load_dataset(repo_id, use_auth_token=hf_token)
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 2109, in load_dataset
    builder_instance = load_dataset_builder(
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1795, in load_dataset_builder
    dataset_module = dataset_module_factory(
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1486, in dataset_module_factory
    raise e1 from None
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1476, in dataset_module_factory
    ).get_module()
File "/opt/conda/lib/python3.10/site-packages/datasets/load.py", line 1032, in get_module
    else get_data_patterns(base_path, download_config=self.download_config)
File "/opt/conda/lib/python3.10/site-packages/datasets/data_files.py", line 458, in get_data_patterns
    raise EmptyDatasetError(f"The directory at {base_path} doesn't contain any data files") from None

### Steps to reproduce the bug

```python
@st.cache_resource
def load_data(repo_id: str, hf_token=None):
    """Load data from HuggingFace Hub
    """
    
    hf_dataset = load_dataset(repo_id, use_auth_token=hf_token)
    hf_dataset = hf_dataset.map(lambda x: json.loads(x["ground_truth"]), remove_columns=["ground_truth"])

    return hf_dataset
```

### Expected behavior

Expect to load. 
Note: works fine with datasets==2.13.1

### Environment info

datasets==2.14.2, 
Ubuntu bionic-based Docker container.
GitHub issue: raise FileNotFoundError("Directory {dataset_path} is neither a `Dataset` directory nor a `DatasetDict` directory." )
State: closed

### Describe the bug

For researchers in some countries or regions, it is usually the case that the download ability of `load_dataset` is disabled due to the complex network environment. People in these regions often prefer to use git clone or other programming tricks to manually download the files to the disk (for example, [How to elegantly download hf models, zhihu zhuanlan](https://zhuanlan.zhihu.com/p/475260268) proposed a crawlder based solution,  and [Is there any mirror for hf_hub, zhihu answer](https://www.zhihu.com/question/371644077) provided some cloud based solutions, and [How to avoid pitfalls on Hugging face downloading, zhihu zhuanlan] gave some useful suggestions), and then use `load_from_disk` to get the dataset object. 
However, when one finally has the local files on the disk, it is still buggy when trying to load the files into objects. 

### Steps to reproduce the bug

Steps to reproduce the bug:
1. Found CIFAR dataset in hugging face: https://huggingface.co/datasets/cifar100/tree/main
2. Click ":" button to show "Clone repository" option, and then follow the prompts on the box:
 ```bash
  cd my_directory_absolute
  git lfs install
  git clone https://huggingface.co/datasets/cifar100
  ls  my_directory_absolute/cifar100 # confirm that the directory exists and it is OK. 
  ```
3. Write A python file to try to load the dataset
```python
from datasets import load_dataset, load_from_disk
dataset = load_from_disk("my_directory_absolute/cifar100")
```
Notice that according to issue #3700 , it is wrong to use load_dataset("my_directory_absolute/cifar100"), so we must use load_from_disk instead. 

4. Then you will see the error reported:
```log
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[5], line 9
      1 from datasets import load_dataset, load_from_disk
----> 9 dataset = load_from_disk("my_directory_absolute/cifar100")

File [~/miniconda3/envs/ai/lib/python3.10/site-packages/datasets/load.py:2232), in load_from_disk(dataset_path, fs, keep_in_memory, storage_options)
   2230     return DatasetDict.load_from_disk(dataset_path, keep_in_memory=keep_in_memory, storage_options=storage_options)
   2231 else:
-> 2232     raise FileNotFoundError(
   2233         f"Directory {dataset_path} is neither a `Dataset` directory nor a `DatasetDict` directory."
   2234     )

FileNotFoundError: Directory my_directory_absolute/cifar100 is neither a `Dataset` directory nor a `DatasetDict` directory.
```

### Expected behavior

The dataset should be load successfully. 

### Environment info

```bash
datasets-cli env
```
-> results:
```txt

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.14.2
- Platform: Linux-4.18.0-372.32.1.el8_6.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
```
GitHub issue: [BUG] Dataset initialized from in-memory data does not create cache.
State: closed

### Describe the bug

`Dataset` initialized from in-memory data (dictionary in my case, haven't tested with other types) does not create cache when processed with the `map` method, unlike `Dataset` initialized by other methods such as `load_dataset`.

### Steps to reproduce the bug

```python
# below code was run the second time so the map function can be loaded from cache if exists

from datasets import load_dataset, Dataset

dataset = load_dataset("tatsu-lab/alpaca")['train']

dataset = dataset.map(lambda x: {'input': x['input'] + 'hi'}) # some random map
print(len(dataset.cache_files))
# 1

# copy the exact same data but initialize from a dictionary
memory_dataset = Dataset.from_dict({
    'instruction': dataset['instruction'],
    'input': dataset['input'],
    'output': dataset['output'],
    'text': dataset['text']})

memory_dataset = memory_dataset.map(lambda x: {'input': x['input'] + 'hi'}) # exact same map
print(len(memory_dataset.cache_files))
# Map: 100%|██████████| 52002[/52002]
# 0
```

### Expected behavior

The `map` function should create cache regardless of the method the `Dataset` was created.

### Environment info

- `datasets` version: 2.14.2
- Platform: Linux-5.15.0-41-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- PyArrow version: 11.0.0
- Pandas version: 1.5.3
GitHub issue: Problems in downloading Amazon reviews from HF
State: closed

### Describe the bug

I have a script downloading `amazon_reviews_multi`.

When the download starts, I get

```
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 243B [00:00, 1.43MB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.54s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 842.40it/s]
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 243B [00:00, 928kB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.42s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 832.70it/s]
Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]
Downloading data: 243B [00:00, 1.81MB/s]
Downloading data files: 100%|██████████| 1/1 [00:01<00:00,  1.40s/it]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 1294.14it/s]
Generating train split:   0%|          | 0/200000 [00:00<?, ? examples/s]
```

the file is clearly too small to contain the requested dataset, in fact it contains en error message:

```
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>AGJWSY3ZADT2QVWE</RequestId><HostId>Gx1O2KXnxtQFqvzDLxyVSTq3+TTJuTnuVFnJL3SP89Yp8UzvYLPTVwd1PpniE4EvQzT3tCaqEJw=</HostId></Error>
```

obviously the script fails:

```
>           raise DatasetGenerationError("An error occurred while generating the dataset") from e
E           datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```



### Steps to reproduce the bug

1. load_dataset("amazon_reviews_multi", name="en", split="train", cache_dir="ADDYOURPATHHERE")

### Expected behavior

I would expect the dataset to be downloaded and processed

### Environment info

* The problem is present with both datasets 2.12.0 and 2.14.2
* python version 3.10.12
GitHub issue: Fix deprecation of use_auth_token in file_utils
State: closed

Fix issues with the deprecation of `use_auth_token`  introduced by:
- #5996

in functions:
- `get_authentication_headers_for_url`
- `request_etag`
- `get_from_cache`

Currently, `TypeError` is raised: https://github.com/huggingface/datasets-server/actions/runs/5711650666/job/15484685570?pr=1588
```
FAILED tests/job_runners/config/test_parquet_and_info.py::test__is_too_big_external_files[None-None-False] - TypeError: get_authentication_headers_for_url() got an unexpected keyword argument 'use_auth_token'

FAILED tests/job_runners/config/test_parquet_and_info.py::test_fill_builder_info[None-False] - libcommon.exceptions.FileSystemError: Could not read the parquet files: get_authentication_headers_for_url() got an unexpected keyword argument 'use_auth_token'
```

Related to:
- #6094
GitHub issue: load local json_file as dataset
State: closed

### Describe the bug

I tried to load local json file as dataset but failed to parsing json file because some columns are 'float' type.

### Steps to reproduce the bug

1. load json file with certain columns are 'float' type. For example `data = load_data("json", data_files=JSON_PATH)`
2. Then, the error will be triggered like `ArrowInvalid: Could not convert '-0.2253' with type str: tried to convert to double

### Expected behavior

Should allow some columns are 'float' type, at least it should convert those columns to str type.
I tried to avoid the error by naively convert the float item to str:
```python
# if col type is not str, we need to convert it to str
mapping = {}
for col in keys:
    if isinstance(dataset[0][col], str):
        mapping[col] = [row.get(col) for row in dataset]
    else:
        mapping[col] = [str(row.get(col)) for row in dataset]
```

### Environment info

- `datasets` version: 2.14.2
- Platform: Linux-5.4.0-52-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.0
- Pandas version: 2.0.1
GitHub issue: HF Datasets data access is extremely slow even when in memory
State: open

### Describe the bug

Doing a simple `some_dataset[:10]` can take more than a minute.

Profiling it:
<img width="1280" alt="image" src="https://github.com/huggingface/datasets/assets/36224762/e641fb95-ff02-4072-9016-5416a65f75ab">

`some_dataset` is completely in memory with no disk cache.

This is proving fatal to my usage of HF Datasets. Is there a way I can forgo the arrow format and store the dataset as PyTorch tensors so that `_tensorize` is not needed? And is `_consolidate` supposed to take this long?

It's faster to produce the dataset from scratch than to access it from HF Datasets!

### Steps to reproduce the bug

I have uploaded the dataset that causes this problem [here](https://huggingface.co/datasets/NightMachinery/hf_datasets_bug1).

```python
#!/usr/bin/env python3
import sys
import time
import torch
from datasets import load_dataset


def main(dataset_name):
    # Start the timer
    start_time = time.time()

    # Load the dataset from Hugging Face Hub
    dataset = load_dataset(dataset_name)

    # Set the dataset format as torch
    dataset.set_format(type="torch")

    # Perform an identity map
    dataset = dataset.map(lambda example: example, batched=True, batch_size=20)

    # End the timer
    end_time = time.time()

    # Print the time taken
    print(f"Time taken: {end_time - start_time:.2f} seconds")


if __name__ == "__main__":
    dataset_name = "NightMachinery/hf_datasets_bug1"
    print(f"dataset_name: {dataset_name}")
    main(dataset_name)
```

### Expected behavior

_

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-5.15.0-76-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3
GitHub issue: TypeError when loading from GCP bucket
State: closed

### Describe the bug

Loading a dataset from a GCP bucket raises a type error. This bug was introduced recently (either in 2.14 or 2.14.1), and appeared during a migration from 2.13.1.

### Steps to reproduce the bug

Load any file from a GCP bucket:

```python
import datasets

datasets.load_dataset("json", data_files=["gs://..."])
```

The following exception is raised:

```python
Traceback (most recent call last):
...
packages/datasets/data_files.py", line 335, in resolve_pattern
    protocol_prefix = fs.protocol + "://" if fs.protocol != "file" else ""
TypeError: can only concatenate tuple (not "str") to tuple
```

With a `GoogleFileSystem`, the attribute `fs.protocol` is a tuple `('gs', 'gcs')` and hence cannot be concatenated with a string.


### Expected behavior

The file should be loaded without exception.

### Environment info

- `datasets` version: 2.14.1
- Platform: macOS-13.2.1-x86_64-i386-64bit
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- PyArrow version: 12.0.1
- Pandas version: 2.0.3