sunny2309/githug-issues huggingface dataset:
Dataset.map() execute twice when in PyTorch DDP mode

### Describe the bug

I use `torchrun --standalone --nproc_per_node=2 train.py` to start training. And write the code following the [docs](https://huggingface.co/docs/datasets/process#distributed-usage). The trick about using `torch.distributed.barrier()` to only execute map at the main process doesn't always work. When I am training model, it will map twice. When I am running a test for dataset and dataloader (just print the batches), it can work. Their code about loading dataset are same.

And on another server with 30 CPU cores, I use 2 GPUs and it can't work neither. 

I have tried to use `rank` and `local_rank` to check, they all didn't make sense.

### Steps to reproduce the bug

use `torchrun --standalone --nproc_per_node=2 train.py` or `torchrun --standalone train.py` to run

This is my code:

```python
            if args.distributed and world_size > 1:
                if args.local_rank > 0:
                    print(f"Rank {args.rank}: Gpu {args.gpu} waiting for main process to perform the mapping", force=True)
                    torch.distributed.barrier()
                print("Mapping dataset")
                dataset = dataset.map(lambda x: cut_reorder_keys(x, num_stations_list=args.num_stations_list, is_pad=True, is_train=True), num_proc=8, desc="cut_reorder_keys")
                dataset = dataset.map(lambda x: random_shift(x, shift_range=(-160, 0), feature_scale=16), num_proc=8, desc="random_shift")
                dataset_test = dataset_test.map(lambda x: cut_reorder_keys(x, num_stations_list=args.num_stations_list, is_pad=True, is_train=False), num_proc=8, desc="cut_reorder_keys")
                if args.local_rank == 0:
                    print("Mapping finished, loading results from main process")
                    torch.distributed.barrier()
```


### Expected behavior

Only the main process will execute `map`, while the sub process will load cache from disk.

### Environment info

server with 64 CPU cores (AMD Ryzen Threadripper PRO 5995WX 64-Cores) and 2 RTX 4090

- `python==3.9.16`
- `datasets==2.13.1`
- `torch==2.0.1+cu117`
- `22.04.1-Ubuntu`

server with 30 CPU cores (Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz) and 2 RTX 4090

- `python==3.9.0`
- `datasets==2.13.1`
- `torch==2.0.1+cu117`
- `Ubuntu 20.04`
Why is the speed difference of gen example so big?

```python
def _generate_examples(self, metadata_path, images_dir, conditioning_images_dir):
        with open(metadata_path, 'r') as file:
            metadata = json.load(file)

        for idx, item in enumerate(metadata):
            image_path = item.get('image_path')
            text_content = item.get('text_content')
            image_data = open(image_path, "rb").read()
            yield idx, {
                "text": text_content,
                "image": {
                    "path": image_path,
                    "bytes": image_data,
                },
                "conditioning_image": {
                    "path": image_path,
                    "bytes": image_data,
                },
            }
```
Hello, 

I use the above function to deal with my local data set, but I am very surprised that the speed at which I generate example is very different. When I start a training task, **sometimes 1000examples/s, sometimes only 10examples/s.**

![image](https://github.com/huggingface/datasets/assets/46072190/cdc17661-8267-4fd8-b30c-b74d505efd9b)

I'm not saying that speed is changing all the time. I mean, the reading speed is different in different training, which will cause me to start training over and over again until the speed of this generation of examples is normal.
Implement proper checkpointing for dataset uploading with resume function that does not require remapping shards that have already been uploaded

Context: issue #5990

In order to implement the checkpointing, I introduce a metadata folder that keeps one yaml file for each set that one is uploading. This yaml keeps track of what shards have already been uploaded, and which one the idx of the latest one was. Using this information I am then able to easily get the push_to_hub function to retrieve on demand past history of uploads and continue mapping and uploading from where it was left off.
Multi-processed `Dataset.map` slows down a lot when `import torch`

### Describe the bug

When using `Dataset.map` with `num_proc > 1`, the speed slows down much if I add `import torch` to the start of the script even though I don't use it.

I'm not sure if it's `torch` only or if any other package that is "large" will also cause the same result.
BTW, `import lightning` also slows it down.

Below are the progress bars of `Dataset.map`, the only difference between them is with or without `import torch`, but the speed varies by 6-7 times.

- without `import torch` ![image](https://github.com/huggingface/datasets/assets/47121592/0233055a-ced4-424a-9f0f-32a2afd802c2)
- with `import torch` ![image](https://github.com/huggingface/datasets/assets/47121592/463eafb7-b81e-4eb9-91ca-fd7fe20f3d59)


### Steps to reproduce the bug

Below is the code I used, but I don't think the dataset and the mapping function have much to do with the phenomenon.

```python3
from datasets import load_from_disk, disable_caching
from transformers import AutoTokenizer
# import torch
# import lightning

def rearrange_datapoints(
    batch,
    tokenizer,
    sequence_length,
):
    datapoints = []
    
    input_ids = []
    for x in batch['input_ids']:
        input_ids += x
        while len(input_ids) >= sequence_length:
            datapoint = input_ids[:sequence_length]
            datapoints.append(datapoint)
            input_ids[:sequence_length] = []

    if input_ids:
        paddings = [-1] * (sequence_length - len(input_ids))
        datapoint = paddings + input_ids if tokenizer.padding_side == 'left' else input_ids + paddings
        datapoints.append(datapoint)
    
    batch['input_ids'] = datapoints
    return batch


if __name__ == '__main__':
    disable_caching()

    tokenizer = AutoTokenizer.from_pretrained('...', use_fast=False)
    dataset = load_from_disk('...')
    
    dataset = dataset.map(
        rearrange_datapoints,
        fn_kwargs=dict(
            tokenizer=tokenizer,
            sequence_length=2048,
        ),
        batched=True,
        num_proc=8,
    )

```

### Expected behavior

The multi-processed `Dataset.map` function speed between with and without `import torch` should be the same.

### Environment info

- `datasets` version: 2.13.1
- Platform: Linux-3.10.0-1127.el7.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- PyArrow version: 12.0.0
- Pandas version: 2.0.1
Change package name from "datasets" to something less generic

### Feature request

I'm repeatedly finding myself in situations where I want to have a package called `datasets.py` or `evaluate.py` in my code and can't because those names are being taken up by Huggingface packages. While I can understand how (even from the user's perspective) it's aesthetically pleasing to have nice terse library names, ultimately a library hogging simple names like this is something I find short-sighted, impractical and at my most irritable, frankly rude.

My preference would be a pattern like what you get with all the other big libraries like numpy or pandas:

```
import huggingface as hf
# hf.transformers, hf.datasets, hf.evaluate
```

or things like

```
import huggingface.transformers as tf
# tf.load_model(), etc
```

If this isn't possible for some technical reason, at least just call the packages something like `hf_transformers` and so on.

I realize this is a very big change that's probably been discussed internally already, but I'm making this issue and sister issues on each huggingface project just to start the conversation and begin tracking community feeling on the matter, since I suspect I'm not the only one who feels like this.

Sorry if this has been requested already on this issue tracker, I couldn't find anything looking for terms like "package name".

Sister issues:
- [transformers](https://github.com/huggingface/transformers/issues/24934)
- **datasets**
- [evaluate](https://github.com/huggingface/evaluate/issues/476)

### Motivation

Not taking up package names the user is likely to want to use.

### Your contribution

No - more a matter of internal discussion among core library authors.
Skipping shard in the remote repo and resume upload

### Describe the bug

For some reason when I try to resume the upload of my dataset, it is very slow to reach the index of the shard from which to resume the uploading.

From my understanding, the problem is in this part of the code:

arrow_dataset.py
```python
for index, shard in logging.tqdm(
            enumerate(itertools.chain([first_shard], shards_iter)),
            desc="Pushing dataset shards to the dataset hub",
            total=num_shards,
            disable=not logging.is_progress_bar_enabled(),
        ):
            shard_path_in_repo = path_in_repo(index, shard)
            # Upload a shard only if it doesn't already exist in the repository
            if shard_path_in_repo not in data_files:
```

In particular, iterating the generator is slow during the call:
```python
self._select_contiguous(start, length, new_fingerprint=new_fingerprint)
```

I wonder if it is possible to avoid calling this function for shards that are already uploaded and just start from the correct shard index.

### Steps to reproduce the bug

1. Start the upload
```python
dataset = load_dataset("imagefolder", data_dir=DATA_DIR, split="train", drop_labels=True)    
dataset.push_to_hub("repo/name")
```
2. Stop and restart the upload after hundreds of shards

### Expected behavior

Skip the uploaded shards faster.

### Environment info

- `datasets` version: 2.5.1
- Platform: Linux-4.18.0-193.el8.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- PyArrow version: 12.0.1
- Pandas version: 2.0.2
when i use datasets.load_dataset, i encounter the http connect error!

### Describe the bug

`common_voice_test = load_dataset("audiofolder", data_dir="./dataset/",cache_dir="./cache",split=datasets.Split.TEST)`
when i run the code above, i got the error as below:
--------------------------------------------
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/2.3.2/datasets/audiofolder/audiofolder.py (ConnectionError(MaxRetryError("HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /huggingface/datasets/2.3.2/datasets/audiofolder/audiofolder.py (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f299ed082e0>: Failed to establish a new connection: [Errno 101] Network is unreachable'))")))


--------------------------------------------------
My all data is on local machine, why does it need to connect the internet? how can i fix it, because my machine cannot connect the internet.

### Steps to reproduce the bug

1

### Expected behavior

no error when i use the load_dataset func

### Environment info

python=3.8.15
Support proxy and user-agent in fsspec calls

Since we switched to the new HfFileSystem we no longer apply user's proxy and user-agent.

Using the HTTP_PROXY and HTTPS_PROXY environment variables works though since we use aiohttp to call the HF Hub.

This can be implemented in `_prepare_single_hop_path_and_storage_options`.

Though ideally the `HfFileSystem` could support passing at least the proxies
Compression kwargs have no effect when saving datasets as csv

### Describe the bug

Attempting to save a dataset as a compressed csv file, the compression kwargs provided to `.to_csv()` that get piped to panda's `pandas.DataFrame.to_csv` do not have any effect - resulting in the dataset not getting compressed.

A warning is raised if explicitly providing a `compression` kwarg, but no warnings are raised if relying on the defaults. This can lead to datasets secretly not getting compressed for users expecting the behaviour to match panda's `.to_csv()`, where the compression format is automatically inferred from the destination path suffix.

### Steps to reproduce the bug

```python
# dataset is not compressed (but at least a warning is emitted)

import datasets

dataset = datasets.load_dataset("rotten_tomatoes", split="train")

dataset.to_csv("uncompressed.csv")
print(os.path.getsize("uncompressed.csv"))  # 1008607

dataset.to_csv("compressed.csv.gz",  compression={'method': 'gzip', 'compresslevel': 1, 'mtime': 1})
print(os.path.getsize("compressed.csv.gz"))  # 1008607
```

```shell
>>>
RuntimeWarning: compression has no effect when passing a non-binary object as input.
  csv_str = batch.to_pandas().to_csv(
```

```python
# dataset is not compressed and no warnings are emitted

dataset.to_csv("compressed.csv.gz")
print(os.path.getsize("compressed.csv.gz"))  # 1008607

# compare with
dataset.to_pandas().to_csv("pandas.csv.gz")
print(os.path.getsize("pandas.csv.gz"))  # 418561
```

---

I think that this is because behind the scenes `pandas.DataFrame.to_csv` is always called with a buf-like `path_or_buf`, but users that are providing a path-like to `datasets.Dataset.to_csv` are likely not to expect / know that - leading to a mismatch in their understanding of the expected behaviour of the `compression` kwarg.

### Expected behavior

The dataset to be saved as a compressed csv file when providing a `compression` kwarg, or when relying on the default `compression='infer'`

### Environment info

`datasets == 2.13.1`
File "/home/zhizhou/anaconda3/envs/pytorch/lib/python3.10/site-packages/datasets/builder.py", line 992, in _download_and_prepare     if str(split_generator.split_info.name).lower() == "all": AttributeError: 'str' object has no attribute 'split_info'. Did you mean: 'splitlines'?

Hi, I use the code below to load local file
```
    def _split_generators(self, dl_manager):
        # TODO: This method is tasked with downloading/extracting the data and defining the splits depending on the configuration
        # If several configurations are possible (listed in BUILDER_CONFIGS), the configuration selected by the user is in self.config.name

        # dl_manager is a datasets.download.DownloadManager that can be used to download and extract URLS
        # It can accept any type or nested list/dict and will give back the same structure with the url replaced with path to local files.
        # By default the archives will be extracted and a path to a cached folder where they are extracted is returned instead of the archive
        # urls = _URLS[self.config.name]
        data_dir = dl_manager.download_and_extract(_URLs)
        print(data_dir)
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN,
                # These kwargs will be passed to _generate_examples
                gen_kwargs={
                    "filepath": os.path.join(data_dir["train"]),
                    "split": "train",
                },
            ),
            datasets.SplitGenerator(
                name=datasets.Split.VALIDATION,
                # These kwargs will be passed to _generate_examples
                gen_kwargs={
                    "filepath": os.path.join(data_dir["dev"]),
                    "split": "dev",
                },
            ),
        ]
```
and error occured
```

Traceback (most recent call last):
  File "/home/zhizhou/data1/zhanghao/huggingface/FineTuning_Transformer/load_local_dataset.py", line 2, in <module>
    dataset = load_dataset("./QA_script.py",data_files='/home/zhizhou/.cache/huggingface/datasets/conversatiom_corps/part_file.json')
  File "/home/zhizhou/anaconda3/envs/pytorch/lib/python3.10/site-packages/datasets/load.py", line 1809, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/zhizhou/anaconda3/envs/pytorch/lib/python3.10/site-packages/datasets/builder.py", line 909, in download_and_prepare
    self._download_and_prepare(
  File "/home/zhizhou/anaconda3/envs/pytorch/lib/python3.10/site-packages/datasets/builder.py", line 1670, in _download_and_prepare
    super()._download_and_prepare(
  File "/home/zhizhou/anaconda3/envs/pytorch/lib/python3.10/site-packages/datasets/builder.py", line 992, in _download_and_prepare
    if str(split_generator.split_info.name).lower() == "all":
AttributeError: 'str' object has no attribute 'split_info'. Did you mean: 'splitlines'?
```
Could you help me?
Documentation links to examples are broken

### Describe the bug

The links at the bottom of [add_dataset](https://huggingface.co/docs/datasets/v1.2.1/add_dataset.html) to examples of specific datasets are all broken, for example

- text classification: [ag_news](https://github.com/huggingface/datasets/blob/master/datasets/ag_news/ag_news.py) (original data are in csv files)

### Steps to reproduce the bug

Click on links to examples from latest documentation

### Expected behavior

Links should be up to date - it might be more stable to link to https://huggingface.co/datasets/ag_news/blob/main/ag_news.py

### Environment info

dataset v1.2.1
Deprecate search API

The Search API only supports Faiss and ElasticSearch as vector stores, is somewhat difficult to maintain (e.g., it still doesn't support ElasticSeach 8.0, difficult testing, ...), does not have the best design (adds a bunch of methods to the `Dataset` class that are only useful after creating an index), the usage doesn't seem to be significant and is not integrated with the Hub. Since we have no plans/bandwidth to improve it and better alternatives such as `langchain` and `docarray` exist, I think it should be deprecated (and eventually removed).

If we decide to deprecate/remove it, the following usage instances need to be addressed:
* [Course](https://github.com/huggingface/course/blob/0018bb434204d9750a03592cb0d4e846093218d8/chapters/en/chapter5/6.mdx#L342 ) and [Blog](https://github.com/huggingface/blog/blob/4897c6f73d4492a0955ade503281711d01840e09/image-search-datasets.md?plain=1#L252) - calling the FAISS API directly should be OK in these instances as it's pretty simple to use for basic scenarios. Alternatively, we can use `langchain`, but this adds an extra dependency
*  [Transformers](https://github.com/huggingface/transformers/blob/50726f9ea7afc6113da617f8f4ca1ab264a5e28a/src/transformers/models/rag/retrieval_rag.py#L183) - we can use the FAISS API directly and store the index as a separate attribute (and instead of building the `wiki_dpr` index each time the dataset is generated, we can generate it once and push it to the Hub repo, and then read it from there

cc @huggingface/datasets @LysandreJik for the opinion
load_dataset hangs on WSL

### Describe the bug

load_dataset simply hangs. It happens once every ~5 times, and interestingly hangs for a multiple of 5 minutes (hangs for 5/10/15 minutes). Using the profiler in PyCharm shows that it spends the time at <method 'connect' of '_socket.socket' objects>. However, a local cache is available so I am not sure why socket is needed. ([profiler result](https://ibb.co/0Btbbp8))

It only happens on WSL for me. It works for native Windows and my MacBook. (cache quickly recognized and loaded within a second).



### Steps to reproduce the bug

I am using Ubuntu 22.04.2 LTS (GNU/Linux 5.15.90.1-microsoft-standard-WSL2 x86_64)

Python 3.10.10 (main, Mar 21 2023, 18:45:11) [GCC 11.2.0] on linux
>>> import datasets
>>> datasets.load_dataset('ai2_arc', 'ARC-Challenge')  # hangs for 5/10/15 minutes

### Expected behavior

cache quickly recognized and loaded within a second

### Environment info

Please let me know if I should provide more environment information.
DownloadConfig.proxies not work when load_dataset_builder calling HfApi.dataset_info  

### Describe the bug

```python
download_config = DownloadConfig(proxies={'https': '<my proxy>'})
builder = load_dataset_builder(..., download_config=download_config)
```
But, when getting the dataset_info from HfApi, the http requests not using the proxies.

### Steps to reproduce the bug

1. Setup proxies in DownloadConfig.
2. Call `load_dataset_build` with download_config.
3. Inspect the call stack in HfApi.dataset_info.

![image](https://github.com/huggingface/datasets/assets/138426806/33e538a8-2e22-4e63-b634-343febe5324b)

### Expected behavior

DownloadConfig.proxies works for getting dataset_info.

### Environment info

https://github.com/huggingface/datasets/commit/406b2212263c0d33f267e35b917f410ff6b3bc00
Python 3.11.4
Argument type for map function changes when using `input_columns` for `IterableDataset`

### Describe the bug

I wrote `tokenize(examples)` function as an argument for `map` function for `IterableDataset`.
It process dictionary type `examples` as a parameter.
It is used in `train_dataset = train_dataset.map(tokenize, batched=True)`
No error is raised.

And then, I found some unnecessary keys and values in `examples` so I added `input_columns` argument to `map` function to select keys and values.
It gives me an error saying
```
TypeError: tokenize() takes 1 positional argument but 3 were given.
```

The code below matters.
https://github.com/huggingface/datasets/blob/406b2212263c0d33f267e35b917f410ff6b3bc00/src/datasets/iterable_dataset.py#L687

For example, `inputs = {"a":1, "b":2, "c":3}`.
If `self.input_coluns` is `None`,
`inputs` is a dictionary type variable and `function_args` becomes a `list` of a single `dict` variable.
`function_args` becomes `[{"a":1, "b":2, "c":3}]`

Otherwise, lets say `self.input_columns = ["a", "c"]`
`[inputs[col] for col in self.input_columns]` results in `[1, 3]`.
I think it should be `[{"a":1, "c":3}]`.

I want to ask if the resulting format is intended.
Maybe I can modify `tokenize()` to have 2 parameters in this case instead of having 1 dictionary.
But this is confusing to me.
Or it should be fixed as `[{col:inputs[col] for col in self.input_columns}]`

### Steps to reproduce the bug

Run `map` function of `IterableDataset` with `input_columns` argument.

### Expected behavior

`function_args` looks better to have same format.
I think it should be `[{"a":1, "c":3}]`.

### Environment info

dataset version: 2.12
python: 3.8
Use new hffs

Thanks to @janineguo 's work in https://github.com/huggingface/datasets/pull/5919 which was needed to support HfFileSystem.

Switching to `HfFileSystem` will help implementing optimization in data files resolution

## Implementation details

I replaced all the from_hf_repo and from_local_or_remote in data_files.py to only use a new `from_patterns` which works for any fsspec path, including hf:// paths, https:// URLs and local paths. This simplifies the codebase since there is no logic duplication anymore when it comes to data files resolution.

I added `_prepare_path_and_storage_options` which returns the right storage_options to use given a path and a `DownloadConfig`. This is the only place where the logic depends on the filesystem type that must be used.

I also removed the `get_metadata_data_files_list ` and `get_patterns_and_data_files` functions added recently, since data files resolution is now handled using a common interface.

## New features

hf:// paths are now supported in data_files

## Breaking changes

DataFilesList and DataFilesDict:
- use `str` paths instead of `Union[Path, Url]`
- require posix paths for windows paths

close https://github.com/huggingface/datasets/issues/6017
[docs] Update return statement of index search

Clarifies in the return statement of the docstring that the retrieval score is `IndexFlatL2` by default (see [PR](https://github.com/huggingface/transformers/issues/24739) and internal Slack [convo](https://huggingface.slack.com/archives/C01229B19EX/p1689105179711689)), and fixes the formatting because multiple return values are not supported.