sunny2309/githug-issues huggingface dataset:
Dataset configuration

Following discussions from https://github.com/huggingface/datasets/pull/5331

We could have something like `config.json` to define the configuration of a dataset.

```json
{
    "data_dir": "data"
    "data_files":  {
        "train": "train-[0-9][0-9][0-9][0-9]-of-[0-9][0-9][0-9][0-9][0-9]*.*"
    }
}
```

we could also support a list for several configs with a 'config_name' field.

The alternative was to use YAML in the README.md.

I think it could also support a `dataset_type` field to specify which dataset builder class to use, and the other parameters would be the builder's parameters. Some parameters exist for all builders like `data_files` and `data_dir`, but some parameters are builder specific like `sep` for csv.

This format would be used in `push_to_hub` to be able to push multiple configs.

cc @huggingface/datasets 

EDIT: actually we're going for the YAML approach in README.md
Support streaming Beam datasets from HF GCS preprocessed data

This PR implements streaming Apache Beam datasets that are already preprocessed by us and stored in the HF Google Cloud Storage:
- natural_questions
- wiki40b
- wikipedia

This is done by streaming from the prepared Arrow files in HF Google Cloud Storage.

This will fix their corresponding dataset viewers. Related to:
- https://github.com/huggingface/datasets-server/pull/988#discussion_r1150767138

Related to:
- https://huggingface.co/datasets/natural_questions/discussions/4
- https://huggingface.co/datasets/wiki40b/discussions/2
- https://huggingface.co/datasets/wikipedia/discussions/9

CC: @severo
raise AttributeError(f"No {package_name} attribute {name}") AttributeError: No huggingface_hub attribute hf_api

### Describe the bug

rta.sh
Traceback (most recent call last):
  File "run.py", line 7, in <module>
    import datasets
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/data_files.py", line 120, in <module>
    dataset_info: huggingface_hub.hf_api.DatasetInfo,
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/huggingface_hub/__init__.py", line 290, in __getattr__
    raise AttributeError(f"No {package_name} attribute {name}")
AttributeError: No huggingface_hub attribute hf_api


### Reproduction

_No response_

### Logs

```shell
Traceback (most recent call last):
  File "run.py", line 7, in <module>
    import datasets
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/__init__.py", line 37, in <module>
    from .builder import ArrowBasedBuilder, BeamBasedBuilder, BuilderConfig, DatasetBuilder, GeneratorBasedBuilder
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/builder.py", line 44, in <module>
    from .data_files import DataFilesDict, _sanitize_patterns
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/datasets/data_files.py", line 120, in <module>
    dataset_info: huggingface_hub.hf_api.DatasetInfo,
  File "/home/appuser/miniconda3/envs/pt2/lib/python3.8/site-packages/huggingface_hub/__init__.py", line 290, in __getattr__
    raise AttributeError(f"No {package_name} attribute {name}")
AttributeError: No huggingface_hub attribute hf_api
```


### System info

```shell
- huggingface_hub version: 0.13.2
- Platform: Linux-5.4.0-144-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /home/appuser/.cache/huggingface/token
- Has saved token ?: False
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: 1.7.1
- Jinja2: N/A
- Graphviz: N/A
- Pydot: N/A
- Pillow: 9.3.0
- hf_transfer: N/A
- ENDPOINT: https://huggingface.co
- HUGGINGFACE_HUB_CACHE: /home/appuser/.cache/huggingface/hub
- HUGGINGFACE_ASSETS_CACHE: /home/appuser/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/appuser/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
```
Wikipedia download_and_prepare for GCS

### Describe the bug

I am unable to download the wikipedia dataset onto GCS.

When I run the script provided the memory firstly gets eaten up, then it crashes. 
I tried running this on a VM with 128GB RAM and all I got was a two empty files: _data_builder.lock_, _data.incomplete/beam-temp-wikipedia-train-1ab2039acf3611ed87a9893475de0093_

I have troubleshot this for two straight days now, but I am just unable to get the dataset into storage.

### Steps to reproduce the bug

Run this and insert a path:
```
import datasets

builder = datasets.load_dataset_builder(
    "wikipedia", language="en", date="20230320", beam_runner="DirectRunner")

builder.download_and_prepare({path}, file_format="parquet")
```
This is where the problem of it eating RAM occurs.



I have also tried several versions of this, based on the docs:
```
import gcsfs
import datasets

storage_options = {"project": "tdt4310", "token": "cloud"}
fs = gcsfs.GCSFileSystem(**storage_options)

output_dir = "gcs://wikipediadata/"
builder = datasets.load_dataset_builder(
    "wikipedia", date="20230320", language="en",  beam_runner="DirectRunner")
builder.download_and_prepare(
    output_dir, storage_options=storage_options, file_format="parquet")
```
The error message that is received here is:

> ValueError: Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: gcs://wikipediadata/wikipedia-train [while running 'train/Save to parquet/Write/WriteImpl/InitializeWrite']

I have ran `pip install apache-beam[gcp]`

### Expected behavior

The wikipedia data loaded into GCS

Everything worked when testing with a smaller demo dataset found somewhere in the docs

### Environment info

Newest published version of datasets. Python 3.9. Also tested with Python 3.7. 128GB RAM Google Cloud VM instance.
Document to compress data files before uploading

In our docs to [Share a dataset to the Hub](https://huggingface.co/docs/datasets/upload_dataset), we tell users to upload directly their data files, like CSV, JSON, JSON-Lines, text,... However, these extensions are not tracked by Git LFS by default, as they are not in the `.giattributes` file. Therefore, if they are too large, Git will fail to commit/upload them.

I think for those file extensions (.csv, .json, .jsonl, .txt), we should better recommend to **compress** their data files (using ZIP for example) before uploading them to the Hub.
- Compressed files are tracked by Git LFS in our default `.gitattributes` file

What do you think?
CC: @stevhliu 

See related issue:
- https://huggingface.co/datasets/tcor0005/langchain-docs-400-chunksize/discussions/1
Add information about patterns search order to the doc about structuring repo

Following [this](https://github.com/huggingface/datasets/issues/5650) issue I think we should add a note about the order of patterns that is used to find splits, see [my comment](https://github.com/huggingface/datasets/issues/5650#issuecomment-1488412527). Also we should reference this page in pages about packaged loaders. 

I have a déjà vu that it had already been discussed as some point but I don't remember....
Fix a description error for interleave_datasets.

There is a description mistake in the annotation of interleave_dataset with "all_exhausted" stopping_strategy.
``` python
d1 = Dataset.from_dict({"a": [0, 1, 2]})
d2 = Dataset.from_dict({"a": [10, 11, 12, 13]})
d3 = Dataset.from_dict({"a": [20, 21, 22, 23, 24]})
dataset = interleave_datasets([d1, d2, d3], stopping_strategy="all_exhausted")
```
According to the interleave way, the correct output of `dataset["a"]` is `[0, 10, 20, 1, 11, 21, 2, 12, 22, 0, 13, 23, 1, 10, 24]`, not `[0, 10, 20, 1, 11, 21, 2, 12, 22, 0, 13, 23, 1, 0, 24]`
Allow load_dataset to take a working dir for intermediate data

### Feature request

As a user, I can set a working dir for intermediate data creation. The processed files will be moved to the cache dir, like
```
load_dataset(…, working_dir=”/temp/dir”, cache_dir=”/cloud_dir”).
```

### Motivation

This will help the use case for using datasets with cloud storage as cache. It will help boost the performance.

### Your contribution

I can provide a PR to fix this if the proposal seems reasonable.
Add support to create a Dataset from spark dataframe

### Feature request

Add a new API `Dataset.from_spark` to create a Dataset from Spark DataFrame.

### Motivation

Spark is a distributed computing framework that can handle large datasets. By supporting loading Spark DataFrames directly into Hugging Face Datasets, we enable take the advantages of spark to processing the data in parallel. 

By providing a seamless integration between these two frameworks, we make it easier for data scientists and developers to work with both Spark and Hugging Face in the same workflow.

### Your contribution

We can discuss about the ideas and I can help preparing a PR for this feature.
Filter datasets by language code

Hi! I use the language search field on https://huggingface.co/datasets
However, some of the datasets tagged by ISO language code are not accessible by this search form.
For example, [myv_ru_2022](https://huggingface.co/datasets/slone/myv_ru_2022) is has `myv` language tag but it is not included in Languages search form.
I've also noticed the same problem with `mhr` (see https://huggingface.co/datasets/AigizK/mari-russian-parallel-corpora)
Stored XSS

### Describe the bug

I found a Stored XSS on a page that can be publicly accessible to all visitors. But I didn't find a suitable place to report. 
Please guide me on this. 

### Steps to reproduce the bug

Due to security restrictions, I don't want to publish it publicly.

### Expected behavior

User inputs must be sanitized before rendering.

### Environment info

https://huggingface.co/ Web UI
Pass down storage options

Remove implementation-specific kwargs from `file_utils.fsspec_get` and `file_utils.fsspec_head`, instead allowing them to be passed down via `storage_options`. This fixes an issue where s3fs did not recognize a timeout arg as well as fixes an issue mentioned in https://github.com/huggingface/datasets/issues/5281 by allowing users to pass down `storage_options` all the way from `datasets.load_dataset` to support implementation-specific credentials

Supports something like the following to provide credentials explicitly instead of relying on boto's methods of locating them

```
load_dataset(..., data_files=["s3://..."], storage_options={"profile": "..."})
```
How to use `load_dataset('glue', 'cola')`

### Describe the bug

I'm new to use HuggingFace datasets but I cannot use `load_dataset('glue', 'cola')`.

- I was stacked by the following problem:

```python
from datasets import load_dataset
cola_dataset = load_dataset('glue', 'cola')

---------------------------------------------------------------------------
InvalidVersion                            Traceback (most recent call last)
File <timed exec>:1

(Omit because of long error message)

File /usr/local/lib/python3.8/site-packages/packaging/version.py:197, in Version.__init__(self, version)
    195 match = self._regex.search(version)
    196 if not match:
--> 197     raise InvalidVersion(f"Invalid version: '{version}'")
    199 # Store the parsed out pieces of the version
    200 self._version = _Version(
    201     epoch=int(match.group("epoch")) if match.group("epoch") else 0,
    202     release=tuple(int(i) for i in match.group("release").split(".")),
   (...)
    208     local=_parse_local_version(match.group("local")),
    209 )

InvalidVersion: Invalid version: '0.10.1,<0.11'
```

- You can check this full error message in my repository: [MLOps-Basics/week_0_project_setup/experimental_notebooks/data_exploration.ipynb](https://github.com/makinzm/MLOps-Basics/blob/eabab4b837880607d9968d3fa687c70177b2affd/week_0_project_setup/experimental_notebooks/data_exploration.ipynb)


### Steps to reproduce the bug

- This is my repository to reproduce: [MLOps-Basics/week_0_project_setup](https://github.com/makinzm/MLOps-Basics/tree/eabab4b837880607d9968d3fa687c70177b2affd/week_0_project_setup)

1. cd `/DockerImage` and command `docker build . -t week0`
2. cd `/` and command `docker-compose up`
3. Run `experimental_notebooks/data_exploration.ipynb`

----

Just to be sure, I wrote down Dockerfile and requirements.txt

- Dockerfile
```Dockerfile
FROM python:3.8

WORKDIR /root/working

RUN apt-get update && \
    apt-get install -y python3-dev python3-pip python3-venv && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

COPY requirements.txt .

RUN pip3 install --no-cache-dir jupyter notebook && pip install --no-cache-dir -r requirements.txt

CMD ["bash"]
```

- requirements.txt
```txt
pytorch-lightning==1.2.10
datasets==1.6.2
transformers==4.5.1
scikit-learn==0.24.2
```


### Expected behavior

There is no bug to implement `load_dataset('glue', 'cola')`

### Environment info

I already wrote it.
Almost identical datasets, huge performance difference

### Describe the bug

I am struggling to understand (huge) performance difference between two datasets that are almost identical. 

### Steps to reproduce the bug

# Fast (normal) dataset speed:
```python
import cv2
from datasets import load_dataset
from torch.utils.data import DataLoader

dataset = load_dataset("beans", split="train")

for x in DataLoader(dataset.with_format("torch"), batch_size=16, shuffle=True, num_workers=8):
    pass
```

The above pass over the dataset takes about 1.5 seconds on my computer. 
However, if I re-create (almost) the same dataset, the sweep takes HUGE amount of time: 15 minutes. Steps to reproduce:

```python
def transform(example):
    example["image2"] = cv2.imread(example["image_file_path"])
    return example

dataset2 = dataset.map(transform, remove_columns=["image"])

for x in DataLoader(dataset2.with_format("torch"), batch_size=16, shuffle=True, num_workers=8):
    pass

```

### Expected behavior

Same timings

### Environment info

python==3.10.9
datasets==2.10.1
Fix unnecessary dict comprehension

After ruff-0.0.258 release, the C416 rule was updated with unnecessary dict comprehensions. See:
- https://github.com/charliermarsh/ruff/releases/tag/v0.0.258
- https://github.com/charliermarsh/ruff/pull/3605

This PR fixes one unnecessary dict comprehension in our code: no need to unpack and re-pack the tuple values.

Fix #5661
integration with imbalanced-learn

### Feature request

Wouldn't it be great if the various class balancing operations from imbalanced-learn were available as part of datasets?

### Motivation

I'm trying to use imbalanced-learn to balance a dataset, but it's not clear how to get the two to interoperate - what would be great would be some examples.  I've looked online, asked gpt-4, but so far not making much progress.

### Your contribution

If I can get this working myself I can submit a PR with example code to go in the docs
Fix `fsspec.open` when using an HTTP proxy

Most HTTP(S) downloads from this library support proxy automatically by reading the `HTTP_PROXY` environment variable (et al.) because `requests` is widely used. However, in some parts of the code, `fsspec` is used, which in turn uses `aiohttp` for HTTP(S) requests (as opposed to `requests`), which in turn doesn't support reading proxy env variables by default. This PR enables reading them automatically.

Read [aiohttp docs on using proxies](https://docs.aiohttp.org/en/stable/client_advanced.html?highlight=trust_env#proxy-support).

For context, [the Python library requests](https://requests.readthedocs.io/en/latest/user/advanced/?highlight=http_proxy#proxies) and [the official Python library via `urllib.urlopen` support this automatically by default](https://docs.python.org/3/library/urllib.request.html#urllib.request.urlopen). Many (most common ones?) programs also do the same, including cURL, APT, Wget, and many others.
Doc: save_to_disk, `num_proc` will affect `num_shards`, but it's not documented

### Describe the bug

[`num_proc`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.save_to_disk.num_proc) will affect `num_shards`, but it's not documented


### Steps to reproduce the bug

Nothing to reproduce

### Expected behavior

[document of `num_shards`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict.save_to_disk.num_shards) explicitly says that it depends on `max_shard_size`, it should also mention `num_proc`.


### Environment info

datasets main document
Copy features

Some users (even internally at HF) are doing
```python
dset_features = dset.features
dset_features.pop(col_to_remove)
dset = dset.map(..., features=dset_features)
```

Right now this causes issues because it modifies the features dict in place before the map.

In this PR I modified `dset.features` to return a copy of the features, so that users can modify it if they want.
expanduser in save_to_disk

### Describe the bug

save_to_disk() does not expand `~`

1. `dataset = load_datasets("any dataset")`
2. `dataset.save_to_disk("~/data")`
3. a folder named "~" created in current folder
4. FileNotFoundError is raised, because the expanded path does not exist (`/home/<user>/data`)

related issue https://github.com/huggingface/transformers/issues/10628

### Steps to reproduce the bug

As described above.

### Expected behavior

expanduser correctly

### Environment info

- datasets 2.10.1
- python 3.10
The index column created with .to_sql() is dependent on the batch_size when writing

### Describe the bug

It seems like the "index" column is designed to be unique? The values are only unique per batch. The SQL index is not a unique index.

This can be a problem, for instance, when building a faiss index on a dataset and then trying to match up ids with a sql export.

### Steps to reproduce the bug

```
from datasets import Dataset
import sqlite3
db = sqlite3.connect(":memory:")
nice_numbers = Dataset.from_dict({"nice_number": range(101,106)})
nice_numbers.to_sql("nice1", db, batch_size=1)
nice_numbers.to_sql("nice2", db, batch_size=2)
print(db.execute("select * from nice1").fetchall()) # [(0, 101), (0, 102), (0, 103), (0, 104), (0, 105)]
print(db.execute("select * from nice2").fetchall()) # [(0, 101), (1, 102), (0, 103), (1, 104), (0, 105)]
```

### Expected behavior

I expected the "index" column to be unique

### Environment info

```
 % datasets-cli env

Copy-and-paste the text below in your GitHub issue.

- `datasets` version: 2.10.1
- Platform: macOS-13.2.1-arm64-arm-64bit
- Python version: 3.9.6
- PyArrow version: 7.0.0
- Pandas version: 1.5.2

zsh: segmentation fault  datasets-cli env
```
flatten_indices doesn't work with pandas format

### Describe the bug

Hi,
I noticed that `flatten_indices` throws an error when the batch format is `pandas`. This is probably due to the fact that flatten_indices uses map internally which doesn't accept dataframes as the transformation function output

### Steps to reproduce the bug

tabular_data = pd.DataFrame(np.random.randn(10,10))
tabular_data = datasets.arrow_dataset.Dataset.from_pandas(tabular_data)
tabular_data.with_format("pandas").select([0,1,2,3]).flatten_indices()

### Expected behavior

No error thrown

### Environment info

- `datasets` version: 2.10.1
- Python version: 3.9.5
- PyArrow version: 11.0.0
- Pandas version: 1.4.1
Bump hfh to 0.11.0

to fix errors like

```
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://hub-ci.huggingface.co/api/datasets/__DUMMY_TRANSFORMERS_USER__/...
```

(e.g. from this [failing CI](https://github.com/huggingface/datasets/actions/runs/4428956210/jobs/7769160997))

0.11.0 is the current minimum version in `transformers`

around 5% of users are currently using versions `<0.11.0`
Less zip false positives

`zipfile.is_zipfile` return false positives for some Parquet files. It causes errors when loading certain parquet datasets, where some files are considered ZIP files by `zipfile.is_zipfile`

This is a known issue: https://github.com/python/cpython/issues/72680

At first I wanted to rely only on magic numbers, but then I found that someone contributed a [fix to is_zipfile](https://github.com/python/cpython/pull/5053) - do you think we should use it @albertvillanova or not ?

IMO it's ok to rely on magic numbers only for now, since in streaming mode we've had no issue checking only the magic number so far.

Close https://github.com/huggingface/datasets/issues/5639
xPath to implement all operations for Path

### Feature request

Current xPath implementation is a great extension of Path in order to work with remote objects. However some methods such as `mkdir` are not implemented correctly. It should instead rely on `fsspec` methods, instead of defaulting do `Path` methods which only work locally.

### Motivation

I'm using xPath to interact with remote objects.

### Your contribution

I could try to make a PR. I'm a bit unfamiliar with chaining right now.
IterableDataset with_format does not support 'device' keyword for jax

### Describe the bug

As seen here: https://huggingface.co/docs/datasets/use_with_jax dataset.with_format() supports the keyword 'device', to put data on a specific device when loaded as jax. However, when called on an IterableDataset, I got the error `TypeError: with_format() got an unexpected keyword argument 'device'`

Looking over the code, it seems IterableDataset support only pytorch and no support for jax device keyword? 
https://github.com/huggingface/datasets/blob/fc5c84f36684343bff3e424cb0fd1ac5ecdd66da/src/datasets/iterable_dataset.py#L1029



### Steps to reproduce the bug

1. Load an IterableDataset (tested in streaming mode)
2. Call with_format('jax',device=device)

### Expected behavior

I expect to call `with_format('jax', device=device)`  as per [documentation](https://huggingface.co/docs/datasets/use_with_jax) without error

### Environment info

Tested with installing newest (dev) and also pip release (2.10.1).
- `datasets` version: 2.10.2.dev0
- Platform: Linux-5.15.89+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- Huggingface_hub version: 0.12.1
- PyArrow version: 11.0.0
- Pandas version: 1.3.5
Pass custom metadata filename to Image/Audio folders

This is a quick fix. 
Now it requires to pass data via `data_files` parameters and include a required metadata file there and pass its filename as `metadata_filename` parameter.
For example, with the structure like:
```
data
   images_dir/
      im1.jpg
      im2.jpg
      ...
   metadata_dir/
      meta_file1.jsonl
      meta_file2.jsonl
      ...
```
to load data with `metadata_file1.jsonl` do:
```python
ds = load_dataset("imagefolder", data_files=["data/images_dir/**", "data/metadata_dir/meta_file1.jsonl"], metadata_filename="meta_file1.jsonl")
```

Note that if you have multiple splits, metadata file should be specified in each of them in `data_files`, smth like:
```python
data_files={
"train": ["data/train/**", "data/metadata_dir/meta_file1.jsonl"], 
"test": ["data/train/**", "data/metadata_dir/meta_file1.jsonl"]
}
```