rachel521/github-issues-simple huggingface dataset:
Distributed data parallel training for streaming datasets
Opened by cyk1337 (open)

### Feature request

Any documentations for the the `load_dataset(streaming=True)` for (multi-node multi-GPU) DDP training? 

### Motivation

Given a bunch of data files, it is expected to split them onto different GPUs. Is there a guide or documentation?

### Your contribution

Does it requires manually split on data files for each worker in `DatasetBuilder._split_generator()`? What is`IterableDatasetShard` expected to do?

Comments: Hi ! According to https://huggingface.co/docs/datasets/use_with_pytorch#stream-data you can use the pytorch DataLoader with `num_workers>0` to distribute the shards across your workers (it uses `torch.utils.data.get_worker_info()` to get the worker ID and select the right subsets of shards to use)


 EDIT: here is a code example 

```python
# ds = ds.with_format("torch")
# dataloader = DataLoader(ds, num_workers=num_workers)
```

EDIT: `with_format("torch")` is not required, now you can just do
```python
dataloader = DataLoader(ds, num_workers=num_workers)
```

@cyk1337 does streaming datasets with multi-gpu works for you? I am testing on one node with multiple gpus, but this is freezing, https://github.com/huggingface/datasets/issues/5123 
In case you could make this work, could you share with me your data-loading codes?
thank you

+1

This has been implemented in `datasets` 2.8:
```python
from datasets.distributed import split_dataset_by_node

ds = split_dataset_by_node(ds, rank=rank, world_size=world_size)
```

docs: https://huggingface.co/docs/datasets/use_with_pytorch#distributed

i'm having hanging issues with this when using DDP and allocating the datasets with `split_dataset_by_node` 🤔

--- 
### edit
I don't want to pollute this thread, but for the sake of following up, I observed hanging close to the final iteration of the dataloader. I think this was happening on the final shard. First, I removed the final shard and things worked. Then (including all shards), I reordered the list of shards: `load_dataset('json', data_files=reordered, streaming=True)` and no hang. 

I won't open an issue yet bc I am not quite sure about this observation.

@wconnell would you mind opening a different bug issue and giving more details?
https://github.com/huggingface/datasets/issues/new?assignees=&labels=&template=bug-report.yml

Thanks.
Add option for named splits when using ds.train_test_split
Opened by nateraw (open)

### Feature Request 🚀 

Can we add a way to name your splits when using the `.train_test_split` function?

In almost every use case I've come across, I have a `train` and a `test` split in my `DatasetDict`, and I want to create a `validation` split. Therefore, its kinda useless to get a `test` split back from `train_test_split`, as it'll just overwrite my real `test` split that I intended to keep.

### Workaround

this is my hack for dealin with this, for now :slightly_smiling_face:

```python
from datasets import load_dataset
​
​
ds = load_dataset('imdb')
ds['train'], ds['validation'] = ds['train'].train_test_split(.1).values()
```


Comments: Yes definitely we should give more flexibility to control the name of the splits outputted by `train_test_split`.

Related is the very interesting feedback from @bramvanroy on how we should improve this method: https://discuss.huggingface.co/t/how-to-split-main-dataset-into-train-dev-test-as-datasetdict/1090/5

And in particular that it should advantageously be able to split in 3 splits as well instead of just 2 like we copied from sklearn.
Empty evidence document ("provenance") in KILT ELI5 dataset
Opened by donggyukimc (closed)

In the  original KILT benchmark(https://github.com/facebookresearch/KILT), 

all samples has its evidence document (i.e. wikipedia page id) for prediction.

For example, a sample in ELI5 dataset has the format including provenance (=evidence document) like this

`{"id": "1kiwfx", "input": "In Trading Places (1983, Akroyd/Murphy) how does the scheme at the end of the movie work? Why would buying a lot of OJ at a high price ruin the Duke Brothers?", "output": [{"answer": "I feel so old. People have been askinbg what happened at the end of this movie for what must be the last 15 years of my life. It never stops. Every year/month/fortnight, I see someone asking what happened, and someone explaining. Andf it will keep on happening, until I am 90yrs old, in a home, with nothing but the Internet and my bladder to keep me going. And there it will be: \"what happens at the end of Trading Places?\""}, {"provenance": [{"wikipedia_id": "242855", "title": "Futures contract", "section": "Section::::Abstract.", "start_paragraph_id": 1, "start_character": 14, "end_paragraph_id": 1, "end_character": 612, "bleu_score": 0.9232808519770748}]}], "meta": {"partial_evidence": [{"wikipedia_id": "520990", "title": "Trading Places", "section": "Section::::Plot.\n", "start_paragraph_id": 7, "end_paragraph_id": 7, "meta": {"evidence_span": ["On television, they learn that Clarence Beeks is transporting a secret USDA report on orange crop forecasts.", "On television, they learn that Clarence Beeks is transporting a secret USDA report on orange crop forecasts. Winthorpe and Valentine recall large payments made to Beeks by the Dukes and realize that the Dukes plan to obtain the report to corner the market on frozen orange juice.", "Winthorpe and Valentine recall large payments made to Beeks by the Dukes and realize that the Dukes plan to obtain the report to corner the market on frozen orange juice."]}}]}}`

However, KILT ELI5 dataset from huggingface datasets library only contain empty list of provenance.

`{'id': '1oy5tc', 'input': 'in football whats the point of wasting the first two plays with a rush - up the middle - not regular rush plays i get those', 'meta': {'left_context': '', 'mention': '', 'obj_surface': [], 'partial_evidence': [], 'right_context': '', 'sub_surface': [], 'subj_aliases': [], 'template_questions': []}, 'output': [{'answer': 'In most cases the O-Line is supposed to make a hole for the running back to go through. If you run too many plays to the outside/throws the defense will catch on.\n\nAlso, 2 5 yard plays gets you a new set of downs.', 'meta': {'score': 2}, 'provenance': []}, {'answer': "I you don't like those type of plays, watch CFL.  We only get 3 downs so you can't afford to waste one.  Lots more passing.", 'meta': {'score': 2}, 'provenance': []}]}
`

should i perform other procedure to obtain evidence documents?

Comments: Why did you close this issue? How did you end up finding the evidence documents? I'm running into a similar issue with other KILT tasks.
Index error when data is large
Opened by ChenchaoZhao (open)

### Describe the bug

At `save_to_disk` step, the `max_shard_size` by default is `500MB`. However, one row of the dataset might be larger than `500MB` then the saving will throw an index error. Without looking at the source code, the bug is due to wrong calculation of number of shards which i think is 
`total_size / min(max_shard_size, row_size)` which should be `total_size / max(max_shard_size, row_size)`

The fix is setting a larger `max_shard_size`

### Steps to reproduce the bug

1. create a dataset with large dense tensors per row
2. set a small `max_shard_size` say 1MB
3. `save_to_disk`

### Expected behavior

```
raise IndexError(f"Index {index} out of range for dataset of size {size}.")

IndexError: Index 10 out of range for dataset of size 10.
```

### Environment info

- `datasets` version: 2.16.0
- Platform: Linux-5.10.201-168.748.amzn2int.x86_64-x86_64-with-glibc2.26
- Python version: 3.10.13
- `huggingface_hub` version: 0.20.2
- PyArrow version: 14.0.2
- Pandas version: 2.1.4
- `fsspec` version: 2023.12.2

Comments: I'm facing this problem while doing my translation of [mteb/stackexchange-clustering](https://huggingface.co/datasets/mteb/stackexchange-clustering). each row has lots of samples (up to 100k samples), because in this dataset, each row represent multiple clusters.
my hack is to setting `max_shard_size` to 20Gb or even larger
```py
final_dataset.push_to_hub(
        output_dataset, 
        private=True,
        max_shard_size="20GB"  # This will ensure appropriate sharding based on data size
    )
```
It will work, but depends on your data size.
Clicking on a metric in the search page points to datasets page giving "Missing dataset" warning
Opened by ogabrielluiz (closed)

Hi! Sorry if this isn't the right place to talk about the website, I just didn't exactly where to write this.

Searching a metric in https://huggingface.co/metrics gives the right results but clicking on a metric (E.g ROUGE) points to https://huggingface.co/datasets/rouge. Clicking on a metric without searching points to the right page.

Thanks for all the great work!

Comments: Thanks for the report, can reproduce. Will fix

Fixed now @ogabrielluiz
Add spanish billion words
Opened by mariagrandury (closed)

Add an unannotated corpus of the Spanish language of nearly 1.5 billion words, compiled from different resources from the web.

The dataset needs 10 GB (download: 1.89 GiB, generated: 8.34 GiB, post-processed: Unknown size, total: 10.22 GiB), the test using dummy data pass but my laptop isn't able to run it on the real data (I left it running for over 8 hours and it didn't finish).

Comments: The tests failed because of ```RemoteDatasetTest``` so I tried ```git rebase``` and messed everything up. I've made a new clean PR (#1347).
Getting Value Error while loading a dataset..
Opened by valmetisrinivas (closed)

### Describe the bug

I am trying to load a dataset using Hugging Face Datasets load_dataset method. I am getting the value error as show below. Can someone help with this? I am using Windows laptop and Google Colab notebook.

```
WARNING:datasets.builder:Using custom data configuration default-a1d9e8eaedd958cd
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-12-5b4fdcb8e6d5>](https://localhost:8080/#) in <module>
      6 )
      7 
----> 8 next(iter(law_dataset_streamed))

17 frames
[/usr/local/lib/python3.8/dist-packages/fsspec/core.py](https://localhost:8080/#) in get_compression(urlpath, compression)
    485         compression = infer_compression(urlpath)
    486     if compression is not None and compression not in compr:
--> 487         raise ValueError("Compression type %s not supported" % compression)
    488     return compression
    489 

ValueError: Compression type zstd not supported
```

### Steps to reproduce the bug

```
!pip install zstandard
from datasets import load_dataset

lds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)
```

### Expected behavior

I expect an iterable object as the output 'lds' to be created.

### Environment info

Windows laptop with Google Colab notebook

Comments: Hi! I can't reproduce this error locally (Mac) or in Colab. What version of `datasets` are you using?

Hi [mariosasko](https://github.com/mariosasko), the datasets version is '2.8.0'.

@valmetisrinivas you get that error because you imported `datasets` (and thus `fsspec`) before installing `zstandard`.

Please, restart your Colab runtime and execute the install commands before importing `datasets`:
```python
!pip install datasets
!pip install zstandard

from datasets import load_dataset

ds = load_dataset(
    "json",
    data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
    split="train",
    streaming=True,
)
next(iter(ds))
```

> @valmetisrinivas you get that error because you imported `datasets` (and thus `fsspec`) before installing `zstandard`.
> 
> Please, restart your Colab runtime and execute the install commands before importing `datasets`:
> 
> ```python
> !pip install datasets
> !pip install zstandard
> 
> from datasets import load_dataset
> 
> ds = load_dataset(
>     "json",
>     data_files="https://the-eye.eu/public/AI/pile_preliminary_components/FreeLaw_Opinions.jsonl.zst",
>     split="train",
>     streaming=True,
> )
> next(iter(ds))
> ```

I guess that was the problem, importing datasets before the installation of zstandard. Thank you for the feedback.
fix regression
Opened by lhoestq (closed)

reported in https://github.com/huggingface/datasets/pull/7557 (I just reorganized the condition)

wanted to apply this change to the original PR but github didn't let me apply it directly - merging this one instead

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7558). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix dataset builder default version
Opened by albertvillanova (closed)

Currently, when using a custom config (subclass of `BuilderConfig`), default version set at the builder level is ignored: we must set default version in the custom config class.

However, when loading a dataset with `config_kwargs` (for a configuration not present in `BUILDER_CONFIGS`), the default version set in the custom config is ignored and "0.0.0" is used instead:
```python
ds = load_dataset("wikipedia", language="co", date="20220501", beam_runner="DirectRunner")
```
generates the following config:
```python
WikipediaConfig(name='20220501.co', version=0.0.0, data_dir=None, data_files=None, description='Wikipedia dataset for co, parsed from 20220501 dump.')
```
with version "0.0.0" instead of "2.0.0".

See as a counter-example, when the config is present in `BUILDER_CONFIGS`:
```python
ds = load_dataset("wikipedia", "20220301.fr", beam_runner="DirectRunner")
```
generates the following config:
```python
WikipediaConfig(name='20220301.fr', version=2.0.0, data_dir=None, data_files=None, description='Wikipedia dataset for fr, parsed from 20220301 dump.')
```
with correct version "2.0.0", as set in the custom config class.


The reason for this is that `DatasetBuilder` has a default VERSION ("0.0.0") that overwrites the default version set at the custom config class.

This PR:
- Removes the default VERSION at `DatasetBuilder` (set to None, so that the class attribute exists but it does not override the custom config default version).
- Note that the `BuilderConfig` class already sets a default version = "0.0.0"; no need to pass this from the builder.

Comments: _The documentation is not available anymore as the PR was closed or merged._

This PR requires one of these other PRs being merged first:
- #4359 
- huggingface/doc-builder#211
Reuters21578 Couldn't reach 
Opened by TingNLP (closed)

``## Adding a Dataset
- **Name:** *Reuters21578*
- **Description:** *ConnectionError: Couldn't reach https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.tar.gz*
- **Data:** *https://huggingface.co/datasets/reuters21578*

`from datasets import load_dataset`
`dataset = load_dataset("reuters21578", 'ModLewis')`

ConnectionError: Couldn't reach https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.tar.gz

And I try to request the link as follow:
`import requests`
`requests.head('https://kdd.ics.uci.edu/databases/reuters21578/reuters21578.tar.gz')`

SSLError: HTTPSConnectionPool(host='kdd.ics.uci.edu', port=443): Max retries exceeded with url: /databases/reuters21578/reuters21578.tar.gz (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

This problem likes #575
What should I do ?


Comments: Hi ! The URL works fine on my side today, could you try again ?

thank you @lhoestq 
it works
can not flatten natural_questions dataset
Opened by Hannibal046 (closed)

## Describe the bug
after downloading the natural_questions dataset, can not flatten the dataset considering there are `long answer` and `short answer` in `annotations`.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset('natural_questions',cache_dir = 'data/dataset_cache_dir')
dataset['train'].flatten()
```

## Expected results
a dataset with `long_answer` as features

## Actual results
Traceback (most recent call last):
  File "temp.py", line 5, in <module>
    dataset['train'].flatten()
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/fingerprint.py", line 413, in wrapper
    out = func(self, *args, **kwargs)
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1296, in flatten
    dataset._data = update_metadata_with_features(dataset._data, dataset.features)
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 536, in update_metadata_with_features
    features = Features({col_name: features[col_name] for col_name in table.column_names})
  File "/Users/hannibal046/anaconda3/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 536, in <dictcomp>
    features = Features({col_name: features[col_name] for col_name in table.column_names})
KeyError: 'annotations.long_answer'
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.13
- Platform: MBP
- Python version: 3.8
- PyArrow version: 6.0.1


Comments: same issue. downgrade it to a lower version.

Thanks for reporting, I'll take a look tomorrow :)
Don't do unnecessary list type casting to avoid replacing None values by empty lists
Opened by lhoestq (closed)

In certain cases, `None` values are replaced by empty lists when casting feature types.

It happens every time you cast an array of nested lists like [None, [0, 1, 2, 3]] to a different type (to change the integer precision for example). In this case you'd get [[], [0, 1, 2, 3]] for example. This issue comes from PyArrow, see the discussion in https://github.com/huggingface/datasets/issues/3676

This issue also happens when no type casting is needed, because casting is supposed to be a no-op in this case. But as https://github.com/huggingface/datasets/issues/3676 shown, it's not the case and `None` are replaced by empty lists even if we cast to the exact same type.

In this PR I just workaround this bug in the case where no type casting is needed. In particular, I only call `pa.ListArray.from_arrays` only when necessary.

I also added a warning when some `None` are effectively replaced by empty lists. I wanted to raise an error in this case, but maybe we should wait a major update to do so

This PR fixes this particular case, that is occurring in `run_qa.py` in `transformers`:
```python
from datasets import Dataset

ds = Dataset.from_dict({"a": range(4)})
ds = ds.map(lambda x: {"b": [[None, [0]]]}, batched=True, batch_size=1, remove_columns=["a"])
print(ds.to_pandas())
# before:
#              b
# 0  [None, [0]]
# 1    [[], [0]]
# 2    [[], [0]]
# 3    [[], [0]]
#
# now:
#              b
# 0  [None, [0]]
# 1  [None, [0]]
# 2  [None, [0]]
# 3  [None, [0]]
```

cc @sgugger 

Comments: _The documentation is not available anymore as the PR was closed or merged._

Quick question about the message in the warning. You say "will be fixed in a future major version" but don't you mean "will raise an error in a future major version"?

Right ! Good catch, thanks, I updated the message to say "will raise an error in a future major version"
Hendrycks Checksum
Opened by DanielHesslow (closed)

Hi, 
The checksum for [hendrycks_test](https://huggingface.co/datasets/hendrycks_test) does not compare correctly, I guess it has been updated on the remote.

```
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://people.eecs.berkeley.edu/~hendrycks/data.tar']
```


Comments: Thanks for reporting, @DanielHesslow. We are fixing it. 

@albertvillanova thanks for taking care of this so quickly!

The dataset metadata is fixed. You can download it normally.
Fix race condition in doc build
Opened by lhoestq (closed)

Following https://github.com/huggingface/datasets/runs/5499386744 it seems that race conditions that create issues when updating the doc. I took the same approach as in `transformers` to fix race conditions

Comments: The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_3891). All of your documentation changes will be reflected on that endpoint.
Some datasets miss dataset_infos.json or dummy_data.zip
Opened by madlag (closed)

While working on dataset REAME generation script at https://github.com/madlag/datasets_readme_generator , I noticed that some datasets miss a dataset_infos.json : 

```
c4
lm1b
reclor
wikihow
```

And some does not have a dummy_data.zip : 

```
kor_nli
math_dataset
mlqa
ms_marco
newsgroup
qa4mre
qangaroo
reddit_tifu
super_glue
trivia_qa
web_of_science
wmt14
wmt15
wmt16
wmt17
wmt18
wmt19
xtreme
```

But it seems that some of those last do have a "dummy" directory .



Comments: Thanks for reporting.
We should indeed add all the missing dummy_data.zip and also the dataset_infos.json at least for lm1b, reclor and wikihow.

For c4 I haven't tested the script and I think we'll require some optimizations regarding beam datasets before processing it.


Closing since the dummy data generation is deprecated now (and the issue with missing metadata seems to be addressed).
passing parameters to the method passed to Dataset.from_generator()
Opened by Basir-mahmood (closed)

Big thanks for providing dataset creation via a generator.
I want to ask whether there is any way that parameters can be passed to the method Dataset.from_generator() method, like as follows. 

```
from datasets import Dataset

def gen(param1):
    for idx in len(custom_dataset):
        yield custom_dataset[idx] + param1

ds = Dataset.from_generator(gen(param1))
```


Comments: Hi! Yes, you can either use the `gen_kwargs` param in `Dataset.from_generator` (`ds = Dataset.from_generator(gen, gen_kwargs={"param1": val})`) or wrap the generator function with `functools.partial`
(`ds = Dataset.from_generator(functools.partial(gen, param1="val"))`) to pass custom parameters to it.
Custom plain text dataset, plain json dataset and plain csv dataset are remove from datasets template folder
Opened by zixiliuUSC (closed)

## Adding a Dataset
- **Name:** text, json, csv
- **Description:** I am developing a customized dataset loading script. The problem is  mainly about my custom dataset is seperate into many files and I only find a dataset loading template in [https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py](https://github.com/huggingface/datasets/blob/1.2.1/datasets/json/json.py) that can handle my circumstance. I'm afraid these templates are too old to use. Could you re-add these three templates to current master branch?



Comments: Hi @zixiliuUSC, 

As explained in the documentation (https://huggingface.co/docs/datasets/loading.html#json), we support loading any dataset in JSON (as well as CSV, text, Parquet) format:
```python
ds = load_dataset('json', data_files='my_file.json')
```