rachel521/github-issues-simple huggingface dataset:
Error of writing with different schema, due to nonpreservation of nullability
Opened by richarddwang (closed)

## 1. Case
```
dataset.map(
  batched=True,
  disable_nullable=True,
)
```
will get the following error at here https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/arrow_writer.py#L516

`pyarrow.lib.ArrowInvalid: Tried to write record batch with different schema`

## 2. Debugging
### 2.1 tracing
During `_map_single`, the following are called
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/arrow_dataset.py#L2523
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/arrow_writer.py#L511
### 2.2. Observation
The problem is, even after `table_cast`, `pa_table.schema != self._schema`
`pa_table.schema` (before/after `table_cast`)
```
input_ids: list<item: int32>
  child 0, item: int32
```
`self._schema`
```
input_ids: list<item: int32> not null
  child 0, item: int32
```
### 2.3. Reason
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/table.py#L1121
Here we lose nullability stored in `schema` because it seems that `Features` is always nullable and don't store nullability.
https://github.com/huggingface/datasets/blob/c9967f55626931f8059dc416526c791444cdfdf7/src/datasets/table.py#L1103
So, casting to schema from such `Features` loses nullability, and eventually causes error of writing with different schema

## 3. Solution
1. Let `Features` stores nullability.
2. Directly cast table with original schema but not schema from converted `Features`. (this PR)
3. Don't `cast_table` when `write_table`

Comments: Hi ! Thanks for reporting, indeed `disable_nullable` doesn't seem to be supported in this case. Maybe at one point we can have `disable_nullable` as a parameter of certain feature types
audiofolder only creates dataset of 13 rows (files) when the data folder it's reading from has 20,000 mp3 files. 
Opened by jcho19 (closed)

### Describe the bug

x = load_dataset("audiofolder", data_dir="x")
When running this, x is a dataset of 13 rows (files) when it should be 20,000 rows (files) as the data_dir "x" has 20,000 mp3 files. Does anyone know what could possibly cause this (naming convention of mp3 files, etc.)


### Steps to reproduce the bug

x = load_dataset("audiofolder", data_dir="x")


### Expected behavior

x = load_dataset("audiofolder", data_dir="x") should create a dataset of 20,000 rows (files).

### Environment info

- `datasets` version: 2.9.0
- Platform: Linux-3.10.0-1160.80.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.9.16
- PyArrow version: 11.0.0
- Pandas version: 1.5.3

Comments: Hi!

> naming convention of mp3 files

Yes, this could be the problem. MP3 files should end with  `.mp3`/`.MP3` to be recognized as audio files.

If the file names are not the culprit, can you paste the audio folder's directory structure to help us reproduce the error (e.g., by running the `tree "x"` command)?

Hi! I'm sorry, I don't want to reveal my entire dataset, but here's a snippet (all of the mp3 files below are some of the ones not being recognized by audiofolder. Also, for another dataset, audiofolder loaded zero mp3 files because "train" was in the name of one of the mp3 files. 
my_dataset
├── data
│   ├── VHA_Innovation_Stories_-_Day_2-123.mp3
│   ├── VHA_Innovation_Stories_-_Day_2-124.mp3
│   ├── ASSOCIATION_OF_GENERAL_PRACTITIONERS_OF_JAMAICA_NEPHROLOGY_CONFERENCE_-_JULY_3,_2022-93.mp3
│   ├── ASSOCIATION_OF_GENERAL_PRACTITIONERS_OF_JAMAICA_NEPHROLOGY_CONFERENCE_-_JULY_3,_2022-94.mp3
│   ├── ASSOCIATION_OF_GENERAL_PRACTITIONERS_OF_JAMAICA_NEPHROLOGY_CONFERENCE_-_JULY_3,_2022-95.mp3
│   ├── Your_Impact\357\274\232_Neurosurgery_equipment-5.mp3
│   └── Your_Impact\357\274\232_Neurosurgery_equipment-6.mp3
└── metadata.csv

Here's a few of the 13 files recognized by the dataset:
British_Heart_Foundation_-_Your_guide_to_a_Coronary_Angiogram,_a_test_for_heart_disease-1.mp3
British_Heart_Foundation_-_Your_guide_to_a_Coronary_Angiogram,_a_test_for_heart_disease-2.mp3
British_Heart_Foundation_-_Your_guide_to_a_Coronary_Angiogram,_a_test_for_heart_disease-3.mp3
IVP_⧸_IVU_test_Procedure_for_Kidneys_intravenous_pyelogram_-_medical_radiology_X-ray_ivp-1.mp3
IVP_⧸_IVU_test_Procedure_for_Kidneys_intravenous_pyelogram_-_medical_radiology_X-ray_ivp-2.mp3
Getting next item from IterableDataset took forever.
Opened by HongtaoYang (closed)

### Describe the bug

I have a large dataset, about 500GB. The format of the dataset is parquet. 

I then load the dataset and try to get the first item
```python
def get_one_item():
    dataset = load_dataset("path/to/datafiles", split="train", cache_dir=".", streaming=True)
    dataset = dataset.filter(lambda example: example['text'].startswith('Ar'))
    print(next(iter(dataset)))
```

However, this function never finish. I waited ~10mins, the function was still running so I killed the process. I'm now using `line_profiler` to profile how long it would take to return one item. I'll be patient and wait for as long as it needs. 

I suspect the filter operation is the reason why it took so long. Can I get some possible reasons behind this?

### Steps to reproduce the bug

Unfortunately without my data files, there is no way to reproduce this bug.

### Expected behavior

With `IteralbeDataset`, I expect the first item to be returned instantly.

### Environment info

- datasets version: 2.11.0 
- python: 3.7.12

Comments: Hi! It can take some time to iterate over Parquet files as big as yours, convert the samples to Python, and find the first one that matches a filter predicate before yielding it...

Thanks @mariosasko, I figured it was the filter operation. I'm closing this issue because it is not a bug, it is the expected beheaviour.
Changing pixel values of images in the Winoground dataset
Opened by ZitengWangNYU (open)

Hi, as I followed the instructions, with lasted "datasets" version:
"
from datasets import load_dataset
examples = load_dataset('facebook/winoground', use_auth_token=<YOUR USER ACCESS TOKEN>)
"
I got slightly different datasets in colab and in my hpc environment. Specifically, the pixel values of images are slightly different. 
I thought it was due to the package version difference, but today's morning I found out that my winoground dataset in colab became the same with the one in my hpc environment. The dataset in colab can produce the correct result but now it is gone as well. 

Can you help me with this? What causes the datasets to have the wrong pixel values?
`DatasetBuilder._split_generators` incomplete type annotation
Opened by JonasLoos (closed)

### Describe the bug

The [`DatasetBuilder._split_generators`](https://github.com/huggingface/datasets/blob/0f27d7b77c73412cfc50b24354bfd7a3e838202f/src/datasets/builder.py#L1449) function has currently the following signature:

```python
class DatasetBuilder:
    def _split_generators(self, dl_manager: DownloadManager):
        ...
```

However, the `dl_manager` argument can also be of type [`StreamingDownloadManager`](https://github.com/huggingface/datasets/blob/0f27d7b77c73412cfc50b24354bfd7a3e838202f/src/datasets/download/streaming_download_manager.py#L962), which has different functionality. For example, the `download` function doesn't download, but rather just returns the given url(s). 

I suggest changing the function signature to:

```python
class DatasetBuilder:
    def _split_generators(self, dl_manager: Union[DownloadManager, StreamingDownloadManager]):
        ...
```

and also adjust the docstring accordingly.

I would like to create a Pull Request to fix this, and have the following questions:
* Are there also other options than `DownloadManager`, and `StreamingDownloadManager`?
* Should this also be changed in other functions?

### Steps to reproduce the bug

Minimal example to print the different class names:

```python
import tempfile
from datasets import load_dataset

example = b'''
from datasets import GeneratorBasedBuilder, DatasetInfo, Features, Value, SplitGenerator

class Test(GeneratorBasedBuilder):
    def _info(self):
        return DatasetInfo(features=Features({"x": Value("int64")}))
    def _split_generators(self, dl_manager):
        print(type(dl_manager))
        return [SplitGenerator('test')]
    def _generate_examples(self):
        yield 0, {'x': 42}
'''

with tempfile.NamedTemporaryFile(suffix='.py') as f:
    f.write(example)
    f.flush()
    load_dataset(f.name, streaming=False)
    load_dataset(f.name, streaming=True)
```

### Expected behavior

complete type annotations

### Environment info

/

Comments: Good catch! Feel free to open a PR with the suggested fix :).

There is also the [`MockDownloadManager`](https://github.com/JonasLoos/datasets/blob/main/src/datasets/download/mock_download_manager.py#L33), which seems like it might get passed here too. However, to me, it doesn't really seem relevant to the users of the datasets library, so I would just ignore it. What do you think, @mariosasko?

The API (`dummy_data` CLI command ) that uses the `MockDownloadManager` has been deprecated, so ignoring it sounds good!
Add CLIMATE-FEVER dataset
Opened by tdiggelm (closed)

As suggested by @SBrandeis , fresh PR that adds CLIMATE-FEVER. Replaces PR #1579.

---

A dataset adopting the FEVER methodology that consists of 1,535 real-world claims regarding climate-change collected on the internet. Each claim is accompanied by five manually annotated evidence sentences retrieved from the English Wikipedia that support, refute or do not give enough information to validate the claim totalling in 7,675 claim-evidence pairs. The dataset features challenging claims that relate multiple facets and disputed cases of claims where both supporting and refuting evidence are present.

More information can be found at:

* Homepage: http://climatefever.ai
* Paper: https://arxiv.org/abs/2012.00614

Comments: Thank you @lhoestq for your comments! 😄 I added your suggested changes, ran the tests and regenerated `dataset_infos.json` and `dummy_data`.
load_dataset() does not support tsv
Opened by dipsivenkatesh (closed)

### Feature request

the load_dataset() for local functions support file types like csv, json etc but not of type tsv (tab separated values).

### Motivation

cant easily load files of type tsv, have to convert them to another type like csv then load

### Your contribution

Can try by raising a PR with a little help, currently went through the code but didn't fully understand

Comments: #self-assign

Hi @dipsivenkatesh,

Please note that this functionality is already implemented. Our CSV builder uses `pandas.read_csv` under the hood, and you can pass the parameter `delimiter="\t"` to read TSV files.

See the list of CSV config parameters in our docs: https://huggingface.co/docs/datasets/package_reference/loading_methods#datasets.packaged_modules.csv.CsvConfig
Add Multiface dataset
Opened by osanseviero (open)

## Adding a Dataset
- **Name:** Multiface dataset
- **Description:** f high quality recordings of the faces of 13 identities, each captured in a multi-view capture stage performing various facial expressions. An average of 12,200 (v1 scripts) to 23,000 (v2 scripts) frames per subject with capture rate at 30 fps
- **Data:** https://github.com/facebookresearch/multiface

The whole dataset is 65TB though, so I'm not sure

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/main/ADD_NEW_DATASET.md).


Comments: Hi @osanseviero I would like to add this dataset.

Hey @nandwalritik! Thanks for offering to help!

This dataset might be somewhat complex and I'm concerned about it being 65 TB, which would be quite expensive to host. @lhoestq @mariosasko I would love your input if you think it's worth adding this dataset.

Thanks for proposing this interesting dataset, @osanseviero.

Please note that the data files are already hosted in a third-party server: e.g. the index of data files for entity "6795937" is at https://fb-baas-f32eacb9-8abb-11eb-b2b8-4857dd089e15.s3.amazonaws.com/MugsyDataRelease/v0.0/identities/6795937/index.html 
- audio.tar: https://fb-baas-f32eacb9-8abb-11eb-b2b8-4857dd089e15.s3.amazonaws.com/MugsyDataRelease/v0.0/identities/6795937/audio.tar
- ...

Therefore, in principle, we don't need to host them on our Hub: it would be enough to just implement a loading script in the corresponding Hub dataset repo, e.g. "facebook/multiface"...
Add LIAR dataset
Opened by hugoabonizio (closed)

Add LIAR dataset from [“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection](https://www.aclweb.org/anthology/P17-2067/).

Comments: @lhoestq done! The failing testes don't seem to be related, it seems to be a connection issue, if I understand it correctly.

merging since the CI is fixed on master
Support multiprocessing for download/extract nestedly
Opened by hgt312 (open)

### Feature request

Current multiprocessing for download/extract is not done nestedly. For example, when processing SlimPajama, there is only 3 processes (for train/test/val), while there are many files inside these 3 folders

```
Downloading data files #0:   0%|          | 0/1 [00:00<?, ?obj/s]
Downloading data files #1:   0%|          | 0/1 [00:00<?, ?obj/s]
Downloading data files #2:   0%|          | 0/1 [00:00<?, ?obj/s]

Extracting data files #0:   0%|          | 0/1 [00:00<?, ?obj/s]
Extracting data files #1:   0%|          | 0/1 [00:00<?, ?obj/s][A
Extracting data files #2:   0%|          | 0/1 [00:00<?, ?obj/s][A[A
```

### Motivation

speedup dataset loading

### Your contribution

I can help test the feature
Add Hypersim dataset
Opened by osanseviero (open)

## Adding a Dataset
- **Name:** Hypersim
- **Description:** photorealistic synthetic dataset for holistic indoor scene understanding
- **Paper:** *link to the dataset paper if available*
- **Data:** https://github.com/apple/ml-hypersim

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
[Tests] Test files locally
Opened by patrickvonplaten (closed)

This PR adds a `aws` and a `local` decorator to the tests so that tests now run on the local datasets. 

By default, the `aws` is deactivated and `local` is activated and `slow` is deactivated, so that only 1 test per dataset runs on circle ci. 

**When local is activated all folders in `./datasets` are tested.**

**Important** When adding a dataset, we should no longer upload it to AWS. The steps are:
1. Open a PR
2. Add a dataset as described in `datasets/README.md`
3. If all tests pass, push to master

Currently we have 49 functional datasets in our code base. 

We have 6 datasets "under-construction" that don't pass the tests - so I put them in a folder "datasets_under_construction" - it would be nice to open a PR to fix them and put them in the `datasets` folder.

**Important** when running tests locally, the datasets are cached so to rerun them delete your local cache via:
`rm -r ~/.cache/huggingface/datasets/*` 

@thomwolf @mariamabarham @lhoestq 

Comments: Super nice, good job @patrickvonplaten!
Give a user feedback if the dataset he loads is streamable or not
Opened by philschmid (open)

**Is your feature request related to a problem? Please describe.**
I would love to know if a `dataset` is with the current implementation streamable or not. 

**Describe the solution you'd like**
We could show a warning when a dataset is loaded with `load_dataset('...',streaming=True)` when its lot streamable, e.g. if it is an archive. 

**Describe alternatives you've considered**
Add a new metadata tag for "streaming"


Comments: #self-assign

I understand it already raises a `NotImplementedError` exception, eg:

```
>>> dataset = load_dataset("journalists_questions", name="plain_text", split="train", streaming=True)

[...]
NotImplementedError: Extraction protocol for file at https://drive.google.com/uc?export=download&id=1CBrh-9OrSpKmPQBxTK_ji6mq6WTN_U9U is not implemented yet
```
Support saving datasets with custom formatting
Opened by mariosasko (open)

Requested in https://discuss.huggingface.co/t/using-set-transform-on-a-dataset-leads-to-an-exception/53036.

I am not sure if supporting this is the best idea for the following reasons:

>For this to work, we would have to pickle a custom transform, which means the transform and the objects it references need to be serializable. Also, deserializing these bytes would make `load_from_disk` unsafe, so I'm not sure this is a good idea. 

@lhoestq WDYT?


Comments: Not a fan of pickling this sort of stuff either.
Note that users can also share the code in their dataset documentation.
Add support for axis in concatenate datasets
Opened by albertvillanova (closed)

Add support for `axis` (0 or 1) in `concatenate_datasets`.

Close #853.

Comments: @lhoestq I am going to implement the consolidation step you mentioned in #1870.

@lhoestq I was thinking that the order of the TableBlocks is not relevant, isn't it?

I mean, in order to consolidate _consecutive_ in-memory table blocks, in this case:
```
blocks = [in_memory_1, memory_mapped, in_memory_2]
```
I could reorder the list:
```
blocks = [in_memory_1, in_memory_2, memory_mapped]
```
so that the first 2 can be consolidated into a single one:
```
blocks = [in_memory_3, memory_mapped]
```

I think the order is important, users won't expect the dataset to be "shuffled" when they add a new item

> I think the order is important, users won't expect the dataset to be "shuffled" when they add a new item

OK, therefore I leave `_consolidate_blocks` as it is, which currently keeps the order of the blocks (no shuffling).

Thank you guys for implementing this. Minor thing I noticed in the [documentation](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.concatenate_datasets): it says "Converts a list of Dataset with **the same schema** into a single Dataset". With the addition of the axis parameter, perhaps this should be reworded, no?
Add C4
Opened by lhoestq (closed)

## Adding a Dataset
- **Name:** *C4*
- **Description:** *https://github.com/allenai/allennlp/discussions/5056*
- **Paper:** *https://arxiv.org/abs/1910.10683*
- **Data:** *https://huggingface.co/datasets/allenai/c4*
- **Motivation:** *Used a lot for pretraining*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Should fix https://github.com/huggingface/datasets/issues/1710

Comments: Update on this: I'm computing the checksums of the data files. It will be available soon

Added in #2575 :)
Support huggingface_hub 0.5
Opened by lhoestq (closed)

Following https://github.com/huggingface/datasets/issues/4105

`huggingface_hub` deprecated some parameters in `HfApi` in 0.5. This PR updates all the calls to HfApi to remove all the deprecations, 
and I set the `hugginface_hub` requirement to `>=0.5.0`


cc @adrinjalali @LysandreJik 

Comments: Looks like GH actions is not able to resolve `huggingface_hub` 0.5.0, I'm investivating

_The documentation is not available anymore as the PR was closed or merged._

I'm glad to see changes in `huggingface_hub` are simplifying code here.

seems to supersede #4102, feel free to close mine :)

maybe just cherry-pick the docstring fix

I think I've found the issue:
- https://github.com/huggingface/huggingface_hub/pull/790

Good catch, `huggingface_hub` doesn't support python 3.6 anymore indeed, therefore we should keep support for 0.4.0. I'm reverting the requirement version bump for now.

We can update the requirement once we drop support for python 3.6 in `datasets`

@lhoestq, I've opened this PR on `huggingface_hub`: 
- https://github.com/huggingface/huggingface_hub/pull/823

Is there any strong reason why `huggingface_hub` no longer supports Python 3.6? 

I think `datasets` can drop support for 3.6 soon. But for now maybe let's keep support for 0.4.0, python 3.6 users are not affected by https://github.com/huggingface/datasets/issues/4105 anyway.

`huggingface_hub` doesn't not have to support 3.6 again just for the CI IMO

@lhoestq I commented on the PR, that IMO it is not a good practice to drop support for Python 3.6 without a previous deprecation cycle.

Re-added support for older versions. I ended up checking `huggingface_hub` version to use the old, deprecated API for <0.5.0

I find it good practice to have all dependency version related code in a single file so that when you decide to remove support for an old version of a dependency it's easy to find and remove them, hence suggesting `utils/_fixes.py` in https://github.com/huggingface/datasets/issues/4105#issuecomment-1090041204

good idea, thanks !

I used your suggestion @adrinjalali , I just replace the try/except with a check on the version of `huggingface_hub`
Fix time type `_arrow_to_datasets_dtype` conversion
Opened by mariosasko (closed)

Fix #4620

The issue stems from the fact that `pa.array([time_data]).type` returns `DataType(time64[unit])`, which doesn't expose the `unit` attribute, instead of `Time64Type(time64[unit])`. I believe this is a bug in PyArrow. Luckily, the both types have the same `str()`, so in this PR I call `pa.type_for_alias(str(type))` to convert them both to the `Time64Type(time64[unit])` format.

cc @severo 

Comments: _The documentation is not available anymore as the PR was closed or merged._
FewRel
Opened by dspoka (closed)

## Adding a Dataset
- **Name:** FewRel
- **Description:**  Large-Scale Supervised Few-Shot Relation Classification Dataset
- **Paper:** @inproceedings{han2018fewrel,
               title={FewRel:A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation},
               author={Han, Xu and Zhu, Hao and Yu, Pengfei and Wang, Ziyun and Yao, Yuan and Liu, Zhiyuan and Sun, Maosong},
               booktitle={EMNLP},
               year={2018}}
- **Data:** https://github.com/ProKil/FewRel
- **Motivation:** relationship extraction dataset that's been used by some state of the art systems that should be incorporated.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Comments: +1

@dspoka Please check the following link : https://github.com/thunlp/FewRel
This link mentions two versions of the datasets. Also, this one seems to be the official link.

I am assuming this is the correct link and implementing based on the same.

Hi @lhoestq,

This issue can be closed, I guess.

Yes :) closing
Thanks again for adding FewRel !

Thanks for adding this @gchhablani ! Sorry didn't see the email notifications sooner!
[doc] Fix broken snippet that had too many quotes
Opened by tomaarsen (closed)

Hello!

### Pull request overview
* Fix broken snippet in https://huggingface.co/docs/datasets/main/en/process that has too many quotes

### Details
The snippet in question can be found here: https://huggingface.co/docs/datasets/main/en/process#map
This screenshot shows the issue, there is a quote too many, causing the snippet to be colored incorrectly:
![image](https://user-images.githubusercontent.com/37621491/190640627-f7587362-0e44-4464-a5d1-a0b98df6986f.png)

The change speaks for itself.

Thank you for the detailed documentation, by the way. 

- Tom Aarsen


Comments: _The documentation is not available anymore as the PR was closed or merged._

Spent the day familiarising myself with the huggingface line of products, and happened to run into some small issues here and there. Magically, I've found exactly one small issue in `transformers`, one in `accelerate` and now one in `datasets`, hah!

As for this PR, the issue seems solved according to the [new PR documentation](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4986/en/process#map):
![image](https://user-images.githubusercontent.com/37621491/190646405-6afa06fa-9eac-48f6-ab30-2677944fb7b6.png)