rIsHu009/github-issues-updated huggingface dataset:
Title: Add ljspeech
Author: patrickvonplaten (CONTRIBUTOR)
State: closed
Created: 2021-02-08T13:22:26Z
Labels: dataset request

speech

## Adding a Dataset
- **Name:** *ljspeech*
- **Description:** *This is a public domain speech dataset consisting of 13,100 short audio clips of a single speaker reading passages from 7 non-fiction books. A transcription is provided for each clip. Clips vary in length from 1 to 10 seconds and have a total length of approximately 24 hours.

The texts were published between 1884 and 1964, and are in the public domain. The audio was recorded in 2016-17 by the LibriVox project and is also in the public domain.)*
- **Paper:** *Homepage*: https://keithito.com/LJ-Speech-Dataset/
- **Data:** *https://keithito.com/LJ-Speech-Dataset/*
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/ljspeech
If interested in tackling this issue, feel free to tag @patrickvonplaten


Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Title: Add common voice
Author: patrickvonplaten (CONTRIBUTOR)
State: closed
Created: 2021-02-08T13:21:05Z
Labels: dataset request

speech

## Adding a Dataset
- **Name:** *common voice*
- **Description:** *Mozilla Common Voice Dataset*
- **Paper:** Homepage: https://voice.mozilla.org/en/datasets
- **Data:** https://voice.mozilla.org/en/datasets
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/common_voice
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
I have started working on adding this dataset.

Hey @BirgerMoell - awesome that you started working on Common Voice. Common Voice is a bit special since, there is no direct download link to download the data. In these cases we usually consider two options:

1) Find a hacky solution to extract the download link somehow from the XLM tree of the website 
2) If this doesn't work we force the user to download the data himself and add a `"data_dir"` as an input parameter. E.g. you can take a look at how it is done for [this](https://github.com/huggingface/datasets/blob/66f2a7eece98d2778bd22bb5034cb7c2376032d4/datasets/arxiv_dataset/arxiv_dataset.py#L66) 

Also the documentation here: https://huggingface.co/docs/datasets/add_dataset.html?highlight=data_dir#downloading-data-files-and-organizing-splits (especially the "note") might be helpful.

Let me know if you have any other questions

I added a Work in Progress pull request (hope that is ok). I've made a card for the dataset and filled out the common_voice.py file with information about the datset (not completely).

I didn't manage to get the tagging tool working locally on my machine but will look into that later.

Left to do.

- Tag the dataset
- Add missing information and update common_voice.py

https://github.com/huggingface/datasets/pull/1886

Awesome! I left a longer comment on the PR :-)

I saw that this current datasets package holds common voice version 6.1, how to add the new version 7.0 that is already available?

Will me merged next week - we're working on it :-)

Common voice still appears to be a 6.1. Is the plan still to upgrade to 7.0?

We actually already have the code and everything ready to add Common Voice 7.0 to `datasets` but are still waiting for the common voice authors to give us the green light :-) 

Also gently pinging @phirework and @milupo here

Common Voice 7.0 is available here now: https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0

For anyone else stumbling upon this thread, the 8.0 version is also available now: https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0
Title: Add Voxforge
Author: patrickvonplaten (CONTRIBUTOR)
State: open
Created: 2021-02-08T13:19:56Z
Labels: dataset request

speech

## Adding a Dataset
- **Name:** *voxforge* 
- **Description:** *VoxForge is a language classification dataset. It consists of user submitted audio clips submitted to the website. In this release, data from 6 languages is collected - English, Spanish, French, German, Russian, and Italian. Since the website is constantly updated, and for the sake of reproducibility, this release contains only recordings submitted prior to 2020-01-01. The samples are splitted between train, validation and testing so that samples from each speaker belongs to exactly one split.*
- **Paper:** *Homepage*: http://www.voxforge.org/
- **Data:** *http://www.voxforge.org/home/downloads*
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/voxforge
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Title: Add tedlium
Author: patrickvonplaten (CONTRIBUTOR)
State: closed
Created: 2021-02-08T13:17:52Z
Labels: dataset request

speech

## Adding a Dataset
- **Name:** *tedlium*
- **Description:** *The TED-LIUM 1-3 corpus is English-language TED talks, with transcriptions, sampled at 16kHz. It contains about 118 hours of speech.*
- **Paper:** Homepage: http://www.openslr.org/7/, https://lium.univ-lemans.fr/en/ted-lium2/ &, https://www.openslr.org/51/
- **Data:** http://www.openslr.org/7/
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/tedlium
If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Hi @patrickvonplaten 
I can have a look to this dataset later since I am trying to add the OpenSLR dataset https://github.com/huggingface/datasets/pull/2173
Hopefully I have enough space since the compressed file is 21GB. The release 3 is even bigger: 54GB :-0

Resolved via https://github.com/huggingface/datasets/pull/4309
Title: Add VCTK
Author: patrickvonplaten (CONTRIBUTOR)
State: closed
Created: 2021-02-08T13:15:28Z
Labels: dataset request

speech

## Adding a Dataset
- **Name:** *VCTK*
- **Description:** *This CSTR VCTK Corpus includes speech data uttered by 110 English speakers with various accents. Each speaker reads out about 400 sentences, which were selected from a newspaper, the rainbow passage and an elicitation paragraph used for the speech accent archive.*
- **Paper:** Homepage: https://datashare.ed.ac.uk/handle/10283/3443
- **Data:** https://datashare.ed.ac.uk/handle/10283/3443
- **Motivation:** Important speech dataset
- **TFDatasets Implementation**: https://www.tensorflow.org/datasets/catalog/vctk

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
@patrickvonplaten I'd like to take this, if nobody has already done it. I have added datasets before through the datasets sprint, but I feel rusty on the details, so I'll look at the guide as well as similar audio PRs (#1878 in particular comes to mind). If there is any detail I should be aware of please, let me know! Otherwise, I'll try to write up a PR in the coming days.

That sounds great @jaketae - let me know if you need any help i.e. feel free to ping me on a first PR :-)
Title: test.json has been removed from the limit dataset repo (breaks dataset)
Author: Paethon (NONE)
State: closed
Created: 2021-02-08T12:45:53Z
Labels: dataset bug

https://github.com/huggingface/datasets/blob/16042b233dbff2a7585110134e969204c69322c3/datasets/limit/limit.py#L51

The URL is not valid anymore since test.json has been removed in master for some reason. Directly referencing the last commit works:

`https://raw.githubusercontent.com/ilmgut/limit_dataset/0707d3989cd8848f0f11527c77dcf168fefd2b23/data`

Comments:
Thanks for the heads up ! I'm opening a PR to fix that
Title: Add CHiME4 dataset
Author: patrickvonplaten (CONTRIBUTOR)
State: open
Created: 2021-02-08T12:36:38Z
Labels: dataset request

speech

## Adding a Dataset
- **Name:** Chime4
- **Description:** Chime4 is a dataset for automatic speech recognition. It is especially useful for evaluating models in a noisy environment and for multi-channel ASR
- **Paper:** Dataset comes from a channel: http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/ . Results paper: 
- **Data:** http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/download.html
- **Motivation:** So far there are very little datasets for speech in `datasets`. Only `lbirispeech_asr` so far.

If interested in tackling this issue, feel free to tag @patrickvonplaten

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
@patrickvonplaten not sure whether it is still needed, but willing to tackle this issue

Hey @patrickvonplaten, I have managed to download the zip on [here]( http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/download.html) and successfully uploaded all the files on a hugging face dataset: 

https://huggingface.co/datasets/ksbai123/Chime4

However I am getting this error when trying to use the dataset viewer:

![Screenshot 2023-12-27 at 18 40 59](https://github.com/huggingface/datasets/assets/35923560/a5a9ed3d-8dbd-41c4-b83a-4e80728b1450)

Can you take a look and let me know if I have missed any files please

@patrickvonplaten ?

Hi @KossaiSbai,

Thanks for your contribution.

As the issue is not strictly related to the `datasets` library, but to the specific implementation of the CHiME4 dataset, I have opened an issue in the Discussion tab of the dataset: https://huggingface.co/datasets/ksbai123/Chime4/discussions/2
Let's continue the discussion there!

Hi, please help me understand this dataset. I am trying to run ESPnet chime4 recepie. I downloaded the dataset (https://www.chimechallenge.org/challenges/chime4/download) and it doesn't look like the official site (https://www.chimechallenge.org/challenges/chime4/data#Audio):

<img width="220" alt="Image" src="https://github.com/user-attachments/assets/91d2d727-bc65-4c45-ad87-b643fc26a40e" />

There is no 
├── audio
├── transcriptions
└── WSJ0
only 
├── annotations

same in your huggingface card. 
How can I access the missing data?
Title: Add OSCAR dataset card
Author: pjox (CONTRIBUTOR)
State: closed
Created: 2021-02-08T01:39:49Z

I added more information and completed the dataset card for OSCAR which was started by @lhoestq in his previous [PR](https://github.com/huggingface/datasets/pull/1824).

Comments:
@lhoestq Thanks for the suggestions! I agree with all of them. Should I accept them one by one or can I accept them all at once? When I try to load the whole diff GitHub is complaining and it does no render them well (probably my browser?) 😅 

I just merged the tables as suggested 😄 . However I noticed something weird, the train sizes are identical for both the original and deduplicated files ... This is not normal, in general the original files are almost twice as big as the deduplicated ones 🤔 

Good catch @pjox ! I just checked and this is because the scripts doesn't handle having several blank lines in a row.
Blank lines introduced by deduplication are currently not ignored so we end up with the same number of examples in the dataset as the original version (but with empty examples...)
I fixed that in this [commit](https://github.com/huggingface/datasets/commit/837a152e4724adc5308e2c4481908c00a8d93383). I'm re-running the metadata generation for deduplicated configs.

I got the new sizes today, will update the dataset_infos.json and the dataset card tomorrow

> I got the new sizes today, will update the dataset_infos.json and the dataset card tomorrow

great, I just wanted to report that I got error message "NonMatchingSplitsSizesError" when I tried to load one of the oscar dataset.

Hi @cahya-wirawan, which configuration of oscar do you have this issue with ?

Ok I see you're having this issue because I haven't updated the sizes yet ! I'm opening a PR

I just checked and indeed there's an issue with the `deduplicated` configurations since the commit I mentioned above.
I'm fixing this by using the new sizes I got yesterday :) 


I just updated the size in the table @pjox it should be good now :) 
I also updated the sizes in the dataset_infos.json in https://github.com/huggingface/datasets/pull/1868 (merged)

Thanks @lhoestq for fixing the issue, it works now

Thank you so much @lhoestq !
Title: Looks like nokogumbo is up-to-date now, so this is no longer needed.
Author: JimmyJim1 (NONE)
State: closed
Created: 2021-02-07T06:52:07Z

Looks like nokogumbo is up-to-date now, so this is no longer needed.

__Originally posted by @dependabot in https://github.com/discourse/discourse/pull/11373#issuecomment-738993432__

Comments:
Title: Add Tweet Eval Dataset
Author: gchhablani (CONTRIBUTOR)
State: closed
Created: 2021-02-06T12:36:25Z

Closes Draft PR #1407. 

Notes:
1. I have excluded `mapping.txt` from the dataset at it only contained the name mappings, which are already present in the ClassLabels.
2. I have also exluded the textual names for the emojis mentioned in the [mapping](https://github.com/cardiffnlp/tweeteval/blob/main/datasets/emoji/mapping.txt).
3. I do not understand @abhishekkrthakur's example generator on #1407. Maybe he was trying to build up on code from some other dataset.

Requesting @lhoestq  to review.

Comments:
Title: Regarding On-the-fly Data Loading
Author: gchhablani (CONTRIBUTOR)
State: closed
Created: 2021-02-05T17:43:48Z

Hi,

I was wondering if it is possible to load images/texts as a batch during the training process, without loading the entire dataset on the RAM at any given point.

Thanks,
Gunjan

Comments:
Possible duplicate

#1776 https://github.com/huggingface/datasets/issues/

really looking PR for this feature

Hi @acul3 

Issue #1776 talks about  doing on-the-fly data pre-processing, which I think is solved in the next release as mentioned in the issue #1825. I also look forward to using this feature, though :)

I wanted to ask about on-the-fly data loading from the cache (before pre-processing).

Hi ! Currently when you load a dataset via `load_dataset` for example, then the dataset is memory-mapped from an Arrow file on disk. Therefore there's almost no RAM usage even if your dataset contains TB of data.
Usually at training time only one batch of data at a time is loaded in memory.

Does that answer your question or were you thinking about something else ?

Hi @lhoestq,

I apologize for the late response. This answers my question. Thanks a lot.
Title: Add OSCAR dataset card
Author: lhoestq (MEMBER)
State: closed
Created: 2021-02-05T10:30:26Z

I started adding the dataset card for OSCAR !

For now it's just basic info for all the different configurations in `Dataset Structure`.
In particular the Data Splits section tells how may samples there are for each config. The Data Instances section show an example for each config, and it also shows the size in MB. Since the Data Instances section is very long the user has to click to expand the info. I was able to generate it thanks to the tools made by  @madlag and @yjernite :D

Cc @pjox could you help me with the other sections ? (Dataset Description, Dataset Creation, Considerations for Using the Data, Additional Information)

Comments:
Hi @lhoestq! When are you planning to release the version with this dataset?

BTW: What a huge README file :astonished:

Next week !

Closing in favor of #1833
Title: Provide better exception message when one of many files results in an exception
Author: david-waterworth (NONE)
State: closed
Created: 2021-02-05T00:49:03Z

I find when I process many files, i.e.

```
train_files = glob.glob('rain*.csv')
validation_files = glob.glob(validation*.csv')
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files))
```

I sometimes encounter an error due to one of the files being misformed (i.e. no data, or a comma in a field that isn't quoted, etc).

For example, this is the tail of an exception which I suspect is due to a stray comma.

>   File "pandas/_libs/parsers.pyx", line 756, in pandas._libs.parsers.TextReader.read
>   File "pandas/_libs/parsers.pyx", line 783, in pandas._libs.parsers.TextReader._read_low_memory
>   File "pandas/_libs/parsers.pyx", line 827, in pandas._libs.parsers.TextReader._read_rows
>   File "pandas/_libs/parsers.pyx", line 814, in pandas._libs.parsers.TextReader._tokenize_rows
>   File "pandas/_libs/parsers.pyx", line 1951, in pandas._libs.parsers.raise_parser_error
> pandas.errors.ParserError: Error tokenizing data. C error: Expected 2 fields in line 559, saw 3

It would be nice if the exception trace contained the name of the file being processed (I have 250 separate files!)

Comments:
Hi!

Thank you for reporting this issue. I agree that the information about the exception should be more clear and explicit.

I could take on this issue.

On the meantime, as you can see from the exception stack trace, HF Datasets uses pandas to read the CSV files. You can pass arguments to `pandas.read_csv` by passing additional keyword arguments to `load_dataset`. For example, you may find useful this argument:
- `error_bad_lines` : bool, default True
  Lines with too many fields (e.g. a csv line with too many commas) will by default cause an exception to be raised, and no DataFrame will be returned. If False, then these “bad lines” will be dropped from the DataFrame that is returned.

You could try:
```python
datasets = load_dataset("csv", data_files=dict(train=train_files, validation=validation_files), error_bad_lines=False)
```
Title: Add metrics usage examples and tests
Author: lhoestq (MEMBER)
State: closed
Created: 2021-02-04T18:23:50Z

All metrics finally have usage examples and proper fast + slow tests :)

I added examples of usage for every metric, and I use doctest to make sure they all work as expected.

For "slow" metrics such as bert_score or bleurt which require to download + run a transformer model, the download + forward pass are only done in the slow test.
In the fast test on the other hand, the download + forward pass are monkey patched.

Metrics that need to be installed from github are not added to setup.py because it prevents uploading the `datasets` package to pypi.
An additional-test-requirements.txt file is used instead. This file also include `comet` in order to not have to resolve its *impossible* dependencies.

Also `comet` is not tested on windows because one of its dependencies (fairseq) can't be installed in the CI for some reason.

Comments:
Title: pyarrow.lib.ArrowInvalid: Column 1 named input_ids expected length 599 but got length 1500
Author: LuCeHe (NONE)
State: closed
Created: 2021-02-04T02:30:23Z

I am trying to preprocess any dataset in this package with GPT-2 tokenizer, so I need to structure the datasets as long sequences of text without padding. I've been following a couple of your tutorials and here you can find the script that is failing right at the end

https://github.com/LuCeHe/GenericTools/blob/master/KerasTools/lm_preprocessing.py

In the last iteration of the last dset.map, it gives the error that I copied in the title. Another issue that I have, if I leave the batch_size set as 1000 in the last .map, I'm afraid it's going to lose most text, so I'm considering setting both writer_batch_size and batch_size to 300 K, but I'm not sure it's the best way to go.

Can you help me?
Thanks!

Comments:
Hi !
The error you have is due to the `input_ids` column not having the same number of examples as the other columns.
Indeed you're concatenating the `input_ids` at this line:

https://github.com/LuCeHe/GenericTools/blob/431835d8e13ec24dceb5ee4dc4ae58f0e873b091/KerasTools/lm_preprocessing.py#L134

However the other columns are kept unchanged, and therefore you end up with an `input_ids` column with 599 elements while the others columns like `attention_mask` have 1500.

To fix that you can instead concatenate them all using
```python
concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
```

Also you may need to drop the "text" column before applying `group_texts` since strings can't be concatenated with lists. You can drop it at the tokenization step:
```python
dset = dset.map(
    tokenize_function,
    batched=True,
    remove_columns=["text"]
)
```

You saved my life.
Title: Support future datasets
Author: lhoestq (MEMBER)
State: closed
Created: 2021-02-03T15:26:49Z

If a dataset is available at the version of the local installation of `datasets` (e.g. 1.2.0), then loading this dataset means loading the script at this version.

However when trying to load a dataset that is only available on master, currently users have to specify `script_version="master"` in `load_dataset` to make it work.

However we could automatically get the dataset from master instead in this case.

I added this feature in this PR.
I also added a warning if a dataset is not available at the version of the local installation of `datasets` but is loaded from master:
```python
>>> load_dataset("silicone", "dyda_da")
Couldn't find file locally at silicone/silicone.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.2.0/datasets/silicone/silicone.py.
The file was picked from the master branch on github instead at https://raw.githubusercontent.com/huggingface/datasets/master/datasets/silicone/silicone.py.
Downloading and preparing dataset silicone/dyda_da (download: 8.46 MiB, generated: 9.39 MiB, post-processed: Unknown size, total: 17.86 MiB) to /Users/quentinlhoest/.cache/huggingface/datasets/silicone/dyda_da/1.0.0/d41d8c0b73c6df035b1369c45774418f0051163ea689b5502b8bda783adf6342...
...
```

Comments:
Title: Unable to add Multi-label Datasets
Author: gchhablani (CONTRIBUTOR)
State: closed
Created: 2021-02-02T11:50:56Z

I am trying to add [CIFAR-100](https://www.cs.toronto.edu/~kriz/cifar.html) dataset. The dataset contains two labels per image - `fine label` and `coarse label`. Using just one label in supervised keys as 
`supervised_keys=("img", "fine_label")` raises no issue. But trying `supervised_keys=("img", "fine_label","coarse_label")` leads to this error : 

```python
Traceback (most recent call last):
  File "test_script.py", line 2, in <module>
    d = load_dataset('./datasets/cifar100')
  File "~/datasets/src/datasets/load.py", line 668, in load_dataset
    **config_kwargs,
  File "~/datasets/src/datasets/builder.py", line 896, in __init__
    super(GeneratorBasedBuilder, self).__init__(*args, **kwargs)
  File "~/datasets/src/datasets/builder.py", line 247, in __init__
    info.update(self._info())
  File "~/.cache/huggingface/modules/datasets_modules/datasets/cifar100/61d2489b2d4a4abc34201432541b7380984ec714e290817d9a1ee318e4b74e0f/cifar100.py", line 79, in _info
    citation=_CITATION,
  File "<string>", line 19, in __init__
  File "~/datasets/src/datasets/info.py", line 136, in __post_init__
    self.supervised_keys = SupervisedKeysData(*self.supervised_keys)
TypeError: __init__() takes from 1 to 3 positional arguments but 4 were given
```
Is there a way I can fix this?

Also, what does adding `supervised_keys` do? Is it necessary? How would I specify `supervised_keys` for a multi-input, multi-label dataset?

Thanks,
Gunjan

Comments:
Thanks for adding this dataset! As far as I know `supervised_keys` is mostly a holdover from TFDS, but isn't really used, so feel free to drop it (@lhoestq  or @thomwolf correct me if I'm wrong). It definitely shouldn't be blocking :) 

I can confirm that it comes from TFDS and is not used at the moment.

Thanks @yjernite @lhoestq 

The template for new dataset makes it slightly confusing. I suppose the comment suggesting its update can be removed.

Closing this issue since it was answered.
Title: Add Hateful Memes Dataset
Author: gchhablani (CONTRIBUTOR)
State: open
Created: 2021-02-02T10:53:59Z
Labels: dataset request

vision

## Add Hateful Memes Dataset
- **Name:** Hateful Memes
- **Description:** [https://ai.facebook.com/blog/hateful-memes-challenge-and-data-set]( https://ai.facebook.com/blog/hateful-memes-challenge-and-data-set)
- **Paper:** [https://arxiv.org/pdf/2005.04790.pdf](https://arxiv.org/pdf/2005.04790.pdf)
- **Data:** [This link](https://drivendata-competition-fb-hateful-memes-data.s3.amazonaws.com/XjiOc5ycDBRRNwbhRlgH.zip?AWSAccessKeyId=AKIARVBOBDCY4MWEDJKS&Signature=DaUuGgZWUgDHzEPPbyJ2PhSJ56Q%3D&Expires=1612816874)
- **Motivation:** Including multi-modal datasets to 🤗 datasets.

I will be adding this dataset. It requires the user to sign an agreement on DrivenData. So, it will be used with a manual download.

The issue with this dataset is that the images are of different sizes. The image datasets added so far (CIFAR-10 and MNIST) have a uniform shape throughout.
So something like 
```python
 datasets.Array2D(shape=(28, 28), dtype="uint8")
```
won't work for the images. How would I add image features then? I checked `datasets/features.py` but couldn't figure out the appropriate class for this. I'm assuming I would want to avoid re-sizing at all since we want the user to be able to access the original images.

Also, in case I want to load only a subset of the data, since the actual data is around 8.8GB, how would that be possible?

Thanks,
Gunjan

Comments:
I am not sure, but would `datasets.Sequence(datasets.Sequence(datasets.Sequence(datasets.Value("int")))` work?

Also, I found the information for loading only subsets of the data [here](https://github.com/huggingface/datasets/blob/master/docs/source/splits.rst).

Hi @lhoestq,

Request you to check this once.

Thanks,
Gunjan

Hi @gchhablani since Array2D doesn't support images of different sizes, I would suggest to store in the dataset the paths to the image file instead of the image data. This has the advantage of not decompressing the data (images are often compressed using jpeg, png etc.). Users can still apply `.map` to load the images if they want to. Though it would en up being Sequences features.

In the future we'll add support for ragged tensors for this case and update the relevant dataset with this feature.
Title: Add FreebaseQA dataset
Author: gchhablani (CONTRIBUTOR)
State: closed
Created: 2021-02-02T08:35:53Z

Adding FreebaseQA dataset suggested in PR #1435 with minor edits. Also closes that PR.
Requesting @lhoestq to review.

Comments:
Hi ! It looks like this PR contains changes about other datasets than freebase_qa such as DuoRC.

Can you remove these changes please ?

Hi @lhoestq,

I think this happened because of rebasing. I'm unable to remove the duorc commit from the branch.  GEM, Arabic sarcasm datasets are also there. I can't see any merge conflicts, however. Before commiting I always rebase (shouldn't have done that).
Can you explain what is to be done? Should I create a clean PR?

Hi @gchhablani 
I think you can simply create another branch and another PR.

If I understand correctly the github diff is messed up because you rebased instead of merge.
Rebasing is supposed to be used only before pushing the branch the first time, or github messes up the diff.
If you want to include changes from master on a branch that is already push you need to use git merge.

Thanks @lhoestq.

I understand the issue now. I missed the instructions on the template. Sorry for bothering you unnecessarily, I'm pretty new to contributing on GitHub. I'll make a fresh PR.


No problem, I'm not a big fan of this weird behavior tbh.
Thanks for making a new PR

@lhoestq Haha, well, it's not as weird as not reading the [instructions](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md#open-a-pull-request-on-the-main-huggingface-repo-and-share-your-work).
Also, I'm enjoying adding new datasets so it's all cool :)
Title: writing Datasets in a human readable format
Author: ghost (NONE)
State: closed
Created: 2021-02-02T02:55:40Z
Labels: enhancement

question

Hi
I see there is a save_to_disk function to save data, but this is not human readable format, is there a way I could save a Dataset object in a human readable  format to a file like json? thanks @lhoestq

Comments:
AFAIK, there is currently no built-in method on the `Dataset` object to do this.
However, a workaround is to directly use the Arrow table backing the dataset, **but it implies loading the whole dataset in memory** (correct me if I'm mistaken @lhoestq).

You can convert the Arrow table to a pandas dataframe to save the data as csv as follows:
```python
arrow_table = dataset.data
dataframe = arrow_table.to_pandas()
dataframe.to_csv("/path/to/file.csv")
```

Similarly, you can convert the dataset to a Python dict and save it as JSON:
```python
import json
arrow_table = dataset.data
py_dict = arrow_table.to_pydict()
with open("/path/to/file.json", "w+") as f:
    json.dump(py_dict, f)
```

Indeed this works as long as you have enough memory.
It would be amazing to have export options like csv, json etc. !

It should be doable to implement something that iterates through the dataset batch by batch to write to csv for example.
There is already an `export` method but currently the only export type that is supported is `tfrecords`.

Hi! `datasets` now supports `Dataset.to_csv` and `Dataset.to_json` for saving data in a human readable format.
Title: Adding an aggregated dataset for the GEM benchmark
Author: yjernite (MEMBER)
State: closed
Created: 2021-02-02T00:39:53Z

This dataset gathers modified versions of several other conditional text generation datasets which together make up the shared task for the Generation Evaluation and Metrics workshop (think GLUE for text generation)

The changes from the original datasets are detailed in the Dataset Cards on the GEM website, which are linked to in this dataset card.

cc @sebastianGehrmann

Comments:
Nice !
Title: add github of contributors
Author: thevasudevgupta (CONTRIBUTOR)
State: closed
Created: 2021-02-01T03:49:19Z

This PR will add contributors GitHub id at the end of every dataset cards.

Comments:
@lhoestq Can you confirm if this format is fine? I will update cards based on your feedback.

On HuggingFace side we also have a mapping of hf user => github user (GitHub info used to be required when signing up until not long ago – cc @gary149 @beurkinger) so we can also add a link to HF profile

All the dataset cards have been updated with GitHub ids :)
Title: [GEM] Updated the source link of the data to update correct tokenized version.
Author: mounicam (CONTRIBUTOR)
State: closed
Created: 2021-01-31T21:17:19Z


Comments:
@mounicam we'll keep the original version in the Turk dataset proper, and use the updated file in the GEM aggregated dataset which I'll add later today

@lhoestq do not merge, I'll close when I've submitted the GEM dataset PR :) 

Closed by https://github.com/huggingface/datasets/pull/1807
Title: Add DuoRC Dataset
Author: gchhablani (CONTRIBUTOR)
State: closed
Created: 2021-01-31T20:01:59Z

Hi,

DuoRC SelfRC is one type of the [DuoRC Dataset](https://duorc.github.io/). DuoRC SelfRC is a crowdsourced Abstractive/Extractive Question-Answering dataset based on Wikipedia movie plots. It contains examples that may have answers in the movie plot, synthesized answers which are not present in the movie plot, or no answers.  I have also added ParaphraseRC - the other type of DuoRC dataset where questions are based on Wikipedia movie plots and answers are based on corresponding IMDb movie plots.

Paper :  [https://arxiv.org/abs/1804.07927](https://arxiv.org/abs/1804.07927)

I want to add this to 🤗 datasets to make it more accessible to the community. I have added  all the details that I could find. Please let me know if anything else is needed from my end.

Thanks,
Gunjan

Comments:
Thanks for approving @lhoestq!
Will apply these changes for the other datasets I've added too.
Title: Update: SWDA - Fixed code to use all metadata features. Added comments and cleaned c…
Author: gmihaila (CONTRIBUTOR)
State: closed
Created: 2021-01-31T19:18:55Z

This is a dataset I currently use my research and I realized some features are not being returned.

Previous code was not using all available metadata and was kind of messy

I fixed code to use all metadata and made some modification to be more efficient and better formatted.


Please let me know if I need to make any changes.

Comments:
@yjernite Pushed all the changes you recommended. Thank you for your help!
Title: Add Arabic sarcasm dataset
Author: mapmeld (CONTRIBUTOR)
State: closed
Created: 2021-01-31T17:38:55Z

This MIT license dataset: https://github.com/iabufarha/ArSarcasm

Via https://sites.google.com/view/ar-sarcasm-sentiment-detection/

Comments:
@lhoestq thanks for the comments - I believe these are now addressed. I re-generated the datasets_info.json and dummy data
Title: Connection error
Author: smile0925 (NONE)
State: closed
Created: 2021-01-30T07:32:45Z

Hi
I am hitting to the error, help me and thanks.

`train_data = datasets.load_dataset("xsum", split="train")`
`ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.0.2/datasets/xsum/xsum.py`

Comments:
Hi ! For future references let me add a link to our discussion here : https://github.com/huggingface/datasets/issues/759#issuecomment-770684693

Let me know if you manage to fix your proxy issue or if we can do something on our end to help you :)
Title: Allow loading dataset in-memory
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-01-29T14:39:50Z

Allow loading datasets either from:
- memory-mapped file (current implementation)
- from file descriptor, copying data to physical memory

Close #708

Comments:
I am wondering how to test their difference...

> ring how to test their difference...

Hmm I don't think pyarrow exposes an API to check if a Table comes from a file that is memory-mapped. In particular since all the buffer/memory logic is in the C++ part of pyarrow.

Otherwise we can still check the difference of RAM used when loading a big chunk of data.

> Hmm I don't think pyarrow exposes an API to check if a Table comes from a file that is memory-mapped. In particular since all the buffer/memory logic is in the C++ part of pyarrow.
> 
> Otherwise we can still check the difference of RAM used when loading a big chunk of data.

@lhoestq I think I found a way: `pa.total_allocated_bytes()` :smirk:
Title: Small fix with corrected logging of train vectors
Author: TezRomacH (CONTRIBUTOR)
State: closed
Created: 2021-01-29T14:26:06Z

Now you can set `train_size` to the whole dataset size via `train_size = -1` and login writes not `Training the index with the first -1 vectors` but (for example) `Training the index with the first 16123 vectors`. And maybe more than dataset length. Logging will be correct

Comments:
Title: How to use split dataset 
Author: kkhan188 (NONE)
State: closed
Created: 2021-01-27T21:37:47Z
Labels: question

![Capture1](https://user-images.githubusercontent.com/78090287/106057436-cb6a1f00-6111-11eb-8c9c-3658065b1fdf.PNG)

Hey,
I want to split the lambada dataset into corpus, test, train and valid txt files (like penn treebank) but I am not able to achieve this. What I am doing is, executing the lambada.py file in my project but its not giving desired results. Any help will be appreciated!

Comments:
By default, all 3 splits will be loaded if you run the following:

```python
from datasets import load_dataset
dataset = load_dataset("lambada")
print(dataset["train"])
print(dataset["valid"])

```

If you wanted to do load this manually, you could do this:

```python
from datasets import load_dataset
data_files = {
   "train": "data/lambada/train.txt",
   "valid": "data/lambada/valid.txt",
   "test": "data/lambada/test.txt",
}
ds = load_dataset("text", data_files=data_files)
```

Thank you for the quick response!
Title: Not enough disk space (Needed: Unknown size) when caching on a cluster
Author: olinguyen (CONTRIBUTOR)
State: closed
Created: 2021-01-27T21:30:59Z

I'm running some experiments where I'm caching datasets on a cluster and accessing it through multiple compute nodes. However, I get an error when loading the cached dataset from the shared disk.

The exact error thrown:

```bash
>>> load_dataset(dataset, cache_dir="/path/to/cluster/shared/path")
OSError: Not enough disk space. Needed: Unknown size (download: Unknown size, generated: Unknown size, post-processed: Unknown size)
```


[`utils.has_sufficient_disk_space`](https://github.com/huggingface/datasets/blob/8a03ab7d123a76ee744304f21ce868c75f411214/src/datasets/utils/py_utils.py#L332) fails on each job because of how the cluster system is designed (`disk_usage(".").free` can't compute on the cluster's shared disk).


This is exactly where the error gets thrown:
https://github.com/huggingface/datasets/blob/master/src/datasets/builder.py#L502

```python
if not utils.has_sufficient_disk_space(self.info.size_in_bytes or 0, directory=self._cache_dir_root):
    raise IOError(
          "Not enough disk space. Needed: {} (download: {}, generated: {}, post-processed: {})".format(
          utils.size_str(self.info.size_in_bytes or 0),
          utils.size_str(self.info.download_size or 0),
          utils.size_str(self.info.dataset_size or 0),
          utils.size_str(self.info.post_processing_size or 0),
       )
    )

```

What would be a good way to circumvent this? my current fix is to manually comment out that part, but that is not ideal. 
Would it be possible to pass a flag to skip this check on disk space?

Comments:
Hi ! 

What do you mean by "disk_usage(".").free` can't compute on the cluster's shared disk" exactly ?
Does it return 0 ?

Yes, that's right. It shows 0 free space even though there is. I suspect it might have to do with permissions on the shared disk.

```python
>>> disk_usage(".")
usage(total=999999, used=999999, free=0)
```

That's an interesting behavior...
Do you know any other way to get the free space that works in your case ?
Also if it's a permission issue could you try fix the permissions and let mus know if that helped ?

I think its an issue on the clusters end (unclear exactly why -- maybe something with docker containers?), will close the issue

Were you able to figure it out?

@philippnoah I had fixed it with a small hack where I patched `has_sufficient_disk_space` to always return `True`. you can do that with an import without having to modify the `datasets` package

@olinguyen Thanks for the suggestion, it works but I had to to edit builder.py in the installed package. Can you please explain how were you able to do this using import?

I was able to patch the builder code in my notebook before the load data call and it works. 
```
import datasets
datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True
```

import datasets
    datasets.builder.has_sufficient_disk_space = lambda needed_bytes, directory='.': True
Title: JSONDecodeError on JSON with multiple lines
Author: gchhablani (CONTRIBUTOR)
State: closed
Created: 2021-01-27T00:19:22Z

Hello :),

I have been trying to load data using a JSON file. Based on the [docs](https://huggingface.co/docs/datasets/loading_datasets.html#json-files), the following format is supported:

```json
{"key1":11, "key2":12, "key3":13}
{"key1":21, "key2":22, "key3":23}
```
 But, when I try loading a dataset with the same format, I get a JSONDecodeError : `JSONDecodeError: Extra data: line 2 column 1 (char 7142)`. Now, this is expected when using `json` to load a JSON file. But I was wondering if there are any special arguments to pass when using `load_dataset` as the docs suggest that this format is supported.

When I convert the JSON file to a list of dictionaries format, I get AttributeError: `AttributeError: 'list' object has no attribute 'keys'`. So, I can't convert them to list of dictionaries either.

Please let me know :)

Thanks,
Gunjan

Comments:
Hi !

The `json` dataset script does support this format. For example loading a dataset with this format works on my side:
```json
{"key1":11, "key2":12, "key3":13}
{"key1":21, "key2":22, "key3":23}
```

Can you show the full stacktrace please ? Also which version of datasets and pyarrow are you using ?



Hi Quentin!

I apologize for bothering you. There was some issue with my pyarrow version as far as I understand. I don't remember the exact version I was using as I didn't check it.

I repeated it with `datasets 1.2.1` and `pyarrow  2.0.0` and it worked.

Closing this issue. Again, sorry for the bother.

Thanks,
Gunjan
Title: Dataset Examples Explorer
Author: ChewKokWah (NONE)
State: closed
Created: 2021-01-26T20:39:02Z

In the Older version of the Dataset, there are a useful Dataset Explorer that allow user to visualize the examples (training, test and validation) of a particular dataset, it is no longer there in current version.

Hope HuggingFace can re-enable the feature that at least allow viewing of  the first 20  examples of a particular dataset, or alternatively can extract 20 examples for each datasets and make those part of the Dataset Card Documentation.

Comments:
Hi @ChewKokWah,

We're working on it! In the meantime, you can still find the dataset explorer at the following URL: https://huggingface.co/datasets/viewer/

Glad to see that it still exist, this existing one is more than good enough for me, it is feature rich, simple to use and concise. 
Hope similar feature can be retain in the future version.
Title: AttributeError: module 'pyarrow' has no attribute 'PyExtensionType' during import 
Author: PalaashAgrawal (NONE)
State: closed
Created: 2021-01-26T04:18:35Z

I'm using Colab. And suddenly this morning, there is this error. Have a look below!

![screenshot-colab research google com-2021 01 26-08-15-36](https://user-images.githubusercontent.com/45964869/105799890-fdaf3b80-5fae-11eb-8f06-11b65cdccc30.png)

Comments:
Hi ! I'm not able to reproduce the issue. Can you try restarting your runtime ?

The PyExtensionType is available in pyarrow starting 0.17.1 iirc. If restarting your runtime doesn't fix this, can you try updating pyarrow ?
```
pip install pyarrow --upgrade
```

We should bump up the version test of pyarrow maybe no?

https://github.com/huggingface/datasets/blob/master/src/datasets/__init__.py#L60

Yes indeed.

Also it looks like Pyarrow 3.0.0 got released on pypi 10 hours ago. This might be related to the bug, I'll investigate
EDIT: looks like the 3.0.0 release doesn't have unexpected breaking changes for us, so I don't think the issue comes from that

Maybe colab moved to pyarrow 0.16 by default (instead of 0.14 before)?

Installing datasets installs pyarrow>=0.17.1 so in theory it doesn't matter which version of pyarrow colab has by default (which is currently pyarrow 0.14.1).

Also now the colab runtime refresh the pyarrow version automatically after the update from pip (previously you needed to restart your runtime).

I guess what happened is that Colab didn't refresh pyarrow for some reason, and the AttributeError was raised *before* the pyarrow version check from `datasets` at https://github.com/huggingface/datasets/blob/master/src/datasets/__init__.py#L60

Yes colab doesn’t reload preloaded library unless you restart the instance. Maybe we should move the check on top of the init 

Yes I'll do that :)

I updated the pyarrow version check in #1782

restarting worked for me
> ```

yes it worked for me
Title: Ignore definition line number of functions for caching
Author: lhoestq (MEMBER)
State: closed
Created: 2021-01-25T16:42:29Z

As noticed in #1718 , when a function used for processing with `map` is moved inside its python file, then the change of line number causes the caching mechanism to consider it as a different function. Therefore in this case, it recomputes everything.

This is because we were not ignoring the line number definition for such functions (even though we're doing it for lambda functions).

For example this code currently prints False:
```python
from datasets.fingerprint import Hasher

# define once
def foo(x):
    return x

h = Hasher.hash(foo)

# define a second time elsewhere
def foo(x):
    return x

print(h == Hasher.hash(foo))
```

I changed this by ignoring the line number for all functions.

Comments:
Title: Narrative QA Manual
Author: rsanjaykamath (CONTRIBUTOR)
State: closed
Created: 2021-01-25T15:22:31Z

Submitting the manual version of Narrative QA script which requires a manual download from the original repository

Comments:
@lhoestq sorry I opened a new pull request because of some issues with the previous code base. This pull request is originally from #1364

Excellent comments. Thanks for those valuable suggestions. I changed everything as you have pointed out :) 

I've copied the same template as NarrativeQA now. Please let me know if this is fine. 

> Awesome thank you !!
> This looks all good :)
> 
> Just before we merge, I was wondering if you knew why the number of examples in the train set went from 1102 to 32747 in your last commit ? I can't see why the changes in the code would cause such a big difference

Ok the change was the way I presented the data. 
In my previous code, I presented a story with a list of questions-answers related to the story per sample. So the total 1102 was the number of stories (not questions) in the train set. 

In the case of `NarrativeQA`, the code presented each sample data with one single question. So the story gets replicated as many times based on number of questions per story. I felt this was not really memory efficient so I had coded the way I did earlier. 

But since this would be inconsistent as you pointed out, I modified my code to suit the `NarrativeQA` approach. Hope it's clear now :) 

Ok I see ! that makes sense

Thanks for your time and helping me with all this :) Really appreciate the hardwork you guys do.
Title: GPT2 MNLI training using run_glue.py
Author: nlp-student (NONE)
State: closed
Created: 2021-01-25T10:53:52Z

Edit: I'm closing this because I actually meant to post this in `transformers `not `datasets`

Running this on Google Colab,

```
!python run_glue.py \
  --model_name_or_path gpt2 \
  --task_name mnli \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_gpu_train_batch_size 10 \
  --gradient_accumulation_steps 32\
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir models/gpt2/mnli/
```

I get the following error,

```
 "Asking to pad but the tokenizer does not have a padding token. "
ValueError: Asking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.
```

Do I need to modify the trainer to work with GPT2 ?

Comments: