t20062443/AT1ds huggingface dataset:
lhoestq (MEMBER) commented on 2020-10-26 09:44:09:
Does this HEAD request return 200 on your machine ?
```python
import requests                                                                                                                                                                                                         
requests.head("https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/cnn_dailymail/cnn_dailymail.py")
```

If it returns 200, could you try again to load the dataset ?
AI678 (NONE) commented on 2020-10-26 16:25:58:
Is it possible I download the dataset manually from google drive and use it for further test ? How can I do this ? I want to reproduce the model in this link https://huggingface.co/patrickvonplaten/bert2bert-cnn_dailymail-fp16. But I can't download the dataset through load_dataset method . I have tried many times  and the connection error always happens .
lhoestq (MEMBER) commented on 2020-10-26 16:37:14:
The head request should definitely work, not sure what's going on on your side.
If you find a way to make it work, please post it here since other users might encounter the same issue.

If you don't manage to fix it you can use `load_dataset` on google colab and then save it using `dataset.save_to_disk("path/to/dataset")`.
Then you can download the directory on your machine and do
```python
from datasets import load_from_disk
dataset = load_from_disk("path/to/local/dataset")
```
ksjae (NONE) commented on 2020-10-26 23:19:46:
Yes, I can confirm it also happens with ```num_proc=2```.
```
tokenizers               0.9.2
datasets                 1.1.2
multiprocess             0.70.10
```
```
Linux nipa2020-0629 4.4.0-178-generic #208-Ubuntu SMP Sun Apr 5 23:45:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
```
lhoestq (MEMBER) commented on 2020-10-27 14:04:29:
Yes currently you need an internet connection because the lib tries to check for the etag of the dataset script online to see if you don't have it locally already.

If we add a way to store the etag/hash locally after the first download, it would allow users to first download the dataset with an internet connection, and still have it working without an internet connection.

I'll let you know when we add this feature.
lhoestq (MEMBER) commented on 2020-10-27 15:47:05:
I can't reproduce on my side unfortunately with the same versions.

Do you have issues when doing multiprocessing with python ?
```python
from tqdm.auto import tqdm
from multiprocess import Pool, RLock

def process_data(shard):
    # implement

num_proc = 8
shards = [] # implement, this must be a list of size num_proc

with Pool(num_proc, initargs=(RLock(),), initializer=tqdm.set_lock) as pool:
    results = [pool.apply_async(process_data, shard=shard) for shard in shards]
    transformed_shards = [r.get() for r in results]
```
stefan-it (CONTRIBUTOR) commented on 2020-10-28 10:00:58:
Hey @gaguilar ,

I just found the ["char2subword" paper](https://arxiv.org/pdf/2010.12730.pdf) and I'm really interested in trying it out on own vocabs/datasets like for historical texts (I've already [trained some lms](https://github.com/stefan-it/europeana-bert) on newspaper articles with OCR errors).

Do you plan to release the code for your paper or is it possible to get the implementation 🤔 Many thanks :hugs:
lhoestq (MEMBER) commented on 2020-10-28 10:34:59:
Indeed you should use `features` in this case. 
```python
features = Features({'text': Value('string'), 'label': Value('float32')})
dataset = load_dataset('csv', data_files=['sst_test.csv'], features=features)
```
Note that because of an issue with the caching when you change the features (see #750 ) you still need to specify the `FORCE_REDOWNLOAD ` flag. I'm working on a fix for this one
thomwolf (MEMBER) commented on 2020-10-28 11:05:21:
Yes definitely we should give more flexibility to control the name of the splits outputted by `train_test_split`.

Related is the very interesting feedback from @bramvanroy on how we should improve this method: https://discuss.huggingface.co/t/how-to-split-main-dataset-into-train-dev-test-as-datasetdict/1090/5

And in particular that it should advantageously be able to split in 3 splits as well instead of just 2 like we copied from sklearn.
AmitMY (CONTRIBUTOR) commented on 2020-10-28 12:13:33:
When I run:
```
python datasets-cli dummy_data datasets/wlasl
```

I get:
```
Checking datasets/wlasl/wlasl.py for additional imports. 
Found main folder for dataset datasets/wlasl/wlasl.py at /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl 
Found specific version folder for dataset datasets/wlasl/wlasl.py at /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786  
Found script file from datasets/wlasl/wlasl.py to /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786/wlasl.py 
Found dataset infos file from datasets/wlasl/dataset_infos.json to /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786/dataset_infos.json 
Found metadata file for dataset datasets/wlasl/wlasl.py at /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786/wlasl.json 
Using custom data configuration default 
Loading Dataset Infos from /home/nlp/amit/.cache/huggingface/modules/datasets_modules/datasets/wlasl/f0cad785350d770804f20c471b0cff2d7e3c7b210b5c3a228c393abb95a04786
Creating dummy folder structure for datasets/wlasl/dummy/0.3.0...   
Dataset datasets with config None seems to already open files in the method `_split_generators(...)`. You might consider to instead only open files in the method `_generate_examples(...)` instead. If this is not possible the dummy data has to be created with less guidance. Make sure you create the file dummy_data. 
Traceback (most recent call last):      
File "datasets-cli", line 36, in         
service.run()                                                                                                                                                                                                                                                                                                                                                                                                            File "/home/nlp/amit/anaconda2/envs/meta-scholar/lib/python3.7/site-packages/datasets-1.1.2-py3.7.egg/datasets/commands/dummy_data.py", line 73, in run 
for split in generator_splits: 
UnboundLocalError: local variable 'generator_splits' referenced before assignment
```
lhoestq (MEMBER) commented on 2020-10-29 16:28:35:
This is an issue on the `mwparserfromhell` side. You could try to update `mwparserfromhell` and see if it fixes the issue. If it doesn't we'll have to create an issue on their repo for them to fix it.
But first let's see if the latest version of `mwparserfromhell` does the job.
lhoestq (MEMBER) commented on 2020-10-29 16:39:15:
Ok found the issue. This is because the batch size used by the writer is set to 10 000 elements by default so it would load your full dataset in memory (the writer has a buffer that flushes only after each batch). Moreover to write in Apache Arrow we have to use python objects so what's stored inside the ArrowWriter's buffer is actually python integers (32 bits).

Lowering the batch size to 10 should do the job.

I will add a flag to the DatasetBuilder class of dataset scripts, so that we can customize the batch size.
lhoestq (MEMBER) commented on 2020-10-30 14:34:41:
Hi ! This looks interesting, thanks for adding it :) 

For metrics there should only be two features fields: references and predictions.
Both of them can be defined as you want using nested structures if you need to.
Also I'm not sure what goes into references and what goes into predictions, could you give more details please ?
All the other computations parameters (model etc.) are fine though. Maybe explain a bit more what they're used for
rathoreanirudh (NONE) commented on 2020-10-30 15:32:58:
> Hi ! This looks interesting, thanks for adding it :)
> 
> For metrics there should only be two features fields: references and predictions.
> Both of them can be defined as you want using nested structures if you need to.
> Also I'm not sure what goes into references and what goes into predictions, could you give more details please ?
> All the other computations parameters (model etc.) are fine though. Maybe explain a bit more what they're used for

The `predictions` are the predicted labels by a model for a particular input. Do you mean making `prob_y_hat` - the probability of the prediction being the predicted label, `prob_y_hat_alpha` - the probability of the prediction being the predicted label when the input is reduced subject to alpha and the `null_difference` is the difference between the probability of the prediction being the predicted label in full information minus the probability in zero information a part of references? Also, I have added the description for other parameters in kwargs_description. I can expand it if that makes sense?
mkserge (CONTRIBUTOR) commented on 2020-10-30 15:38:42:
> What do the `sentence1` and `sentence2` correspond to exactly ?

`sentence1` is a question, and `sentence2` is a candidate answer sentence. The labels are [1, 2, 3, 4] defining a relation between the answer sentence and the question. For example, label 4 means that the answer sentence is inside the _long_answer_ passage AND that the _short_answer_ is within the answer sentence. All the other labels are the negatives with different characteristics. (the short_answer, long_answer terminology is borrowed from Google's NQ dataset)

Should I label them simply as `question` and `answer`? I was going more with what I saw in the examples/run_glue.py script, but I realize now there is no restriction around this.

> Also maybe you could use the `ClassLabel` feature type for the `label` field (see [snli](https://github.com/huggingface/datasets/blob/master/datasets/snli/snli.py) for example)

I am finding it difficult to assign names to each class, but perhaps it's possible. Here's the description of each class from the paper.

1. Sentences from the document that are in the long answer but do not contain the annotated short answers. It is possible that these sentences might contain the short answer.
2. Sentences from the document that are not in the long answer but contain the short answer string, that is, such occurrence is purely accidental.
3. Sentences from the document that are neither in the long answer nor contain the short answer.
4. Sentences from the document that are in the long answer and do contain the annotated short answers.

Any ideas?
lhoestq (MEMBER) commented on 2020-10-30 17:13:29:
Yes it's better to have explicit feature names. Maybe go with question/answer or question/sentence.
I read in the paper that 1,2 and 3 are considered negative and 4 positive.
We could have a binary classification label `label` (either positive of negative) and then two boolean fields `short_answser_in_sentence` and `sentence_in_long_answer`. What do you think ?
mkserge (CONTRIBUTOR) commented on 2020-10-30 19:55:58:
> Yes it's better to have explicit feature names. Maybe go with question/answer or question/sentence.
> I read in the paper that 1,2 and 3 are considered negative and 4 positive.
> We could have a binary classification label `label` (either positive of negative) and then two boolean fields `short_answser_in_sentence` and `sentence_in_long_answer`. What do you think ?

Ok, sounds good. I went with `sentence` to keep it consistent with `short_answer_in_sentence` and `sentence_in_long_answer`. 

I changed it to a ClassLabel with pos and neg classes and added the two above as features. Let me know if this is not what you had in mind.
patrickvonplaten (CONTRIBUTOR) commented on 2020-11-04 14:06:02:
Small poll  @thomwolf @yjernite @lhoestq @JetRunner @qiweizhen .

As stated in the XGLUE paper: https://arxiv.org/pdf/2004.01401.pdf , for each of the 11 down-stream tasks training data is only available in English, whereas development and test data is available in multiple different language *cf.* here: 

![Screenshot from 2020-11-04 15-02-17](https://user-images.githubusercontent.com/23423619/98120893-d7499a80-1eae-11eb-9d0b-57dfe5d4ee68.png)

So, I'd suggest to have exactly 11 "language-independent" configs: "ner", "pos", ... and give the sample in each dataset in the config a "language" label being one of "ar", "bg", .... => To me this makes more sense than making languaga specific config, *e.g.* "ner-de", ...especially because training data is only available in English. Do you guys agree?
yjernite (MEMBER) commented on 2020-11-04 14:50:21:
In this case we should have named splits, so config `ner` has splits `train`, `validation`, `test-en`, `test-ar`, `test-bg`, etc...

This is more in the spirit of the task afaiu, and will avoid making users do the filtering step themselves when testing different models or different configurations of the same model.
PaulLerner (CONTRIBUTOR) commented on 2020-11-05 10:34:59:
Just found out about https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md
(Not very clear in https://huggingface.co/datasets/kilt_tasks links to http://github.com/huggingface/datasets/datasets/kilt_tasks/README.md which is dead, closing the issue though :))
lhoestq (MEMBER) commented on 2020-11-09 09:18:06:
Hello !
Could you give more details ?

If you mean iter through one dataset then yes, `Dataset` object does implement the `__iter__` method so you can use 
```python
for example in dataset:
    # do something
```

If you want to iter through several datasets you can first concatenate them
```python
from datasets import concatenate_datasets

new_dataset = concatenate_datasets([dataset1, dataset2])
```
Let me know if this helps !
rabeehkarimimahabadi (NONE) commented on 2020-11-09 09:23:15:
Hi Huggingface/Datasets team,
I want to use the datasets inside Seq2SeqDataset here
https://github.com/huggingface/transformers/blob/master/examples/seq2seq/utils.py
and there I need to return back each line from the datasets and I am not
sure how to access each line and implement this?
It seems it also has get_item attribute? so I was not sure if this is
iterative dataset? or if this is non-iterable datasets?
thanks.



On Mon, Nov 9, 2020 at 10:18 AM Quentin Lhoest <notifications@github.com>
wrote:

> Hello !
> Could you give more details ?
>
> If you mean iter through one dataset then yes, Dataset object does
> implement the __iter__ method so you can use
>
> for example in dataset:
>     # do something
>
> If you want to iter through several datasets you can first concatenate them
>
> from datasets import concatenate_datasets
> new_dataset = concatenate_datasets([dataset1, dataset2])
>
> Let me know if this helps !
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/815#issuecomment-723881199>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHHYRLSSYW6NZN2HYDBTSO6XV5ANCNFSM4TPB7OWA>
> .
>
rabeehkarimimahabadi (NONE) commented on 2020-11-09 09:24:44:
could you tell me please if datasets also has __getitem__ any idea on how
to integrate it with Seq2SeqDataset is appreciated thanks

On Mon, Nov 9, 2020 at 10:22 AM Rabeeh Karimi Mahabadi <rabeeh@google.com>
wrote:

> Hi Huggingface/Datasets team,
> I want to use the datasets inside Seq2SeqDataset here
> https://github.com/huggingface/transformers/blob/master/examples/seq2seq/utils.py
> and there I need to return back each line from the datasets and I am not
> sure how to access each line and implement this?
> It seems it also has get_item attribute? so I was not sure if this is
> iterative dataset? or if this is non-iterable datasets?
> thanks.
>
>
>
> On Mon, Nov 9, 2020 at 10:18 AM Quentin Lhoest <notifications@github.com>
> wrote:
>
>> Hello !
>> Could you give more details ?
>>
>> If you mean iter through one dataset then yes, Dataset object does
>> implement the __iter__ method so you can use
>>
>> for example in dataset:
>>     # do something
>>
>> If you want to iter through several datasets you can first concatenate
>> them
>>
>> from datasets import concatenate_datasets
>> new_dataset = concatenate_datasets([dataset1, dataset2])
>>
>> Let me know if this helps !
>>
>> —
>> You are receiving this because you authored the thread.
>> Reply to this email directly, view it on GitHub
>> <https://github.com/huggingface/datasets/issues/815#issuecomment-723881199>,
>> or unsubscribe
>> <https://github.com/notifications/unsubscribe-auth/ARPXHHYRLSSYW6NZN2HYDBTSO6XV5ANCNFSM4TPB7OWA>
>> .
>>
>
lhoestq (MEMBER) commented on 2020-11-09 10:16:45:
`datasets.Dataset` objects implement indeed `__getitem__`. It returns a dictionary with one field per column.

We've not added the integration of the datasets library for the seq2seq utilities yet. The current seq2seq utilities are based on text files.

However as soon as you have a `datasets.Dataset` with columns "tgt_texts" (str), "src_texts" (str), and "id" (int) you should be able to implement your own Seq2SeqDataset class that wraps your dataset object. Does that make sense to you ?
yjernite (MEMBER) commented on 2020-11-09 13:36:18:
Yes: TriviaQA has a private test set for the leaderboard [here](https://competitions.codalab.org/competitions/17208)

For the KILT training and validation portions, you need to link the examples from the TriviaQA dataset as detailed here:
https://github.com/huggingface/datasets/blob/master/datasets/kilt_tasks/README.md
jc-hou (NONE) commented on 2020-11-09 14:58:13:
and also for 'blog_authorship_corpus'
https://huggingface.co/nlp/viewer/?dataset=blog_authorship_corpus
![image](https://user-images.githubusercontent.com/30210529/98557329-5c182800-22a4-11eb-9b01-5b910fb8fcd4.png)
rabeehkarimimahabadi (NONE) commented on 2020-11-09 17:37:04:
Hi
I am sorry for asking it multiple times but I am not getting the dataloader
type, could you confirm if the dataset library returns back an iterable
type dataloader or a mapping type one where one has access to __getitem__,
in the former case, one can iterate with __iter__, and how I can configure
it to return the data back as the iterative type? I am dealing with
large-scale datasets and  I do not want to bring all in memory
thanks for your help
Best regards
Rabeeh

On Mon, Nov 9, 2020 at 11:17 AM Quentin Lhoest <notifications@github.com>
wrote:

> datasets.Dataset objects implement indeed __getitem__. It returns a
> dictionary with one field per column.
>
> We've not added the integration of the datasets library for the seq2seq
> utilities yet. The current seq2seq utilities are based on text files.
>
> However as soon as you have a datasets.Dataset with columns "tgt_texts"
> (str), "src_texts" (str), and "id" (int) you should be able to implement
> your own Seq2SeqDataset class that wraps your dataset object. Does that
> make sense ?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/815#issuecomment-723915556>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHHYOC22EM7F666BZSOTSO66R3ANCNFSM4TPB7OWA>
> .
>
lhoestq (MEMBER) commented on 2020-11-09 17:42:59:
`datasets.Dataset` objects are both iterative and mapping types: it has both `__iter__` and `__getitem__`
For example you can do
```python
for example in dataset:
    # do something
```
or
```python
for i in range(len(dataset)):
    example = dataset[i]
    # do something
```
When you do that, one and only one example is loaded into memory at a time.
lhoestq (MEMBER) commented on 2020-11-09 18:11:25:
To show the issue:
```
python -c "from datasets.fingerprint import Hasher; a=[]; func = lambda : len(a); print(Hasher.hash(func))"
```
doesn't always return the same ouput since `globs` is a dictionary with "a" and "len" as keys but sometimes not in the same order
nikitakit (NONE) commented on 2020-11-09 21:34:00:
+1, the amount of logging is excessive.

Most of it indeed comes from `filelock.py`, though there are occasionally messages from other sources too. Below is an example (all of these messages were logged after I already called `datasets.logging.set_verbosity_error()`)

```
I1109 21:26:01.742688 139785006901056 filelock.py:318] Lock 139778216292192 released on /home/kitaev/.cache/huggingface/datasets/9ed4f2e133395826175a892c70611f68522c7bc61a35476e8b51a31afb76e4bf.e6f3e3f3e3875a07469d1cfd32e16e1d06b149616b11eef2d081c43d515b492d.py.lock
I1109 21:26:01.747898 139785006901056 filelock.py:274] Lock 139778216290176 acquired on /home/kitaev/.cache/huggingface/datasets/_home_kitaev_.cache_huggingface_datasets_glue_mnli_1.0.0_7c99657241149a24692c402a5c3f34d4c9f1df5ac2e4c3759fadea38f6cb29c4.lock
I1109 21:26:01.748258 139785006901056 filelock.py:318] Lock 139778216290176 released on /home/kitaev/.cache/huggingface/datasets/_home_kitaev_.cache_huggingface_datasets_glue_mnli_1.0.0_7c99657241149a24692c402a5c3f34d4c9f1df5ac2e4c3759fadea38f6cb29c4.lock
I1109 21:26:01.748412 139785006901056 filelock.py:274] Lock 139778215853024 acquired on /home/kitaev/.cache/huggingface/datasets/_home_kitaev_.cache_huggingface_datasets_glue_mnli_1.0.0_7c99657241149a24692c402a5c3f34d4c9f1df5ac2e4c3759fadea38f6cb29c4.lock
I1109 21:26:01.748497 139785006901056 filelock.py:318] Lock 139778215853024 released on /home/kitaev/.cache/huggingface/datasets/_home_kitaev_.cache_huggingface_datasets_glue_mnli_1.0.0_7c99657241149a24692c402a5c3f34d4c9f1df5ac2e4c3759fadea38f6cb29c4.lock
I1109 21:07:17.029001 140301730502464 filelock.py:274] Lock 140289479304360 acquired on /home/kitaev/.cache/huggingface/datasets/b16d3a04bf2cad1346896852bf120ba846ea1bebb1cd60255bb3a1a2bbcc3a67.ec871b06a00118091ec63eff0a641fddcb8d3c7cd52e855bbb2be28944df4b82.py.lock
I1109 21:07:17.029341 140301730502464 filelock.py:318] Lock 140289479304360 released on /home/kitaev/.cache/huggingface/datasets/b16d3a04bf2cad1346896852bf120ba846ea1bebb1cd60255bb3a1a2bbcc3a67.ec871b06a00118091ec63eff0a641fddcb8d3c7cd52e855bbb2be28944df4b82.py.lock
I1109 21:07:17.058964 140301730502464 filelock.py:274] Lock 140251889388120 acquired on /home/kitaev/.cache/huggingface/metrics/glue/mnli/default_experiment-1-0.arrow.lock
I1109 21:07:17.060933 140301730502464 filelock.py:318] Lock 140251889388120 released on /home/kitaev/.cache/huggingface/metrics/glue/mnli/default_experiment-1-0.arrow.lock
I1109 21:07:17.061067 140301730502464 filelock.py:274] Lock 140296072521488 acquired on /home/kitaev/.cache/huggingface/metrics/glue/mnli/default_experiment-1-0.arrow.lock
I1109 21:07:17.069736 140301730502464 metric.py:400] Removing /home/kitaev/.cache/huggingface/metrics/glue/mnli/default_experiment-1-0.arrow
I1109 21:07:17.069949 140301730502464 filelock.py:318] Lock 140296072521488 released on /home/kitaev/.cache/huggingface/metrics/glue/mnli/default_experiment-1-0.arrow.lock
```
ngdodd (CONTRIBUTOR) commented on 2020-11-09 22:03:49:
Done! PR [https://github.com/huggingface/datasets/pull/820](https://github.com/huggingface/datasets/pull/820)

Updated links and also regenerated the metadata and dummy data for v1.3 in order to pass verifications as described here: [https://huggingface.co/docs/datasets/share_dataset.html#adding-tests-and-metadata-to-the-dataset](https://huggingface.co/docs/datasets/share_dataset.html#adding-tests-and-metadata-to-the-dataset).
shexuan (NONE) commented on 2020-11-10 02:34:10:
> Hi !
> The url works on my side.
> 
> Is the url working in your navigator ?
> Are you connected to internet ? Does your network block access to `raw.githubusercontent.com` ?

I tried another server, it's working now. Thanks a lot.

And I'm curious about why download things from "github" when I load dataset from local files ?  Dose datasets work if my network crashed?