t20062443/AT1ds huggingface dataset:
yjernite (MEMBER) commented on 2020-06-02 14:09:38:
You're right, there's a pretty interesting trade-off here between robustness and sensitivity :) The flip side of your argument is that we also still want the metric to be sensitive to model mistakes. How we think about number normalization for example has evolved a fair bit since the Perl script was written: at the time, ROUGE was used mostly to evaluate short-medium text summarization systems, where there were only a few numbers in the input and it was assumed that the most popular methods in use at the time would get those right. However, as your example showcases, that assumption does not hold any more, and we do want to be able to penalize a model that generates a wrong numerical value.

Also, we think that abstracting away tokenization differences is the role of the model/tokenizer: if you use the 🤗Tokenizers library for example, it will handle that for you ;)

Finally, there is a lot of active research on developing model-powered metrics that are both more sensitive and more robust than ROUGE. Check out for example BERTscore, which is implemented in this library!
astariul (NONE) commented on 2020-06-02 23:44:37:
`from_generator` is not working on TPU, I met the following error :

```
File "/usr/local/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/usr/.venv/bart/lib/python3.6/site-packages/tensorflow_core/python/eager/context.py", line 1900, in execution_mode
    executor_new.wait()
  File "/home/usr/.venv/bart/lib/python3.6/site-packages/tensorflow_core/python/eager/executor.py", line 67, in wait
    pywrap_tensorflow.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.NotFoundError: No registered 'PyFunc' OpKernel for 'CPU' devices compatible with node {{node PyFunc}}
        .  Registered:  <no registered kernels>

         [[PyFunc]]
```

---

@lhoestq It seems you merged some changes that allow lazy-loading. **Can you give an example of how to use ?** Maybe the Colab notebook should be updated with this method as well.
patrickvonplaten (CONTRIBUTOR) commented on 2020-06-03 08:26:19:
I like this feature! I think the first question we should decide on is how to convert all datasets into the same format. In T5, the authors decided to format every dataset into a text-to-text format. If the dataset had "multiple" inputs like MNLI, the inputs were concatenated. So in MNLI the input:

> - **Hypothesis**: The St. Louis Cardinals have always won.
> 
> - **Premise**: yeah well losing is i mean i’m i’m originally from Saint Louis and Saint Louis Cardinals when they were there were uh a mostly a losing team but 

was flattened to a single input:

> mnli hypothesis: The St. Louis Cardinals have always won. premise:
> yeah well losing is i mean i’m i’m originally from Saint Louis and Saint Louis Cardinals
> when they were there were uh a mostly a losing team but.

This flattening is actually a very simple operation in `nlp` already. You would just need to do the following:

```python 
def flatten_inputs(example):
    return {"input": "mnli hypothesis: " + example['hypothesis'] + " premise: " + example['premise']}

t5_ready_mnli_ds = mnli_ds.map(flatten_inputs, remove_columns=[<all columns except output>])
```

So I guess converting the datasets into the same format can be left to the user for now. 
Then the question is how we can merge the datasets. I would probably be in favor of a simple 

```python 
dataset.add()
```

function that checks if the dataset is of the same format and if yes merges the two datasets. Finally, how should the sampling be implemented? **Examples-proportional mixing** corresponds to just merging the datasets and shuffling. For the other two sampling approaches we would need some higher-level features, maybe even a `dataset.sample()` function for merged datasets. 

What are your thoughts on this @thomwolf @lhoestq  @ghomasHudson @enzoampil ?
astariul (NONE) commented on 2020-06-03 08:56:43:
I'm using TF2.2

Here is my code :
```
import nlp
from transformers import BartTokenizer

tokenizer = BartTokenizer.from_pretrained('bart-large')

def encode(sample):
    article_inputs = tokenizer.encode_plus(sample["article"], max_length=tokenizer.model_max_length, pad_to_max_length=True)
    summary_inputs = tokenizer.encode_plus(sample["highlights"], max_length=tokenizer.model_max_length, pad_to_max_length=True)

    article_inputs.update({"lm_labels": summary_inputs['input_ids']})
    return article_inputs

cnn_dm = nlp.load_dataset('cnn_dailymail', '3.0.0', split='test')
cnn_dm = cnn_dm.map(encode)

def gen():
    for sample in cnn_dm:
        s = {}
        s['input_ids'] = sample['input_ids']
        s['attention_mask'] = sample['attention_mask']
        s['lm_labels'] = sample['lm_labels']
        yield s

dataset = tf.data.Dataset.from_generator(gen, output_types={k: tf.int32 for k in ['input_ids', 'attention_mask', 'lm_labels']}, output_shapes={k: tf.TensorShape([tokenizer.model_max_length]) for k in ['input_ids', 'attention_mask', 'lm_labels']}
```
ghomasHudson (CONTRIBUTOR) commented on 2020-06-03 10:01:08:
I agree that we should leave the flattening of the dataset to the user for now. Especially because although the T5 framing seems obvious, there are slight variations on how the T5 authors do it in comparison to other approaches such as gpt-3 and decaNLP.

In terms of sampling, Examples-proportional mixing does seem the simplest to implement so would probably be a good starting point.

Temperature-scaled mixing would probably most useful, offering flexibility as it can simulate the other 2 methods by setting the temperature parameter. There is a [relevant part of the T5 repo](https://github.com/google-research/text-to-text-transfer-transformer/blob/03c94165a7d52e4f7230e5944a0541d8c5710788/t5/data/utils.py#L889-L1118) which should help with implementation.

According to the T5 authors, equal-mixing performs worst. Among the other two methods, tuning the K value (the artificial dataset size limit) has a large impact.
patrickvonplaten (CONTRIBUTOR) commented on 2020-06-03 10:49:13:
This trick from @thomwolf seems to be the most reliable solution to fix this colab notebook issue:

```python
# install nlp
!pip install -qq nlp==0.2.0

# Make sure that we have a recent version of pyarrow in the session before we continue - otherwise reboot Colab to activate it
import pyarrow
if int(pyarrow.__version__.split('.')[1]) < 16:
    import os
    os.kill(os.getpid(), 9)
```
lhoestq (MEMBER) commented on 2020-06-03 12:48:33:

This was actually the right name. `dataset_infos.json` is used to have the infos of all the dataset configurations.

On the other hand `dataset_info.json` (without 's') is a cache file with the info of one specific configuration.

To fix #228, we probably just have to clear and reload the nlp-viewer cache.
enzoampil (NONE) commented on 2020-06-03 23:23:37:
I agree with going with temperature-scaled mixing for its flexibility!

For the function that combines the datasets, I also find `dataset.add()` okay while also considering that users may want it to be easy to combine a list of say 10 data sources in one go.

`dataset.sample()` should also be good. By the looks of it, we're planning to have as main parameters: `temperature`, and `K`.

On converting the datasets to the same format, I agree that we can leave these to the users for now. But, I do imagine it'd be an awesome feature for the future to have this automatically handled, based on a chosen *approach* to formatting :smile: 

E.g. T5, GPT-3, decaNLP, original raw formatting, or a contributed way of formatting in text-to-text.
parthplc (NONE) commented on 2020-06-04 01:57:06:
I am getting this error when running this command
```
val = nlp.load_dataset('squad', split="validation")
```

FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/huggingface/datasets/squad/plain_text/1.0.0/dataset_info.json'

Can anybody help?
patrickvonplaten (CONTRIBUTOR) commented on 2020-06-04 20:22:39:
I think it would be nicer to not create a new folder `datasets_experimental` , but just put your datasets also into the folder `datasets` for the following reasons:

- From my point of view, the datasets are not very different from the other datasets (assuming that we soon have C4, and the beam datasets) so I don't see why we require a new dataset folder

- I'm not a big fan of adding a boolean flag to the `load_dataset()` function that basically switches between folder names on S3. The user has to know whether a dataset script is experimental or not. User installing nlp with pip won't see that there are folders called `datasets` and `datasets_experimental`

- If we do this just to bypass the test, I think a good solution could be: For all tests that are too complicated to be currently tested with the testing framework, we can add a class variable called `do_test = False` to the dataset builder class and a default `do_test = True` to the abstract dataset class and skip all tests that have that variable in the dataset test framework similar to what is done to beam datasets: https://github.com/huggingface/nlp/blob/2e0a8639a79b1abc848cff5c669094d40bba0f63/tests/test_dataset_common.py#L79 
We can also print a warning for all dataset tests having `do_test = False`. This variable would only concern testing and we would not have a problem removing it at a later stage IMO.

- This way the datascripts are backward compatible and can be used with earlier versions of `nlp` (not that this matters too much atm) 

What is your opinion on this @lhoestq @thomwolf ?
lhoestq (MEMBER) commented on 2020-06-04 21:05:45:
Very cool to have add those datasets :)
I understand that making the dummy data for this case is not fun. I'm sure we'll be able to add them soon. For now it's still interesting to have them in the library, even if we can't test all the code with dummy data.

I like the idea of the `do_tests=False` class variable. 
However it would be cool to test at least that we can load the module and instantiate the builder (only ignore the dummy data test for now). In that case a better name could be `test_dummy_data=False` or something like that.

If we want to be picky we can also add a warning in `_download_and_prepare` to tell the user that datasets with `test_dummy_data=False` are still experimental.
patrickvonplaten (CONTRIBUTOR) commented on 2020-06-04 21:14:04:
> Very cool to have add those datasets :)
> I understand that making the dummy data for this case is not fun. I'm sure we'll be able to add them soon. For now it's still interesting to have them in the library, even if we can't test all the code with dummy data.
> 
> I like the idea of the `do_tests=False` class variable.
> However it would be cool to test at least that we can load the module and instantiate the builder (only ignore the dummy data test for now). In that case a better name could be `test_dummy_data=False` or something like that.
> 
> If we want to be picky we can also add a warning in `_download_and_prepare` to tell the user that datasets with `test_dummy_data=False` are still experimental.

`test_dummy_data=False` sounds good to me!
richarddwang (CONTRIBUTOR) commented on 2020-06-05 06:05:06:
Hi @MaveriQ , although what I am doing is to commit a new dataset, but I think looking at imdb script might help.
You may want to use `dl_manager.download_custom`, give it a url(arbitrary string), a custom_download(arbitrary function) and return a path, and finally use _get sample to fetch a sample.
mariamabarham (CONTRIBUTOR) commented on 2020-06-05 08:07:26:
Hi @aleSuglia, thanks for this great PR. Indeed you can have both datasets in one file. You need to add a config class which will allows you to specify the different subdataset names and then  you will be able to load them as follow.
nlp.load_dataset("compguesswhat", "compguesswhat-gameplay") 
nlp.load_dataset("compguesswhat", "compguesswhat-zs-gameplay").

Maybe you can refer to this file https://github.com/huggingface/nlp/blob/master/datasets/discofuse/discofuse.py
patrickvonplaten (CONTRIBUTOR) commented on 2020-06-05 09:06:40:
I think using `sorted(glob.glob())` would actually solve this problem. Can you think of other reasons why dataset loading might not be deterministic? @mariamabarham @yjernite @lhoestq @thomwolf . 

I can do a sweep through the dataset scripts and fix the glob.glob() if you guys are ok with it
aleSuglia (CONTRIBUTOR) commented on 2020-06-05 20:46:44:
Not sure whether it's the most appropriate place but I'll ask another design question. For Vision+Language dataset, is very common to have visual features associated with each example. At the moment, for instance, I'm only integrating the image identifier so that people can later on lookup the image features during training. Do you recommend this approach or do you think it should be done in a different way?

Thank you for your answer!
aleSuglia (CONTRIBUTOR) commented on 2020-06-05 21:31:55:
@thomwolf As far as I know, people using Language+Vision tasks they typically have their reference dataset (either in JSON or JSONL format) and for each example in it they have an identifier that specifies the reference image. Currently, images are represented by either pooling-based visual features (average pooling of ResNet or VGGNet features, see [DeVries et.al, 2017](https://arxiv.org/abs/1611.08481), [Shekhar et.al, 2019](https://www.aclweb.org/anthology/N19-1265.pdf)) where you have a single vector for every image. Another option is to use a set of feature maps for every image extracted from a specific layer of a CNN (see [Xu et.al, 2015](https://arxiv.org/abs/1502.03044)). A more common and recent option, especially with large-scale multi-modal transformers [Li et. al, 2019](https://arxiv.org/abs/1908.03557), is to use FastRCNN features. 

For all these types of features, people use either HD5F or NumPy compressed representations. In my personal projects, I've ditched altogether HD5F because it doesn't have out-of-the-box support for multi-processing (unless you have an ad-hoc installation of it). I've been successfully using a NumPy compressed file for each image so that I can store any sort of information in it (see [numpy.savez](https://numpy.org/doc/stable/reference/generated/numpy.savez.html)). However, I believe that Apache Arrow would be a really good fit for this type of features. 

Looking forward to hearing your thoughts about it!
richarddwang (CONTRIBUTOR) commented on 2020-06-06 02:57:04:
Hi, I rebase my local copy to `fix-empty-cache-dir`, and try to run again `python nlp-cli test datasets/bookcorpus --save_infos --all_configs`.

I got this,  
```
Traceback (most recent call last):
  File "nlp-cli", line 10, in <module>
    from nlp.commands.run_beam import RunBeamCommand
  File "/home/yisiang/nlp/src/nlp/commands/run_beam.py", line 6, in <module>
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
```
And after I installed it. I got this
```
File "/home/yisiang/nlp/src/nlp/datasets/bookcorpus/aea0bd5142d26df645a8fce23d6110bb95ecb81772bb2a1f29012e329191962c/bookcorpus.py", line 88, in _split_generators
    downloaded_path_or_paths = dl_manager.download_custom(_GDRIVE_FILE_ID, download_file_from_google_drive)
  File "/home/yisiang/nlp/src/nlp/utils/download_manager.py", line 128, in download_custom
    downloaded_path_or_paths = map_nested(url_to_downloaded_path, url_or_urls)
  File "/home/yisiang/nlp/src/nlp/utils/py_utils.py", line 172, in map_nested
    return function(data_struct)
  File "/home/yisiang/nlp/src/nlp/utils/download_manager.py", line 126, in url_to_downloaded_path
    return os.path.join(self._download_config.cache_dir, hash_url_to_filename(url))
  File "/home/yisiang/miniconda3/envs/nlppr/lib/python3.7/posixpath.py", line 80, in join
    a = os.fspath(a)
```
The problem is when I print `self._download_config.cache_dir` using pdb, it is `None`.

Did I miss something ?  Or can you provide a workaround first so I can keep testing my script ?
thomwolf (MEMBER) commented on 2020-06-06 16:39:25:
Yes, this is because the test sets for glue are hidden so the labels are
not publicly available. You can read the glue paper for more details.

On Sat, 6 Jun 2020 at 18:16, Jack Morris <notifications@github.com> wrote:

> this also happened to me with nlp.load_datasets('glue', 'mnli')
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/nlp/issues/245#issuecomment-640083980>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABYDIHMVQD2EDX2HTZUXG5DRVJTWRANCNFSM4NVG3AKQ>
> .
>
thomwolf (MEMBER) commented on 2020-06-06 16:42:11:
Everything is already cached by default in 🤗nlp (in particular dataset
loading and all the “map()” operations) so I don’t think you need to do any
specific caching in streamlit.

Tell us if you feel like it’s not the case.

On Sat, 6 Jun 2020 at 13:02, Fabrizio Milo <notifications@github.com> wrote:

> For example if I want to use streamlit with a nlp dataset:
>
> @st.cache
> def load_data():
>     return nlp.load_dataset('squad')
>
> This code raises the error "uncachable object"
>
> Right now I just fixed with a constant for my specific case:
>
>     @st.cache(hash_funcs={pyarrow.lib.Buffer: lambda b: 0})
>
> But I was curious to know what is the best way in general
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/nlp/issues/246>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABYDIHKAKO7CWGX2QY55UXLRVIO3ZANCNFSM4NV333RQ>
> .
>
thomwolf (MEMBER) commented on 2020-06-06 19:48:02:
That's great!

I think it would be nice to test "deterministic-ness" of datasets in CI if we can do it (should be left for future PR of course)

Here is a possibility (maybe there are other ways to do it): given that we may soon have efficient and large-scale hashing (cf our discussion on versioning/tracability), we could incorporate a hash of the final Arrow Dataset to the `dataset.json` file and have a test on it as well as CI on a diversity of platform to test the hash (Win/Mac/Linux + various python/env).
What do you think @lhoestq @patrickvonplaten?
patrickvonplaten (CONTRIBUTOR) commented on 2020-06-07 11:22:44:
> That's great!
> 
> I think it would be nice to test "deterministic-ness" of datasets in CI if we can do it (should be left for future PR of course)
> 
> Here is a possibility (maybe there are other ways to do it): given that we may soon have efficient and large-scale hashing (cf our discussion on versioning/tracability), we could incorporate a hash of the final Arrow Dataset to the `dataset.json` file and have a test on it as well as CI on a diversity of platform to test the hash (Win/Mac/Linux + various python/env).
> What do you think @lhoestq @patrickvonplaten?

I think that's a great idea! The test should be a `RUN_SLOW` test, since it takes a considerable amount of time to download the dataset and generate the examples, but I think we should add some kind of hash test for each dataset.
lhoestq (MEMBER) commented on 2020-06-08 08:49:57:
To load a dataset you need to have a script that defines the format of the examples, the splits and the way to generate examples. As your dataset has the same format of squad, you can just copy the squad script (see the [datasets](https://github.com/huggingface/nlp/tree/master/datasets) forlder) and just replace the url to load the data to your local or remote path.

Then what you can do is `load_dataset(<path/to/your/script>)`
lhoestq (MEMBER) commented on 2020-06-08 09:02:09:
Hello ! Thanks for noticing this bug, let me fix that.

Also for information, as specified in the changelog of the latest release, C4 currently needs to have a runtime for apache beam to work on. Apache beam is used to process this very big dataset and it can work on dataflow, spark, flink, apex, etc. You can find more info on beam datasets [here](https://github.com/huggingface/nlp/blob/master/docs/beam_dataset.md).

Our goal in the future is to make available an already-processed version of C4 (as we do for wikipedia for example) so that users without apache beam runtimes can load it.
ghomasHudson (CONTRIBUTOR) commented on 2020-06-08 11:09:00:
I guess we're looking at the end user writing something like:
``` python
ds = nlp.load_dataset('multitask-t5',datasets=["squad","cnn_dm",...], k=1000, t=2.0)
```
Using the t5 method of combining here (or this could be a function passed in as an arg) 

Passing kwargs to each 'sub-dataset' might become tricky.
lhoestq (MEMBER) commented on 2020-06-08 12:40:25:
Thanks for adding this one !

About the three points you mentioned:
1. I think the `toronto_books_corpus` branch can be removed @mariamabarham ? 
2. You can use the download manager to download from google drive. For you case you can just do something like 
```python
URL = "https://drive.google.com/uc?export=download&id=16KCjV9z_FHm8LgZw05RSuk4EsAWPOP_z"
...
arch_path = dl_manager.download_and_extract(URL)
```
Also this is is an unofficial host of the dataset, we should probably host it ourselves if we can.
3. Not sure about the copyright here, but I maybe @thomwolf has better insights about it.
TheophileBlard (CONTRIBUTOR) commented on 2020-06-08 19:16:08:
Hi @mariamabarham 

FLUE can indeed become a very interesting benchmark for french NLP !
Unfortunately, it seems that we've both been working on adding it to the repo...
I was going to open a pull request before I came across yours.
I didn't want to open a duplicate, that's why I'm commenting here (I hope it's not rude).

When I look at your code there is one issue that jump out at me: for both `vsd` and `nsd`, the labels are missing. I believe this is more a data issue, as they were not kept in the cleaned dataframes of #223. I think the *word sense disambiguation* task was a bit misunderstood. 

Maybe you should directly use the data provided by FLUE for these ?
lbourdois (NONE) commented on 2020-06-09 08:05:22:
Hi,

As @TheophileBlard pointed out, the labels for the vsd and nsd stains are missing.

For the wsd, it is my mistake, I added the files containing the labels on the drive.
There is still the join to do between the files that I didn't have time to do. It can be done after importing the two files, however if you wish to have a single dataframe already containing all the information, I could do it but only when I have free time because I have a lot of work at the moment at INSERM with the covid.

For the nsd, I've downloaded the files at https://zenodo.org/record/3549806, and if you do the same you'll see that they don't contain any labels.
In the files, you can see that some words have a WN code. I don't know what it corresponds to. On the FLUE github, they say to use the disambiguate tool (https://github.com/getalp/disambiguate) but I don't understand what he's doing.

@mariamabarham  for the parsing datasets, I have them in my possession. What it does that I haven't shared them is that they are licensed and you have to make a request to their creators. They give them away very easily for research purposes. For another use, you have to ask a commercial licence. All this means that if the data is freely available on your librairy, their licence and their application form are no longer of interest, which is why I did not add them.
Afterwards, maybe the authors will change their policies and decide to make the data freely available through your librairy