Title: [Feature request] Add FLUE dataset 
Author: lbourdois (NONE)
State: closed
Created: 2020-05-30T08:52:15Z
Labels: dataset request

Hi,

I think it would be interesting to add the FLUE dataset for francophones or anyone wishing to work on French.

In other requests, I read that you are already working on some datasets, and I was wondering if FLUE was planned.

If it is not the case, I can provide each of the cleaned FLUE datasets (in the form of a directly exploitable dataset rather than in the original xml formats which require additional processing, with the French part for cases where the dataset is based on a multilingual dataframe, etc.).

Comments:
Hi @lbourdois, yes please share it with us

@mariamabarham 
I put all the datasets on this drive: https://1drv.ms/u/s!Ao2Rcpiny7RFinDypq7w-LbXcsx9?e=iVsEDh


Some information : 
• For FLUE, the quote used is

> @misc{le2019flaubert,
>     title={FlauBERT: Unsupervised Language Model Pre-training for French},
>     author={Hang Le and Loïc Vial and Jibril Frej and Vincent Segonne and Maximin Coavoux and Benjamin Lecouteux and Alexandre Allauzen and Benoît Crabbé and Laurent Besacier and Didier Schwab},
>     year={2019},
>     eprint={1912.05372},
>     archivePrefix={arXiv},
>     primaryClass={cs.CL}
> }

• The Github repo of FLUE is avaible here : https://github.com/getalp/Flaubert/tree/master/flue



Information related to the different tasks of FLUE : 

**1. Classification**
Three dataframes are available: 
- Book
- DVD
- Music
For each of these dataframes is available a set of training and test data, and a third one containing unlabelled data.

Citation  : 
>@dataset{prettenhofer_peter_2010_3251672,
  author       = {Prettenhofer, Peter and
                  Stein, Benno},
  title        = {{Webis Cross-Lingual Sentiment Dataset 2010 (Webis- 
                   CLS-10)}},
  month        = jul,
  year         = 2010,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.3251672},
  url          = {https://doi.org/10.5281/zenodo.3251672}
}


**2. Paraphrasing** 
French part of the PAWS-X dataset (https://github.com/google-research-datasets/paws).
Three dataframes are available: 
- train
- dev
- test 

Citation : 
> @InProceedings{pawsx2019emnlp,
>   title = {{PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification}},
>   author = {Yang, Yinfei and Zhang, Yuan and Tar, Chris and Baldridge, Jason},
>   booktitle = {Proc. of EMNLP},
>   year = {2019}
> }



**3. Natural Language Inference**
French part of the XNLI dataset (https://github.com/facebookresearch/XNLI).
Three dataframes are available: 
- train
- dev
- test 

For the dev and test datasets, extra columns compared to the train dataset were available so I left them in the dataframe (I didn't know if these columns could be useful for other tasks or not). 
In the context of the FLUE benchmark, only the columns gold_label, sentence1 and sentence2 are useful.


Citation : 

> @InProceedings{conneau2018xnli,
>   author = "Conneau, Alexis
>         and Rinott, Ruty
>         and Lample, Guillaume
>         and Williams, Adina
>         and Bowman, Samuel R.
>         and Schwenk, Holger
>         and Stoyanov, Veselin",
>   title = "XNLI: Evaluating Cross-lingual Sentence Representations",
>   booktitle = "Proceedings of the 2018 Conference on Empirical Methods
>                in Natural Language Processing",
>   year = "2018",
>   publisher = "Association for Computational Linguistics",
>   location = "Brussels, Belgium",


**4. Parsing**
The dataset used by the FLUE authors for this task is not freely available.
Users of your library will therefore not be able to access it.
Nevertheless, I think maybe it is useful to add a link to the site where to request this dataframe: http://ftb.linguist.univ-paris-diderot.fr/telecharger.php?langue=en 
(personally it was sent to me less than 48 hours after I requested it).


**5. Word Sense Disambiguation Tasks**
5.1 Verb Sense Disambiguation

Two dataframes are available: train and test
For both dataframes, 4 columns are available: document, sentence, lemma and word.
I created the document column thinking that there were several documents in the dataset but afterwards it turns out that there were not: several sentences but only one document. It's up to you to keep it or not when importing these two dataframes.

The sentence column is used to determine to which sentence the word in the word column belongs. It is in the form of a dictionary {'id': 'd000.s001', 'idx': '1'}. I thought for a while to keep only the idx because the id doesn't matter any more information. Nevertheless for the test dataset, the dictionary has an extra value indicating the source of the sentence. I don't know if it's useful or not, that's why I left the dictionary just in case. The user is free to do what he wants with it.

Citation : 

> Segonne, V., Candito, M., and Crabb ́e, B. (2019). Usingwiktionary as a resource for wsd: the case of frenchverbs. InProceedings of the 13th International Confer-ence on Computational Semantics-Long Papers, pages259–270

5.2 Noun Sense Disambiguation
Two dataframes are available:  2 train and  1 test

I confess I didn't fully understand the procedure for this task.

Citation : 

> @dataset{loic_vial_2019_3549806,
>   author       = {Loïc Vial},
>   title        = {{French Word Sense Disambiguation with Princeton 
>                    WordNet Identifiers}},
>   month        = nov,
>   year         = 2019,
>   publisher    = {Zenodo},
>   version      = {1.0},
>   doi          = {10.5281/zenodo.3549806},
>   url          = {https://doi.org/10.5281/zenodo.3549806}
> }

Finally, additional information about FLUE is available in the FlauBERT publication : 
https://arxiv.org/abs/1912.05372 (p. 4).


Hoping to have provided you with everything you need to add this benchmark  :) 


https://github.com/huggingface/datasets/pull/943