FQ u AD : F rench Question Answering Dataset (2020)

Abstract: Recent advances in the field of language modeling have improved state-of-the-art results on many Natural Language Processing tasks. Among them, Reading Comprehension has made significant progress over the past few years. However, most results are reported in English since labeled resources available in other languages, such as French, remain scarce. In the present work, we introduce the French Question Answering Dataset (FQuAD). FQuAD is a French Native Reading Comprehension dataset of questions and answers on a set of Wikipedia articles that consists of 25,000+ samples for the 1.0 version and 60,000+ samples for the 1.1 version. We train a baseline model which achieves an F1 score of 92.2 and an exact match ratio of 82.1 on the test set. In an effort to track the progress of French Question Answering models we propose a leaderboard and we have made the 1.0 version of our dataset freely available at https://illuin-tech.github.io/FQuAD-explorer/ .

## introduction current progress in language modeling has led to increasingly successful results on various natural language processing (nlp) tasks. this is namely the case of the reading comprehension task @xcite . however, reading comprehension datasets are costly and difficult to collect and are essentially native english datasets. indeed, datasets such as squad1.1 @xcite , squad2.0 @xcite , or coqa @xcite have fostered important and impressive progress for english question answering models over the past few years. the lack of native language annotated datasets apart from english is one of the main reasons why the development of language specific question answering models is lagging behind and this is namely the case for french. in order to fill the gap for the french language, we introduce a french reading comprehension dataset similar to squad1.1. the dataset consists of french native questions and answers samples annotated by a team of university students. the dataset comes in two versions. first fquad1.0, containing over 25,000+ samples. second, fquad1.1 containing over 60,000+ samples. the 35,000+ additional samples have been annotated with more demanding guidelines to strengthen complexity of the data and model to make the task harder. more specifically, the training, development, and test sets of fquad1.0 contain respectively 20,703, 3,188, and 2,189 samples. and the training, development, and test sets of fquad1.1 contain respectively 50,741, 5,668, and 5,594 samples. in order to evaluate the fquad dataset, we perform various experiments by fine-tuning bert based question answering models on both versions of the fquad dataset. the experiments involve the fine-tuning of french monolingual model camembert @xcite , and multilingual models mbert @xcite and xlm-roberta @xcite . we perform also two types of cross-lingual reading comprehension experiences. first, we evaluate the performance of the zero-shot cross-lingual transfer learning approach as stated in @xcite and @xcite on our newly obtained native french dataset. second, we evaluate the performance of the translation approach by finetuning models on the french translated version of squad1.1. the results of these two experiments help to better understand how the two cross-lingual approaches actually perform on a native dataset. the reading comprehension task (rc) @xcite attempts to solve the question answering (qa) problem by finding the text span in one or several documents or paragraphs that answers a given question @xcite . ## dataset collection the collection was conducted in two distinct steps: the first one resulted in fquad1.0 with 25,000+ question and answer pairs, and the second one resulted in fquad1.1 with 60,000+ question and answer pairs. apart from that, the collection follows 1 rajpurkar.github.io/squad-explorer the same standards and guidelines as squad1.1 @xcite . ## conclusion in the present work, we introduce the french question answering dataset. the contexts are collected from the set of high quality wikipedia articles. with the help of french college students, 60,000+ questions have been manually annotated. the fquad dataset is the result of two different annotation processes. first, fquad1.0 is collected to build a 25,000+ questions dataset. second, the dataset is enriched to reach 60,000+ questions resulting in fquad1.1. the development and test sets have both been enriched with additional answers for the evaluation process. we find that the human performances for fquad1.1 on the test and development sets reach respectively a f1-score of 91.2% and an exact match of 75.9%, and a f1-score of 92.1% and an exact match of 78.3%. furthermore, we find that the human performances on fquad1.1 reach comparable scores to squad1.1. various experiments were carried out to evaluate the performances of monolingual and multilingual language models. our best model, camembert large , achieves a f1-score and an exact match of respectively 92.2% and 82.1%, surpassing the established human performance in terms of f1-score and exact match. the experiments show that multilingual models reach promising results but monolingual models of comparable sizes perform better. the fquad1.0 training and fquad1.1 development sets are made publicly available in order to foster research in the french nlp area. we believe our dataset can boost french research in other nlp fields such as nlu, information retrieval or open domain question answering to cite a few. the extension of the dataset to adversarial questions similarly to squad2.0 is left for future works.