On the Usability of Transformers-based Models for a F rench Question-Answering Task (2021)

Abstract: For many tasks, state-of-the-art results have been achieved with Transformer-based architectures, resulting in a paradigmatic shift in practices from the use of task-specific architectures to the fine-tuning of pre-trained language models. The ongoing trend consists in training models with an ever-increasing amount of data and parameters, which requires considerable resources. It leads to a strong search to improve resource efficiency based on algorithmic and hardware improvements evaluated only for English. This raises questions about their usability when applied to small-scale learning problems, for which a limited amount of training data is available, especially for under-resourced languages tasks. The lack of appropriately sized corpora is a hindrance to applying data-driven and transfer learning-based approaches with strong instability cases. In this paper, we establish a state-of-the-art of the efforts dedicated to the usability of Transformer-based models and propose to evaluate these improvements on the question-answering performances of French language which have few resources. We address the instability relating to data scarcity by investigating various training strategies with data augmentation, hyperparameters optimization and cross-lingual transfer. We also introduce a new compact model for French FrALBERT which proves to be competitive in low-resource settings.

## introduction recent advances in the field of natural language processing (nlp) have been made with the development of transfer learning and the availability of pre-trained language models based on transformer architectures @xcite , such as bert @xcite . as they provide contextualized semantic representation they contribute both to advance the state-of-the-art on several nlp tasks and also to evolve training practices through the use of fine-tuning. the trend of recent years consists in training large pre-trained language models on ever larger corpora, with an ever-increasing amount of parameters, which requires considerable computational resources that only a few companies and institutions can afford. for example, the base model of bert with 110 million parameters was pre-trained on 16 gigabytes (gb) of text, while the gpt-3 model @xcite was pre-trained on 45 terabytes (tb) of text and has 175 billion parameters. in fact, deploying ever-larger models raises questions and concerns about the increasing magnitude of the temporal, financial, and environmental cost of training and usability @xcite . typically, due to their resource requirements, these models are trained and deployed for industrial operations on remote servers. this leads to a high use of over-the-air communications, which are particularly resourceintensive @xcite . in particular, some nlp applications (speech recognition, speech to text, etc.) have some known problems related to network latency, transmission path difficulties, or privacy concerns. to reduce the impact of these communications, there is a solution that is to allow these models to run directly on peripheral or mobile devices, that is, in environments with limited resources that require lightweight, responsive models and energy efficiency. reducing the size of the models is therefore one of the increasingly favoured avenues, especially for the reduction of memory resources and computation time involved in training and use. to meet these constraints, compact models represent one of the most promising solutions. as far as we know, they have only been evaluated on the comprehension tasks covered by glue @xcite and the question-answering task with the squad corpus @xcite with abundant data, in english. the improvements resulting from the algorithmic optimizations of the models, although significant, raise questions about their effectiveness on lower-scale learning problems on poorly endowed languages. the works of @xcite and @xcite have furthermore shown degraded performance in these conditions. these two reflections are at the origin of a double question which our contribution attempts to answer. on the one hand, what is the behavior of a transformer-based model in the context of a question-answering task in french, a task that is poorly endowed in this language? on the other hand, what are the impacts of algorithmic improvements of these same models in this context? to answer these questions, we first establish in section 2 a state-of-the-art that is meant to be broad enough to have a shallow overview depicting the ins and outs and issues around the usability of transformer-based models whose breadcrumb trail is the issue of resources. then, we present in the section 3 the recent progress of the questionanswering task, through the use of these latest models. in sections 4 and 5 we introduce our model and present our experiments on the usability of transformers models in a question-answering task for french, on fquad (d'hoffschmidt et al., 2020) and piaf @xcite corpora. we propose to address the instability relating to data scarcity by investigating various training strategies with data augmentation, hyperparameters optimization and cross-lingual transfer. finally, we present a new compact model for french based on albert @xcite foot_0 , and compare it to existing monolingual and multilingual models, large and compact, under constrained conditions (notably on learning data). ## usability of transformers in this section we present the ins and out of the transformer models to understand how the approaches meet the need for better usability. ## the question-answering task question-answering (qa) based on machine reading comprehension corresponds to the task of extracting an answer given a question and a context document such as from a news or wikipedia article. ## fralbert and transformer models considered as mentioned in the previous section there is no compact model for french. we therefore decided to pre-train a new version of albert from scratch we called fralbert, thus overcoming some of the discussed limitations. albert is based on parameter sharing/reduction techniques that allows to reduce the computational complexity and speed up training and inference phases. compared to previous compact models such as distilbert @xcite , q-bert @xcite or ternarybert @xcite , albert is to the date the smallest pre-trained models with 12 million parameters @xmath0. fralbert is pre-trained on the french version of the wikipedia encyclopedia of 04/05/2021, i.e. 4 gb of text and 17 million (m) sentences. beyond concerns about the rights to use data from common our experiments are also based on the large monolingual french model camembert @xcite as well as on the two large multilingual models: xlm-r @xcite and mbert @xcite , both pre-trained from massive corpora dataset in more than 100 languages such as the common crawl (cc-100) or wikipedia (wiki-100). we also exploit two compact multilingual models with a distilled version of mbert: distil-mbert @xcite and small-mbert @xcite , a mbert model whose the original vocabulary has been reduced to two languages (english and french). ## experiments we propose an assessment of the comparative advantage gains in performance when using different training strategies (data augmentation, hyperparameter search and cross-lingual transfer) over monolingual and multilingual pre-trained models, larges and compacts for a qa task in french under resource constraints. ## conclusion and outlook recently, important progress has been made in neural language modeling using transformer networks. its popularity now well established lies in its effectiveness in modeling long-term dependencies. in this study, we have shown that a number of significant shortcomings of usability have recently been pointed out and that some solutions have been drawn up with compact models. we have also overviewed how the use of transformer-based pretrained language models have sparked a paradigmatic shift in question-answering training practices from task-specific architectures to the use of transfer learning through fine-tuning. comparing performances on a french questionanswering task using large and compact models provides insight into the usability of these models for under-resourced languages. as others, we argued that large and compact models cannot be used with limited data. our experimental results suggest that training strategy such as hyperparameter tuning or data augmentation can help to alleviate the data-gathering burden, with performances close to those of a high-resourced language such as english. finally, we present a new compact model for french fralbert (12m parameters), which proves to be as competitive as the large monolin-gual model camembert (110m parameters) pretrained on the same amount of text. in term of computational cost, we shown this compact model is twice less greedy than the bert base models. we also release a high-quality translated version of the squad corpus in french consisting of around 90k+ qa pairs. in a future work, we aim to continue this study from a meta-learning perspective with a modelagnostic approach generalizable to low-resource languages. we also plan to extend our model to other languages and to evaluate it on other nlp tasks such as named entity recognition or natural language understanding.