BART hez: a Skilled Pretrained F rench Sequence-to-Sequence Model (2021)

Abstract: Inductive transfer learning has taken the entire NLP field by storm, with models such as BERT and BART setting new state of the art on countless NLU tasks. However, most of the available models and research have been conducted for English. In this work, we introduce BARThez, the first large-scale pretrained seq2seq model for French. Being based on BART, BARThez is particularly well-suited for generative tasks. We evaluate BARThez on five discriminative tasks from the FLUE benchmark and two generative tasks from a novel summarization dataset, OrangeSum, that we created for this research. We show BARThez to be very competitive with state-of-the-art BERT-based French language models such as CamemBERT and FlauBERT. We also continue the pretraining of a multilingual BART on BARThez’ corpus, and show our resulting model, mBARThez, to significantly boost BARThez’ generative performance.

## introduction inductive transfer learning, that is, solving tasks with models that have been pretrained on very large amounts of data, was a game changer in computer vision @xcite . in nlp, while annotated data are scarce, raw text is virtually unlimited and readily available. it thus emerged that the ability to learn good representations from plain text could greatly improve general natural language understanding. trained on gigantic amounts of raw data and with hundreds of gpus, models based on the transformer architecture @xcite , such as gpt (rad) and bert @xcite , have set new state-of-the-art performance in every nlu task. moreover, users around the world can easily benefit from these improvements, by finetuning the publicly available pretrained models to their specific applications. this also saves considerable amounts of time, resources and energy, compared with training models from scratch. bart @xcite combined a bertliked bidirectional encoder with a gpt-like forward decoder, and pretrained this seq2seq architecture as a denoising autoencoder with a more general formulation of the masked language modeling objectives of bert. since not only bart's encoder but also its decoder is pretrained, bart excels on tasks involving text generation. while the aforementioned efforts have made great strides, most of the research and resources were dedicated to the english language, despite a few notable exceptions. in this paper, we partly address this limitation by contributing barthez foot_0 , the first pretrained seq2seq model for french. barthez, based on bart, was pretrained on a very large monolingual french corpus from past research that we adapted to suit bart's specific perturbation schemes. unlike already existing bertbased french language models such as camem-bert @xcite and flaubert @xcite , barthez is particularly well-suited for generative tasks. we evaluate barthez on five sentiment analysis, paraphrase identification, and natural language inference tasks from the recent flue benchmark, and two generative tasks from a novel french summarization dataset, orangesum, that we created for this research. we also continue the pretraining of an already pretrained multilingual bart on barthez' corpus. our resulting model, mbarthez, significantly boosts barthez' performance on generative tasks. our contributions are as follows: • we publicly release the first large-scale pretrained seq2seq model dedicated to the french language, barthez, featuring 165m parameters, and trained on 101 gb of text for 60 hours with 128 gpus. we evaluate barthez on five discriminative tasks and two generative tasks, with automated and human evaluation, and show that barthez is very competitive with the state of the art. ## related work learning without labels is enabled via selfsupervised learning foot_4 , a setting in which a system learns to predict part of its input from other parts of its input. in practice, one or more supervised tasks are created from the unlabeled data, and the model learns to solve these tasks with custom objectives. some of the earliest and most famous selfsupervised representation learning approaches in nlp are word2vec @xcite , glove @xcite and fasttext @xcite . while these methods were significant advancements, they produce static representations, which is a major limitation, as words have different meanings depending on the unique contexts in which they are used. deep pretrained language models. elmo (peters et al., 2018) provided the first contextualized embeddings, by extracting and combining the internal states of a pretrained deep bi-lstm language model. except for the word embeddings and the softmax layer, the forwards and backwards rnns have different parameters. the authors of elmo showed that the learned representations could be transferred with great benefits to downstream architectures, to solve a variety of supervised nlu tasks. beyond simply combining internal states, @xcite proposed ulmfit, a universal transfer learning method for text classification where the language model is pretrained on a large, general dataset, finetuned on a specific dataset, and finally augmented with classification layers trained from scratch on downstream tasks. with the openai gpt, rad capitalized on the transformer architecture @xcite , superior and conceptually simpler than recurrent neural networks. more precisely, they pretrained a left-to-right transformer decoder as a general language model, and finetuned it on 12 language understanding tasks by applying different transformations to the input. by combining ideas from all the aforementioned models, and introducing bidirectional pretraining, bert @xcite disrupted the nlp field by setting new state-of-the-art performance on 11 nlu tasks, with very wide margins. more precisely, bert uses a bidirectional transformer encoder with a masked language model objective, making the learned representations capture both the left and the right contexts, instead of just the left context. the sheer size of bert, with up to 24 transformer blocks, plays a role in performance too. with gpt-2, a version of gpt with over an order of magnitude more parameters than gpt, @xcite showed that as long as they have very large capacities, general language models can reach reasonable performance on many specific nlu tasks out-of-the-box, without any finetuning, i.e., accomplish zero-shot transfer. this demonstrates the fundamental nature and importance of the language modeling objective for inductive transfer learning. in roberta, @xcite showed that the performance of bert could be improved by optimizing its hyperparameters and training procedure. the study of why and how bert works so well has now its own dedicated research field, known as bertology @xcite . languages. following the success of bert for the english language, some bert models were pretrained and evaluated in other languages. some examples include arabic (antoun et al., 2020), dutch @xcite , french @xcite , italian @xcite , portuguese @xcite , russian @xcite , and spanish @xcite . in addition to the aforelisted monolingual models, multilingual models were also proposed, notably mbert @xcite , xlm @xcite and xlm-r @xcite . abstractive summarization. abstractive summarization is an important and challenging task, requiring diverse and complex natural language understanding and generation capabilities. a good summarization model needs to read, comprehend, and write well. gpt-2 can be used for summarization, by sampling a certain numbers of tokens from a given start seed. however, while the generated text is grammatical and fluent, summarization performance is only slightly superior to that of a random extractive baseline. being a bidirectional encoder, bert cannot be used out-of-the-box for language generation, unlike gpt-2. furthermore, bert produces singlesentence representations, whereas for summarization, reasoning over multiple sentence and paragraph representations is necessary. @xcite proposed a way to overcome these challenges. at the input level, they introduced special tokens to encode individual sentences, interval segment embeddings, and used more position embeddings than in bert. then, they combined a pretrained bert encoder with a transformer-based decoder initialized at random and jointly trained the two models with different optimizers and learning rates. bart and mbart. bart @xcite is a denoising auto-encoder that jointly pretrains a bidirectional encoder (like in bert) and a forward decoder (like in gpt) by learning to reconstruct a corrupted input sequence. both the encoder and the decoder are transformers. since not only the encoder but also the decoder is pretrained, bart is particularly effective when applied to text generation tasks. @xcite pretrained a multilingual bart (mbart) on 25 different languages. they showed that this multilingual pretraining brings significant performance gains on a variety of machine translation tasks. mass @xcite is another multilingual pretrained sequence to sequence model, that learns to predict a masked span in the input sequence. the main difference between mass and bart, is that the former only predicts the masked fragment of the sentence, while the latter learns to reconstruct the entire corrupted sentence. this difference makes mass less effective in discriminative tasks, given that only the masked span is fed to the decoder @xcite . prophet-net @xcite which also adopts the encoderdecoder structure, introduces a new learning objective called future n-gram prediction. this objective reduces overfitting on local correlations by learning to predict the next n-grams (instead of unigrams) at each time step given the previous context 3 barthez our model is based on bart @xcite , a denoising auto-encoder. it consists of a bidirectional encoder and a left-to-right auto-regressive decoder. ## mbarthez we continued the pretraining of the pretrained mbart on barthez' corpus (see subsection 3.4) for about 30 hours on 128 nvidia v100 gpus, which allowed for 4 passes over barthez' corpus. this can be seen as an instance of languageadaptive pretraining, which goes a step further than domain-adaptive pretraining @xcite . the initial learning rate was set to 10 -4 and linearly decreased towards zero. we call the resulting model mbarthez. note that we eliminated the tokens containing non-latin characters from mbarthez' embedding layers, reducing its number of parameters from 610m to 458m. ## orangesum bart-based models are particularly well-suited to generative tasks, but flue @xcite , the french equivalent of glue, only contains discriminative tasks foot_6 @xcite . we therefore decided to create one such task. we opted for single-document abstractive summarization, as it is a generative task that also requires the model to encode its input very well. in other words, for a model to summarize well, it needs to both read, comprehend, and write well, making abstractive summarization one of the most central and challenging evaluation tasks in nlp. motivation. our strategy here was to create a french equivalent of the recently introduced xsum dataset @xcite . xsum was created by scraping articles and their one-sentence summaries from the bbc website, where the summaries are not catchy headlines, but capture the gist of the articles. unlike the historical cnn, dai-lymail, and ny times datasets @xcite , which favor extractivity, performing well on xsum requires a high degree of abstractivity. data collection. we adopted an analogous strategy, and scraped the "orange actu" website 8 . orange s.a. is a large french telecommunications corporation, with 266m customers worldwide. our scraped pages cover almost a decade from feb 2011 to sep 2020. they belong to five main categories: france, world, politics, automotive, and society 9 . the society category is divided into 8 subcategories: health, environment, people, culture, media, hightech, unusual ("insolite" in french), and miscellaneous. each article featured a single-sentence title as well as a very brief abstract, both professionally written by the author of the article. we extracted these two fields from each page, thus creating two summarization tasks: orangesum title and or-8 https://actu.orange.fr/ , 'actu' means news. 9 root urls are https://actu.orange.fr/ for all categories except https://auto.orange.fr/news/ for automotive. angesum abstract. gold summaries are respectively 11.42 and 32.12 words in length on average (see for both orangesum title and orangesum abstract, we set aside 1500 pairs for testing, 1500 for validation, and used all the remaining ones for training. we make the dataset publicly available 10 . an example document with its summaries is provided in ## experiments we compare barthez and mbarthez with the following models, summarized in ## conclusion we released barthez and mbarthez, the first french large-scale pretrained seq2seq models, as well as a novel french summarization dataset, or-angesum, inspired by the xsum dataset. we showed our proposed models to be very competitive with the state of the art on generative and discriminative tasks. by evaluating our models on the summarization dataset we showed that: (1) barthez is on par with mbart while having four times less parameters, and that (2) mbarthez pro-vides a significant boost over mbart by simply adding a relatively affordable language-adaptive phase to the pretraining. in addition, we evaluated barthez and mbarthez on 5 sentiment analysis, paraphrasing, and natural language inference tasks against cutting edge bert-based french language models (flaubert and camembert), and obtained very competitive results. an interesting area for future work is to further explore languageadaptive pretraining.