Data-Efficient F rench Language Modeling with C amem BERT a (2023)

Abstract: Recent advances in NLP have significantly improved the performance of language models on a variety of tasks. While these advances are largely driven by the availability of large amounts of data and computational power, they also benefit from the development of better training methods and architectures. In this paper, we introduce CamemBERTa, a French DeBERTa model that builds upon the DeBERTaV3 architecture and training objective. We evaluate our model’s performance on a variety of French downstream tasks and datasets, including question answering, part-of-speech tagging, dependency parsing, named entity recognition, and the FLUE benchmark, and compare against CamemBERT, the state-of-the-art monolingual model for French. Our results show that, given the same amount of training tokens, our model outperforms BERT-based models trained with MLM on most tasks. Furthermore, our new model reaches similar or superior performance on downstream tasks compared to CamemBERT, despite being trained on only 30% of its total number of input tokens. In addition to our experimental results, we also publicly release the weights and code implementation of CamemBERTa, making it the first publicly available DeBERTaV3 model outside of the original paper and the first openly available implementation of a DeBERTaV3 training objective.

## introduction advances in natural language processing (nlp) have been driven mainly by scaling up the size of pre-trained language models, along with the amount of data and compute required for training @xcite @xcite @xcite . however, these are not the only factors to determine a model's downstream performance, as the model's architecture and training objective are also important. @xcite showed that we can improve a model's performance by using disentangled attention, which uses two vectors to represent a token, one for position and one for content. @xcite later showed that performance could be further improved by using electra's @xcite self-supervised and sample-efficient replaced token detection objective. another crucial aspect lies in the ability to train models faster, which allows for quick iteration and thus accelerates the research process and allows for more efficient exploration of new ideas @xcite @xcite . this research aims to develop data-efficient and optimized training techniques that can improve performance in downstream tasks, while reducing the required training corpus size and compute. to achieve this goal, we propose a new data-efficient french language model based on debertav3 @xcite . our proposed model aims to optimize the training process by using a sampleefficient training objective, a state-of-the-art model architecture, and an efficient implementation. we evaluate downstream performance with a variety of nlp tasks, including dependency parsing, partof-speech tagging, named entity recognition, text classification, and question answering. we compare our model to a bert model trained with the masked language modeling (mlm) objective using the same tokenizer and training corpus, and to the state-of-the-art french language model, camem-bert @xcite , which required three times as many training iterations. our results show that our proposed model reaches or establishes a new state-of-the-art using one third of the computational budget of its main predecessors. our contributions can be summarized as follows: pre-trained french language models. current language models available for french are either trained using masked language modeling (mlm) or causal language modeling (clm). camembert @xcite and flaubert @xcite are two of the most popular contemporary french models, both trained with masked language modeling. other models include fralbert @xcite , a french version of albert @xcite , lepetit @xcite which is a small version of camembert, and d'alembert @xcite , a roberta @xcite based language model targeted towards early modern french. barthez @xcite is a sequence-to-sequence model trained with bart's objective @xcite , and pagnol @xcite and cedille @xcite are models trained with the clm objective. to the best of our knowledge, there is no prior effort in developing language models with this improved disentangled attention mechanism and objectives other than mlm/clm beyond english. ## camemberta: methodology the following section details our proposed architecture and pre-training objective, along with descriptions for the downstream tasks. architecture camemberta is based on the debertav3 @xcite architecture which uses two vectors to encode the word and its position, with the premise being that the relative position of a word pair should also directly affect the computed attention weights. the v3 version optimizes the initial deberta architecture by sharing the relative position embedding projection layers across all the encoder layers, and by adding a convolution layer aside the first encoder layer. 3 we use a base model configuration with 12 layers and 12 attention heads, 768 hidden dimensions with 32k for vocabulary size. training objective we follow the deber-tav3 @xcite pretraining strategy by using the replaced token detection (rtd) pre-training loss first introduced in electra @xcite , with a generator and discriminator based on the deberta architecture. during pre-training we project the generator embeddings to 256 dimensions and keep the generator model at 12 layers. during pre-training the generator model is trained using the mlm objective where we dynamically mask 15% of the input tokens. we then sample from the generator the masked tokens, and feed the output along with the unmasked tokens to the discriminator which is tasked to identify tokens that were replaced by the generator. the rtd objective increases sample efficiency since the model is predicting over all input tokens instead of the 15% masked tokens. in debertav3, the authors hypothesized and showed that sharing token embeddings between the generator and the discriminator results in a tugof-war situation, where the mlm and rtd tasks pull the embedding vectors into opposing directions. to alleviate this problem, the authors implemented gradient-disentangled embedding sharing (gdes), a method that re-parameterize the discriminator's token embeddings as pre-training we pre-train on the french subset of ccnet foot_3 @xcite , the same corpus used to pre-train camembert ccn et @xcite . 5 moreover we reuse camembert ccn et 's tokenizer @xcite . by reusing the pre-training corpus and tokenizer, we isolate the performance differences to the model architecture and training objective variables. optimization to speed up the pre-training experiments, we split the pre-training into two phases; in phase 1, the model is trained with a maximum sequence length of 128 tokens for 10,000 steps with 2,000 warm-up steps and a very large batch size of 67,584. in phase 2, maximum sequence length is increased to the full model capacity of 512 tokens for 3,300 steps with 200 warm-up steps and a batch size of 27,648. because we use very large batch sizes, we optimize the model using the lamb optimizer @xcite with a learning rate of 6e -3 , β 1 = 0.878, and β 2 = 0.974. ## experiments and results pre-training setup we re-implement the debertav3 rtd pre-training objective with gdes, since no public implementation was available at the time of writing. our training implementation is based on nvidia's electra and bert tensorflow2 implementations. 6 we train our models for 8 days on 6 nvidia a40 with horovod @xcite , and make use of xla compilation, mixed-precision and gradient accumulation to speed-up training and to fit large batch sizes with our limited compute. during pre-training, our model would have seen 133b tokens compared to 419b tokens for camembert ccn et which was trained for 100k steps. this represents roughly 30% of camem-bert's full training. hence for a fair comparison, we train a roberta model, which we dub camembert 30% , using our same exact pretraining setup but with the mlm objective. ## discussion our experiments clearly show that given the same training corpus, tokenizer, and total number of examples seen during training, camemberta outperforms the mlm trained camembert model on all tasks except ner on ftb and pos tagging on rhapsodie. moreover, our model implementation is able to match or outperform a fully trained camembert model, trained on around 3 times more samples and more compute. the strong performance of our model on higher level flue tasks suggest that lower level tasks such as pos tagging and dependency parsing are less challenging for current generation models, since they mostly require surface level information which the model can capture early in the training process, as suggested by @xcite , compared to tasks such as question answering and text classification which require more complex processing. taking a step back and looking at the only de-berta model that includes french, mdeberta @xcite we can see (cf. this confirms the interest in using such training paradigms in compute limited scenarios for semantically demanding tasks such as question-answering or natural-language inference. last but not least, other competitive language models for french are available and although not the primary focus of this paper, we conducted a comparative analysis involving flaubert @xcite and fralbert @xcite . the results, presented in ## conclusion we presented camemberta, a data-efficient french language model trained on a large corpus of french text and the first publicly available debertav3-style pretrained model and implementation. for a fair evaluation we reused the same corpus and tokenizer as camembert ccn et , but using only 30% of the total number of input training tokens. we compared the performance of both models in addition to an mlm model trained from scratch under the same setup as camem-berta, camembert 30% , on a variety of downstream tasks. our experiments showed that our model outperforms camembert 30% on all tasks except ner on ftb, and that it is able to match and even surpass camembert ccn et . furthermore, we have also made our optimized code implementation and pretrained model weights publicly available for others to use.