C amem BERT : a Tasty F rench Language Model (2020)

Abstract: Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models –in all languages except English– very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB). Our best performing model CamemBERT reaches or improves the state of the art in all four downstream tasks.

## introduction pretrained word representations have a long history in natural language processing (nlp), from noncontextual @xcite @xcite to contextual word embeddings @xcite . word representations are usually obtained by training language model architectures on large amounts of textual data and then fed as an input to more complex task-specific architectures. more recently, these specialized architectures have been replaced altogether by large-scale pretrained language models which are fine-tuned for each application considered. this shift has resulted in large improvements in performance over a wide * equal contribution. order determined alphabetically. these transfer learning methods exhibit clear advantages over more traditional task-specific approaches. in particular, they can be trained in an unsupervized manner, thereby taking advantage of the information contained in large amounts of raw text. yet they come with implementation challenges, namely the amount of data and computational resources needed for pretraining, which can reach hundreds of gigabytes of text and require hundreds of gpus @xcite . this has limited the availability of these state-of-the-art models to the english language, at least in the monolingual setting. this is particularly inconvenient as it hinders their practical use in nlp systems. it also prevents us from investigating their language modelling capacity, for instance in the case of morphologically rich languages. although multilingual models give remarkable results, they are often larger, and their results, as we will observe for french, can lag behind their monolingual counterparts for high-resource languages. in order to reproduce and validate results that have so far only been obtained for english, we take advantage of the newly available multilingual corpora oscar @xcite to train a monolingual language model for french, dubbed camembert. we also train alternative versions of camembert on different smaller corpora with different levels of homogeneity in genre and style in order to assess the impact of these parameters on downstream task performance. camembert uses the roberta architecture @xcite , an improved variant of the high-performing and widely used bert architecture @xcite . we evaluate our model on four different downstream tasks for french: part-of-speech (pos) tagging, dependency parsing, named entity recognition (ner) and natural language inference (nli). camembert improves on the state of the art in all four tasks compared to previous monolingual and multilingual approaches including mbert, xlm and xlm-r, which confirms the effectiveness of large pretrained language models for french. we make the following contributions: ## downstream evaluation tasks in this section, we present the four downstream tasks that we use to evaluate camembert, namely: part-of-speech (pos) tagging, dependency parsing, named entity recognition (ner) and natural language inference (nli). we also present the baselines that we will use for comparison. tasks pos tagging is a low-level syntactic task, which consists in assigning to each word its corresponding grammatical category. dependency parsing consists in predicting the labeled syntactic tree in order to capture the syntactic relations between words. for both of these tasks we run our experiments using the universal dependencies (ud) foot_2 framework and its corresponding ud pos tag set @xcite and ud treebank collection @xcite , which was used for the conll 2018 shared task @xcite . we perform our evaluations on the four freely available french ud treebanks in ud v2.2: gsd @xcite , sequoia foot_3 @xcite , spoken @xcite foot_4 , and partut @xcite . a brief overview of the size and content of each treebank can be found in we also evaluate our model in ner, which is a sequence labeling task predicting which words refer to real-world objects, such as people, locations, artifacts and organisations. we use the french treebank foot_5 (ftb) @xcite in its 2008 version introduced by @xcite and with ner annotations by @xcite . the ftb contains more than 11 thousand entity mentions distributed among 7 different entity types. a brief overview of the ftb can also be found in finally, we evaluate our model on nli, using the french part of the xnli dataset @xcite . nli consists in predicting whether a hypothesis sentence is entailed, neutral or contradicts a premise sentence. the xnli dataset is the exten-sion of the multi-genre nli (multinli) corpus @xcite to 15 languages by translating the validation and test sets manually into each of those languages. the english training set is machine translated for all languages other than english. the dataset is composed of 122k train, 2490 development and 5010 test examples for each language. as usual, nli performance is evaluated using accuracy. baselines in dependency parsing and postagging we compare our model with: in french, no extensive work has been done on ner due to the limited availability of annotated corpora. thus we compare our model with the only recent available baselines set by @xcite , who trained both crf @xcite and bilstm-crf @xcite architectures on the ftb and enhanced them using heuristics and pretrained word embeddings. additionally, as for pos and dependency parsing, we compare our model to a fine-tuned version of mbert for the ner task. for xnli, we provide the scores of mbert which has been reported for french by @xcite . we report scores from xlm mlm-tlm (described above), the best model from @xcite . we also report the results of xlm-r @xcite . ## camembert: a french language model in this section, we describe the pretraining data, architecture, training objective and optimisation setup we use for camembert. ## impact of corpus origin and size in this section we investigate the influence of the homogeneity and size of the pretraining corpus on downstream task performance. with this aim, we train alternative version of camembert by varying the pretraining datasets. for this experiment, we fix the number of pretraining steps to 100k, and allow the number of epochs to vary accordingly (more epochs for smaller dataset sizes). all models use the base architecture. in order to investigate the need for homogeneous clean data versus more diverse and possibly noisier data, we use alternative sources of pretraining data in addition to oscar: in order to make the comparison between these three sources of pretraining data, we randomly sample 4gb of text (at the document level) from os-car and ccnet, thereby creating samples of both common-crawl-based corpora of the same size as the french wikipedia. these smaller 4gb samples also provides us a way to investigate the impact the downstream task performances of the models trained on the 4gb version of ccnet and os-car are much more similar. 16 16 we provide the results of a model trained on the whole ccnet corpus in the appendix. the conclusions are similar when comparing models trained on the full corpora: downstream results are similar when using oscar or ccnet. ## discussion since the pre-publication of this work @xcite , many monolingual language models have appeared, e.g. @xcite @xcite , for as much as 30 languages @xcite . in almost all tested configurations they displayed better results than multilingual language models such as mbert @xcite . interestingly, @xcite showed that using their flaubert, a roberta-based language model for french, which was trained on less but more edited data, in conjunction to camem-bert in an ensemble system could improve the performance of a parsing model and establish a new state-of-the-art in constituency parsing of french, highlighting thus the complementarity of both models. 18 as it was the case for english when bert was first released, the availability of similar scale language models for french enabled interesting applications, such as large scale anonymization of legal texts, where camembert-based models established a new state-of-the-art on this task @xcite , or the first large question answering experiments on a french squad data set that was released very recently @xcite where the authors matched human performance using camembert large . being the first pre-trained language model that used the opensource common crawl oscar corpus and given its impact on the community, camembert paved the way for many works on monolingual language models that followed. furthermore, the availability of all its training data favors reproducibility and is a step towards better understanding such models. in that spirit, we make the models used in our experiments available via our website and via the huggingface and fairseq apis, in addition to the base camembert model. ## conclusion in this work, we investigated the feasibility of training a transformer-based language model for lan-guages other than english. using french as an example, we trained camembert, a language model based on roberta. we evaluated camem-bert on four downstream tasks (part-of-speech tagging, dependency parsing, named entity recognition and natural language inference) in which our best model reached or improved the state of the art in all tasks considered, even when compared to strong multilingual models such as mbert, xlm and xlm-r, while also having fewer parameters. our experiments demonstrate that using web crawled data with high variability is preferable to using wikipedia-based data. in addition we showed that our models could reach surprisingly high performances with as low as 4gb of pretraining data, questioning thus the need for large scale pretraining corpora. this shows that state-of-the-art transformer-based language models can be trained on languages with far fewer resources than english, whenever a few gigabytes of data are available. this paves the way for the rise of monolingual contextual pre-trained language-models for under-resourced languages. the question of knowing whether pretraining on small domain specific content will be a better option than transfer learning techniques such as fine-tuning remains open and we leave it for future work. pretrained on pure open-source corpora, camem-bert is freely available and distributed with the mit license via popular nlp libraries (fairseq and huggingface) as well as on our website camembert-model.fr.