BERT weet FR : Domain Adaptation of Pre-Trained Language Models for F rench Tweets (2021)

Abstract: We introduce BERTweetFR, the first large-scale pre-trained language model for French tweets. Our model is initialised using a general-domain French language model CamemBERT which follows the base architecture of BERT. Experiments show that BERTweetFR outperforms all previous general-domain French language models on two downstream Twitter NLP tasks of offensiveness identification and named entity recognition. The dataset used in the offensiveness detection task is first created and annotated by our team, filling in the gap of such analytic datasets in French. We make our model publicly available in the transformers library with the aim of promoting future research in analytic tasks for French tweets.

## introduction vector representations of words have given rise to the application of deep learning methods in nlp. traditional pre-training approaches, such as word2vec @xcite and glove @xcite are static, learning a single representation for every word regardless of context. however, words are often polysemous with different meanings depending on the context. more recently, models are trained to integrate contextual meaning : the output word embeddings depend on the whole input sequence rather than only the word itself. while the idea was initially implemented with recurrent neural networks (dai and @xcite , recent models have predominantly been based on the transformers architecture @xcite , with bert @xcite and roberta @xcite among the most popular. these contextualized word representation models have opened new doors for researchers as they can be applied to numerous downstream tasks by fine-tuning or prompting. in fact, large-scale pre-trained language models have become the go-to tool for building new nlp applications, saving researchers from the enormous amount of computational resource and data required for training model weights from scratch. in the past few years, human society has become more digitally connected than ever before. people use social media to report the latest news, but also to express their opinions and feelings about realworld events. as one of the most popular microblogging platforms, twitter has become a primary source for social media user-generated data @xcite . however, tweets are more often written in informal language compared to the carefully edited texts that are published in traditional data sources such as wikipedia and printed media. they have their own set of features such as the recurrent use of irregular or abbreviated words, the large quantity of spelling or grammatical mistakes, the employment of improper sentence structures and the occurrence of mixed languages @xcite . this presents challenges to standard nlp methods when applied to twitter data. domain-adaptive pre-training has been revealed to provide significant gains in helping models encode the complexity of specific textual domains @xcite . while efforts on domain adaptation of large-scale language models to twitter language have been made in english @xcite , there has been no similar work in any other language. we start addressing this problem by releasing bertweetfr, a pre-trained language model for french tweets together with a manually labeled dataset for offensiveness identification in french tweets. we evaluate our model on two downstream tasks : offensiveness identification and named entity recognition. we compare the performance of our model to the best-performing transformer-based models pretrained on general domain french texts, namely camembert @xcite , flaubert @xcite and @xcite . experiments show that our model outperforms all three of them on both of the downstream tasks. in order to facilitate future research on french tweets, we make our model publicly available on huggingface's model hub 1 as well as on our team's website dedicated to french linguistic resources 2 . we will also release our offensiveness identification dataset while respecting the limitations of twitter's developper policy. ## building the model in this section, we describe the collection steps and pre-procesing pipeline for our pre-training dataset, present the model architecture and introduce the training objective and optimization setup. ## downstream task datasets we evaluate the performance of bertweetfr on two downstream twitter nlp tasks. the datasets used in these downstream tasks are either constructed by ourselves or obtained from shared tasks in past conferences. ## baselines as our bertweetfr model is the first pre-trained language model for french tweets, we compare it with the following general-domain language models for french. camembert as mentioned in 2.2, caembert @xcite is the model from which we ## experiments and results in this section, we first describe our fine-tuning approaches. the baseline models follow the same fine-tuning procedures as bertweetfr. we then report our results and compare with the baselines. as a result, our model substantially outperforms all baselines. ## conclusion in this work, we investigated the effectiveness of applying domain adaptation to the twitter domain for large-scale pre-trained french language models. we demonstrate the value of our model showing that it outperforms all previous general-domain french language models on two downstream twitter nlp tasks of offensiveness identification and named entity recognition. our contributions are as follows : with around 70% of all twitter posts being in non-english languages, the lack of corresponding language models strongly hinders the community from exploiting the information contained in these valuable resources. for future work, we plan to train and release a series of such pre-trained language models for tweets in other low resource languages. we also call upon researchers from all over the world involved in natural language processing for social media to adapt language models in their respective languages and make them publicly available.