BERT abaporu: Assessing a Genre-Specific Language Model for P ortuguese NLP (2023)

Abstract: Transformer-based language models such as Bidirectional Encoder Representations from Transformers (BERT) are now mainstream in the NLP field, but extensions to languages other than English, to new domains and/or to more specific text genres are still in demand. In this paper we introduced BERTabaporu, a BERT language model that has been pre-trained on Twitter data in the Brazilian Portuguese language. The model is shown to outperform the best-known general-purpose model for this language in three Twitter-related NLP tasks, making a potentially useful resource for Portuguese NLP in general.

## introduction transformer-based language models such as bidirectional encoder representations from transformers (bert) @xcite are now mainstream in the nlp field, but extensions are still much in demand. new bert models have been fine-tuned or built from scratch for many languages other than english (e.g., maltese in @xcite ), for specific domains (e.g., mental health in @xcite ), and for particular text genres (e.g., twitter data in @xcite ). in the case of our target language -brazilian portuguese -the first and best-known representative of this trend is bertimbau, a general purpose bert model built from a large collection of web documents @xcite . despite its popularity among the portuguese nlp community, however, we notice that the particular kind of language employed on contemporary social media may be distinct from the training data considered in previous work, making existing models potentially less suitable to handle recent social media text. in particular, we notice that tweets are not only shorter and less structured than pieces of news, but words such as 'covid' may not be recognised by older language models. based on these observations, we may ask whether twitter-related applications may benefit from a more genre-specific model built from social media data -as opposed to more standard or general text -perhaps along the lines of bertweet, a twitter-specific model for the english language described in @xcite , or the multilingual twhin-bert @xcite , also built from twitter data. to shed light on this issue, this work introduces bertabaporu, a bert model built from a collection of 238 million tweets written by over 100 thousand unique twitter users, and conveying over 2.9 billion tokens in total. the model has been evaluated in three twitter-related text classification tasks, and its results are compared to those obtained by the general purpose, web-based bertimbau in @xcite . in doing so, our goal is to introduce a novel resource for portuguese nlp, and foster further applications based on twitter text data in this language. the main contributions made in this work are (i) a novel bert model trained on large twitter corpus in the portuguese language; and (ii) comparison with the best-known existing portuguese bert in three twitter text classification tasks, namely, stance, mental health and political alignment prediction. the rest of this article is structured as follows. section 2 reviews existing work that have introduced bert-like models for portuguese, for other languages and domains. section 3 describes how our current work, the bertabaporu model, has been built. section 4 introduces the downstream tasks in which bertabaporu is to be assessed. section 5 reports results obtained for each of the evaluation tasks, and compares these to the results obtained by existing work. finally, section 6 draws our conclusions and suggestions of future work. ## background since the original english and multilingual (or mbert) models described in @xcite , similar resources devoted to these and to dozens of other languages have been created. models of this kind are either trained from scratch or fine-tuned (often from mbert) using either general purpose or domain-or genre-specific text data. noteworthy examples include bertweet (nguyen et al., 2020), a 16-billion token model built from twitter data in the english language, domain-specific models fine-tuned for mental health @xcite , abusive language use @xcite , biomedical texts @xcite and others, alongside general purpose models for a wide range of languages including, e.g., @xcite , maltese @xcite , romanian @xcite and czech @xcite . in the case of the portuguese language, although model repositories such as hugging face provide a considerable number of bert models -particularly for the legal domain, and even including a few models trained on twitter data -we have identified only three models of this kind that are more fully documented in the nlp literature. these are summarised in two of the existing portuguese models -biobertpt and petrobert -are fine-tuned to perform domain-specific tasks in the clinical/biomedical and oil and gas industry domains, respectively. this makes web-based bertimbau @xcite the more closely related alternative to our own work (bertabaporu, on the top row of the table). bertimbau is the first, and arguably the best-known portuguese bert model to date, and it has been trained from scratch using a general purpose web corpus in which great care has been taken to minimise the effects of duplicate data, making it a suitable baseline to our current work. based on these observations, our present work bertabaporu may be seen as a general purpose, twitter-specific alternative to bertimbau built from a slightly larger dataset (2.9 billion tokens against 2.7 billion in bertimbau), and which should be able to outperform the existing webbased model in twitter-oriented tasks. ## the bertabaporu model in order to gather unlabelled text data to built bertabaporu, we selected a number of existing tweet repositories and collected additional data online. as a means to minimise the effect of duplicated training data, bertabaporu has been built from tweets originally posted by over 100 thousand unique twitter users excluding their retweets. in this user-centred method, although some data duplication may still occur (namely, if an individual rewrites the same text that another user has authored), we assume that the effect of duplicates is likely to be small. users were selected from a number of pseudorandom tweet sources based on a number of seed topics, such as covid-19, politics, mental health and others, and then their entire public timelines were collected regardless of the topics under discussion. thus, it should be clear that the data is by no means limited to these topics, and that we did not search for individual tweets about any particular topic, but rather used the seed topics as a guideline to identify users timelines, and then collect all their publications (which will inevitably discuss a very broad range of subjects besides the seed topic.) in other words, our data consists of a collection of pseudo-random user timelines, and not a collection of tweets about the seed topics, and should not be seen as being significantly biased towards any particular topic. selected user timelines comprised three main categories: (i) timelines of random users (about 33%); (ii) timelines of users who discussed covid-19, politics, mental health issues, vaccines or other covid-19 measures at least once (about 46%); and timelines of friends with whom these users most frequently interact (about 21%). from the selected timelines, all non-portuguese tweets were removed. from the remainder, emoticons, non-alphabetic characters, urls and usernames were removed, and numbers were replaced by '1'. finally, timelines conveying fewer than 80 tweets were discarded. from the above unlabelled data, we pre-trained a monolingual bert model from scratch using both bert-base and bert-large architectures. the base version uses 12 transformer layers, a hidden size of 768, and 8 attention heads. the large version uses 24 transformer layers, a hidden size of 1024, and 16 attention heads. in both cases, the vocabulary is initialised with 64k tokens. pretraining is performed across 1m steps, with a sequence length of 128 for the first 90% of the steps and a sequence length of 512 for the remaining 10% steps. the models use a batch size of 512, and a warm-up of 1% of the total number of steps. training was performed on v2-8 tpus, taking approximately 120 hours for both configurations. the resulting language model is publicly available for reuse 1 . ## evaluation we envisaged a number of twitter text classification experiments to compare our current twitter bertabaporu model with the general-purpose alternative in @xcite . in doing so, we would like to show that genre-specific bertabaporu obtains superior results in these evaluation scenarios. ## results from these results we notice that domainspecific bertabaporu (right side of ## final remarks this paper introduced bertabaporu, a bert language model pre-trained on twitter data in the brazilian portuguese language. compared to previous work, the present models has been found to outperform the best-known general-purpose model for this language in three twitter-related text classification tasks, namely, stance, mental health statues, and political alignment prediction, and may be potentially useful to many others twitter-related applications in the portuguese language. as future work, we intended to extend the present analysis by assessing the use of bertabaporu in other portuguese nlp tasks. moreover, since the present model has been trained on a considerably large dataset, there is the question of whether bertabaporu may be helpful even in non-twitter evaluation settings. an investigation along these lines is also left as future work.