C o NTACT : A D utch COVID -19 Adapted BERT for Vaccine Hesitancy and Argumentation Detection (2022)

Abstract: We present CoNTACT: a Dutch language model adapted to the domain of COVID-19 tweets. The model was developed by continuing the pre-training phase of RobBERT (Delobelle et al., 2020) by using 2.8M Dutch COVID-19 related tweets posted in 2021. In order to test the performance of the model and compare it to RobBERT, the two models were tested on two tasks: (1) binary vaccine hesitancy detection and (2) detection of arguments for vaccine hesitancy. For both tasks, not only Twitter but also Facebook data was used to show cross-genre performance. In our experiments, CoNTACT showed statistically significant gains over RobBERT in all experiments for task 1. For task 2, we observed substantial improvements in virtually all classes in all experiments. An error analysis indicated that the domain adaptation yielded better representations of domain-specific terminology, causing CoNTACT to make more accurate classification decisions. For task 2, we observed substantial improvements in virtually all classes in all experiments. An error analysis indicated that the domain adaptation yielded better representations of domain-specific terminology, causing CoNTACT to make more accurate classification decisions.

## introduction since the development of covid-19 vaccines, an us-against-them mentality has emerged between advocates and adversaries of vaccines. social media functions as a catalyst in this polarization, because it enables a rapid spread of unsolicited opinions, ranging from nuanced to radical, and well-reasoned to emotional. the turbulent vaccine debate that has been hosted by social media will unquestionably continue to influence future views on vaccination, whether it regards new covid-19 boosters or other vaccines. in order to help process the large amounts of online content related to vaccines and covid-19 in general, we present contact (contextual neural transformer adapted to covid-19 tweets). contact was developed by fine-tuning robbert @xcite (a robertabase model @xcite pre-trained on dutch data) on masked language modeling using 2.8m dutch-language tweets related to covid-19 that were posted in 2021. the model was evaluated on two tasks: (1) binary vaccine hesitancy classification and (2) classification of arguments for vaccine hesitancy. in order to measure the effect of the domain adaptation, the results were compared to out-of-the-box robbert. moreover, the aforementioned tasks were not only performed on tweets, but also on facebook comments to show the crossgenre benefits of the domain adaptation. afterwards, a qualitative error analysis was conducted to show where contact improved compared to rob-bert and where it could potentially improve further. in this analysis, special attention was paid to what domain-specific language the model learned compared to out-of-the-box robbert. in earlier research, an english language model pre-trained on covid-19 related tweets (covid-twitter-bert) was developed @xcite . we apply the same methodology for the first time to dutch and extensively test the effect on two covid-19 related classification tasks. ## related research traditional machine learning assumes that models are trained and tested on large amounts of data from the same domain, which is not always feasible due to lack of labelled data. transfer learning, which involves the transfer of knowledge from one domain to another, is a technique that has been utilized frequently in machine learning, both in nlp (e.g. @xcite , @xcite , rostami and galstyan (2021)) and computer vision (e.g. @xcite , @xcite , @xcite ) to combat this issue. an effective approach to transfer learning that has dominated in various nlp tasks in recent years is the pretraining of language models, such as bert @xcite , on large amounts of unsupervised data. the knowledge from this pre-training phase is then transferred to the subsequent fine-tuning phase on task-and domain-specific data, which has shown significant improvements on several benchmark datasets, e.g., @xcite and @xcite , even when using relatively small amounts of labeled data. subsequently, several languageand domain-specific adaptations of language models have been developed for non-english data or to further improve the performance of the original models on specific tasks. examples are bertje and @xcite , the dutch equivalents of bert and roberta, respectively), and camembert @xcite , a french bert model. domain adaptation, a special case of transfer learning where the model is first trained on large amounts of unsupervised data from the domain of an intended task, aims to improve results even further than traditional transfer learning "by minimizing the difference between the domain distributions" @xcite , p. 1), thus creating a model that optimally learns from the training data. regarding the domain of covid-19, covid-twitter-bert @xcite , which is a bert-large model pre-trained on covid-19 related tweets, has shown statistically significant gains over the baseline bert-large model in various applications, including vaccine stance classification. in other research related to vaccine stance classification, the effectiveness of various rule-based, statistical and deep learning approaches on the classification of stance towards vaccines have been compared @xcite . concretely, the task consisted of multiclass classification of vaccine stance in social media messages ("for", "against" or "undecided"). the authors concluded that both pre-trained language models and statistical ensemble models achieved equally high results on this task. this work focused on vaccine stance in general, but since the start of the covid-19 pandemic vaccine stance classification has become almost inextricably linked to covid-19 due to its societal relevance. an example is @xcite who present covaxlies, a covid-19 vaccine misinformation dataset, and demonstrate that their knowledge-graph-based model outperforms widely used classification methods for the detection of vaccine misinformation, an important cause of vaccine hesitancy. specifically for dutch, wang et al. (2020) collected dutch tweets using keywords, and comments from reddit and nu.nl foot_1 threads related to covid-19 in order to investigate polarity ("positive"/"negative") and stance ("support"/"reject"/"other") towards face masks and the social distance measure between march and october 2020. for polarity analysis, the pattern library @xcite was used, whereas manual annotations were used to train a stance classifier consisting of a linear feed forward neural network using stochastic gradient descend and a subword embedding layer, which achieved a test set accuracy of 65%. after applying the polarity analyzer and stance classifier to the abovementioned data, it was shown that a more negative polarity was found in covid-19 related messages than in a subset of messages that were unrelated to covid-19. more specifically, a more negative polarity (and also stance) was found in messages mentioning face masks than in messages mentioning the social distancing measure. the various social media platforms that were used showed similar trends over time.