AIT _ FHSTP at G erm E val 2021: Automatic Fact Claiming Detection with Multilingual Transformer Models (2021)

Abstract: Spreading ones opinion on the internet is becoming more and more important. A problem is that in many discussions people often argue with supposed facts. This year’s GermEval 2021 focuses on this topic by incorporating a shared task on the identification of fact-claiming comments. This paper presents the contribution of the AIT FHSTP team at the GermEval 2021 benchmark for task 3: “identifying fact-claiming comments in social media texts”. Our methodological approaches are based on transformers and incorporate 3 different models: multilingual BERT, GottBERT and XML-RoBERTa. To solve the fact claiming task, we fine-tuned these transformers with external data and the data provided by the GermEval task organizers. Our multilingual BERT model achieved a precision-score of 72.71%, a recall of 72.96% and an F1-Score of 72.84% on the GermEval test set. Our fine-tuned XML-RoBERTa model achieved a precision-score of 68.45%, a recall of 70.11% and a F1-Score of 69.27%. Our best model is GottBERT (i.e., a BERT transformer pre-trained on German texts) fine-tuned on the GermEval 2021 data. This transformer achieved a precision of 74.13%, a recall of 75.11% and an F1-Score of 74.62% on the test set.

## introduction today's social media platforms allow any individual to share information and opinions easily and quickly across a wide audience with almost no restrictions. however, not only obviously offensive comments, but also comments and posts with false information are becoming a serious problem on the internet. the sheer amount of available information and content generated every day makes it impossible to verify all information. thus, misinformation and false information can easily spread and influence people and their decisions, which has a strong impact on our society. as a workshop part of the konvens 2021 (konferenz zur verarbeitung natürlicher sprache / conference of natural language processing) the germeval 2021 focuses on the problem of fact claiming, i.e., the identification of content in social media that contains potential facts that need to be checked @xcite . the identification of such fact claiming content is a first step in the information verification process to separate relevant from irrelevant information for fact checking. our team participated in the fact claiming task (task 3: identification of fact-claiming comments) of germeval 2021 and this paper presents our methodology and the results. to solve the task we fine-tuned (supervised) the pre-trained transformer models with the original germeval 2021 data and external data, i.e., the claimbuster dataset @xcite . the employed datasets and our general approach are described in section 2. a detailed description of the transformer-based mod-els is provided in section 3. in section 4, our experimental setup is introduced. the results can be found in section 5 followed by a brief discussion and conclusion in section 6. ## methodological approach the germeval 2021 provided one labeled dataset for all three tasks (task 1 and 2 not considered in our contribution). the data for task 3 contained approx. 1/3 of content that mentions claimed facts and 2/3 with no claimed facts. we applied three pre-trained transformer models @xcite to encode and classify the content for this task, namely: german oscar text trained bert (gottbert) @xcite , multilingual bert (mbert) @xcite and xlm-roberta (xlm-r) @xcite . transformers are usually pre-trained on a large general corpus and can be used for many natural language processing (nlp) downstream tasks, which makes them especially useful for small training corpora @xcite . compared to mbert and xlm-r, which are both pre-trained on multilingual data, gottbert is the only one that was trained on one language (german) only. we fine-tuned these models in a supervised manner for binary classification into fact claiming comments and non fact claiming comments. since we employ two multilingual models, we chose to fine-tune one of those (mbert) on the germeval 2021 data and an additional dataset. in comparison, we fine-tuned our second multilingual model (xlm-r) and the german gottbert model using only the training data provided by the germeval 2021 shared task. the applied method is derived from our approach @xcite presented in the ex-ist 2021 challenge. the first shared task on sexism identification in social networks (exist) at iberlef 2021 @xcite , covering a wide spectrum of sexist content and aims to differentiate different types of sexist content. in our exist 2021 contribution a comparable set of transformer models and processing steps were applied @xcite . ## models in total we used three different architectures, which are all based on the original transformer @xcite model: mbert is a multilingual transformer based on the original structure of bert (bidirectional encoder representations from transformers) @xcite . however, bert was only trained on english data in comparison to the multilingual model which was additionally trained on wikipedia data in 100 languages @xcite . bert in general consists -unlike the original transformer with its encoder / decoder architecture @xcite -only of an encoder and is pre-trained using two different strategies: masked language modeling (mlm) and next sentence prediction (nsp) @xcite . mlm masks words with a specific pattern in a sequence that the model has to predict using its bidirectionality and multi-headed attention (reading a sentence from left-to-right and right-to-left). nsp is the task of predicting the following sentence in the text input @xcite . gottbert: is a monolingual transformer model, which is based on roberta (robustly optimized bert pretraining approach) @xcite . the latter used the bert architecture, but was trained with more data over a longer time period. additionally, nsp was not used for pre-training the model and mlm was changed from static to dynamic, where they use a different mask pattern for every sequence during training instead of the same as in bert. roberta outperforms bert in several nlp downstream tasks @xcite . since the original roberta model was only trained on english data, gottbert was trained from scratch, with the same parameters as the german bert version, on the german data of the oscar corpus @xcite . xlm-r: is a self-supervised cross-lingual model that was -similarly as mbert -trained with monolingual commoncrawl data in 100 languages @xcite . the architecture is based on roberta (similarly as gottbert) in combination with the multilingual xlm transformer @xcite . xlm uses more language modeling approaches (conneau and lample, 2019) than roberta and is only trained monolingually with mlm @xcite . xlm-r outperforms mbert on multiple tasks @xcite . evaluation results showed that xlm-r especially works well for languages with less available data in comparison to other models @xcite . the three models do not only differ in the number of languages that they were trained on: bert and roberta have different pre-training strategies, whereas the strategy of roberta are used by gottbert as well as xlm-r. as more training data is used, the vocabulary increases, resulting in longer pre-training and fine-tuning intervals. this usually has a positive influence on the performance of downstream tasks. ## experimental setup figure 1 provides an overview of our experimental setup and the training strategies used to solve the fact claiming task. the two main approaches take two distinct parts of input data, i.e., only ger-meval 2021 data or in addition claimbuster data as input. to evaluate the proposed methods we performed experiments by utilizing the following pretrained transformer models provided by the hug-gingface @xcite library: mbert (devlin et al., 2019a), gottbert 2 , and xlm-r 3 (conneau et al., 2019). the experimental setup for the three models is described in detail below. ## results the validation and test results in terms of precision, recall, and macro-averaged f1-score are presented in run 1: the mbert seems to generalize well, as the f1-score on the test set of 72.84% is at a similar performance level as on the validation set (76.09%). this result on the test set is the second highest achieved in our experiments. abbreviation "val" stands for our validation set and "test" for the official benchmark test set. the performance measures are expressed in percent (%). run 2: the fine-tuned gottbert on the ger-meval 2021 data achieved an overall f1-score of 74.62% on the test set (78.90% on the validation set). these results speak for the generalization ability of this network because the test performance is at a similar performance level as on the validation set. this result is the highest obtained for all our models. run 3: the xlm-r fine-tuned on the ger-meval 2021 data achieved the lowest f1-score on the test set (69.27%). this approach seems to exhibit a strong overfitting behavior, as the results on the validation set are considerably higher (f1-score of 76.73%). in conclusion, the gottbert model (run 2) achieves the highest results in our experiments. these results indicate that the model that is pretrained on german data allows for a better modeling of the semantics of the task than a multilingual model. all other models are also beyond the zerorule baseline which is at 66% for the test set. a more detailed analysis of the results shows that all three models consistently predicted the same class in 560 cases (corresponds to approx. 60% of the test set). in the following, two examples are given for both classes: example 1 "@user sie würden wahrscheinlich auch einen kriegstreiber/in wählen, wenn es gegen trump ginge, warten sie es ab , vielleicht geht ihr wunsch ja in erfüllung...." the ground truth and predictions of all models for this example are "0" (not fact claiming). example 2 "@user , ich glaube,sie verkrnnen gründlich die situation. deutschland mischt sich nicht ein, weil die letzte einmischung in der ukraine noch nicht bereinigt ist. es geht nicht ums militär" the ground truth and predictions of all models for this example are "1" (fact claiming). furthermore, all three models consistently predicted the wrong class in 107 cases ( corresponds to approx. 11% of the test set). in the following, two examples are given for both classes: example 1 "hackt nicht nimmer auf den fussball rum. bei allem sportarten sind wieder zuschauer erlaubt. hygienekonzept vorausgesetzt." the ground truth is "1" (fact claiming) and predictions of all models are "0" (not fact claiming). example 2 "biden gewinnt, corona wird weggehen, amerika wird reich,k alle bekommen arbeit und die welt wird schön. also was sollst." the ground truth is "0" (not fact claiming) and predictions of all models are "1" (fact claiming). in the remaining 273 cases (corresponds to approx. 29% of the test set), one of the models did not predict the same as the others. mbert and got-tbert predicted equally in 100 cases (70 correctly and 30 incorrectly). gottbert and xlm-r predicted equally in 100 cases (47 correctly and 53 incorrectly). mbert and xlm-r predicted equally in 73 cases (28 correctly and 45 incorrectly). these results show that even though both pairs mbert and gottbert on one side and mbert and xlm-r on the other side predict equally in most cases (100), mbert and gottbert predict correctly in significantly more cases (70). ## conclusion & future work in this paper, we described our submission to the "fact-claiming comment classification" task of the germeval 2021. in our experiments gottbert, a transformer-based machine learning model pretrained on german data only, achieved the best results, leading to an f1-score of 74.62% on the test set. for the multilingual transformer models, we obtained better results with mbert (potentially because it was trained with an additional dataset) than with xlm-r, which seems to have slightly overfitted on the training data. future work will focus on evaluating the different models and approaches in more detail and to investigate how they specifically adapt to the underlying data. we will further investigate how the use of external data impacts the performance of all three investigated models, especially gottbert, which seem to be the most promising option. due to the similarity of the presented approaches in this challenge and our previous submission to the exist 2021 challenge, see @xcite , we plan to perform comparisons on how the applied models converge with respect to the different datasets, semantic concepts and downstream tasks addressed in the benchmarks. furthermore, we will analyze whether the findings from this comparison can be applied to related tasks such as disinformation detection @xcite .