CL a C at S em E val-2023 Task 3: Language Potluck R o BERT a Detects Online Persuasion Techniques in a Multilingual Setup (2023)

Abstract: This paper presents our approach to the SemEval-2023 Task 3 to detect online persuasion techniques in a multilingual setup. Our classification system is based on the RoBERTa-base model trained predominantly on English to label the persuasion techniques across 9 different languages. Our system was able to significantly surpass the baseline performance in 3 of the 9 languages: English, Georgian and Greek. However, our wrong assumption that a single classification system trained predominantly on English could generalize well to other languages, negatively impacted our scores on the other 6 languages. In this paper, we provide a description of the reasoning behind the development of our final model and what conclusions may be drawn from its performance for future work.

## introduction the semeval-2023 shared task 3 @xcite focuses on 3 subtasks: detecting the news genre (subtask 1), the framing (subtask 2) and the persuasion techniques (subtask 3) used in online news across different languages. training data provided by the organizers was collected from news and web articles between the years 2020 and 2022 in six different languages, namely english, french, german, italian, polish, and russian. during the testing phase of the shared task, three additional surprise languages were included only as testing data, georgian, greek and spanish, to explore the language-agnostic robustness of the trained models. in this paper, we focus on the detection of persuasion techniques used to influence the reader of these online news as part of subtask 3 @xcite . we did not participate in the news genre categorisation (subtask 1) and framing detection (subtask 2). in total, 23 different persuasion techniques were annotated at the paragraph-level for each article across all the training languages. such techniques include loaded language, guilt by association and the straw man fallacy. a more detailed description of each persuasion technique is available in the annotation guidelines of the shared task @xcite . we applied a roberta @xcite based model trained predominantly on the english training dataset enriched with specific instances from the other training languages to the detection of persuasion techniques across all languages. section 3 provides an overview of our final classification pipeline. in section 4, we provide a detailed description of the experiments and data augmentation techniques that led to our final model decision. however, our assumption that a predominately english trained model would be able to generalize well and even outperform models trained on languages specific to the prediction task was proven wrong by the official leaderboard results of the shared task as further discussed in section 5. with the exception of english and two of the three surprise test languages, greek and georgian, where we were able to attain significantly higher scores than the baseline, our final model either performed worse or only slightly better than the baseline. all of the code used in the implementation of the models described in this paper is made available on github foot_0 . ## background the massively widespread of misinformation to deliberately influence opinions and beliefs through the use of rhetorical and psychological persuasion techniques, commonly referred to as fake news, has become a major societal problem capable of even swinging elections @xcite . it has, thus, become of paramount importance to develop systems capable of automatically detecting such persuasion techniques @xcite . with the advent of the transformer @xcite architecture and ensuing pre-trained large language models, such as bert @xcite and gpt-3 @xcite , the automatic detection of misinformation in online news has become feasible with high performance and hence an active area of research @xcite . the detection of propaganda techniques in news articles was in fact the goal of the semeval-2020 shared task 11 @xcite . based on the approach followed by the best participants at the semeval-2020 shared task 11 @xcite and the recent successes of transformer based pre-trained large language models, we based our system for the semeval-2023 shared task 3 @xcite on roberta @xcite . ## system overview the goal of the task is to identify all persuasion techniques (if any) employed at the paragraph-level in online news articles. therefore, our model takes as input an entire paragraph and classifies it as exhibiting none, one or multiple persuasion techniques as output. figure 1 shows an overview of the classification pipeline we employed for this subtask. as figure 1 shows, our classification system is composed of a roberta-base foot_1 model with a feedforward layer of 23 nodes, one for each of the annotated techniques. we used a sigmoid activation function at the output layer and we considered the entire task a 23-class multi-label binary classification problem. since the roberta-base model was pre-trained on the english language exclusively, for all the other training languages we first had to translate the datasets into english before feeding them to our model. we did this using the googletrans foot_2 api. we then augmented the datasets of each language (see section 4.2). note that for each language we could also have used a different roberta based model pre-trained on that specific language, but instead we opted to first translate the different languages into english and then use roberta-base. this decision was motivated by our experiments with the development set (see section 4.3). ## experimental setup before opting for our final classification system configuration, we experimented with different augmentation techniques, we considered different language models and we explored training our models with specific language datasets to classify the persuasion techniques across all languages. in this section we will describe in detail our experiments and the decision process that led to our final model. ## results based on the results of language baseline our score best score (see ## conclusion in this paper we described our approach to the semeval-2023 task 3 to detect online persuasion techniques in a multilingual setup. we applied a single classification system based on the robertabase model trained predominantly on the english language to detect persuasion techniques across 9 different languages. based on the results obtained with the development set, we assumed that our english trained model could generalize well to every other language and, hence, we did not develop language specific models. given the actual test set, the model was only able to significantly outperform the baseline for english and the two surprise, georgian and greek, for which no previous training data was available. in the future, we plan to explore the use of other language models, such as albert @xcite , and to evaluate how our final classification system would have performed if trained on the same language of the prediction task. we also plan on exploring a cascading model of classification systems for each persuasion technique on a one-vsall scenario. finally, given the different structure of the different languages, we would also like to investigate which techniques are more frequently classified correctly and which techniques are more frequently classified incorrectly on each language. this may lead to practical insights on how subjective information may be lost using automatic machine translation.