KI n ITV era AI at S em E val-2023 Task 3: Simple yet Powerful Multilingual Fine-Tuning for Persuasion Techniques Detection (2023)

Abstract: This paper presents the best-performing solution to the SemEval 2023 Task 3 on the subtask 3 dedicated to persuasion techniques detection. Due to a high multilingual character of the input data and a large number of 23 predicted labels (causing a lack of labelled data for some language-label combinations), we opted for fine-tuning pre-trained transformer-based language models. Conducting multiple experiments, we find the best configuration, which consists of large multilingual model (XLM-RoBERTa large) trained jointly on all input data, with carefully calibrated confidence thresholds for seen and surprise languages separately. Our final system performed the best on 6 out of 9 languages (including two surprise languages) and achieved highly competitive results on the remaining three languages.

## introduction the subtask 3 of the semeval 2023 task 3 aims at identifying persuasion techniques. the task is a multi-label one, where a model is required to identify which of the 23 persuasion techniques (e.g., an appeal to authority) are present in a given paragraph. the paragraphs are obtained from articles in 6 languages @xcite collected between 2020 and mid 2022, revolving around widely discussed topics such as covid-19, climate change, abortion, migration etc. media sources are both mainstream and alternative news and web portals. furthermore, the model is tested on 3 surprise languages @xcite , for which labeled training data were not available. the importance of the task is eminent -automatically detected persuasion techniques can be utilized as credibility signals to assess content credibility and thus also to improve disinformation detection. the detailed description of the task is available in @xcite . in this paper, we propose a multilingual system, consisting of a single model tackling all languages. our main strategy is to fine-tune a large pre-trained transformer-based language model. to find the best performing system, we experimented with different language models (and finally opted for xlm-roberta large due to its performance), hyperparameter tuning as well as confidence threshold calibration by changing the threshold for prediction in the multi-label classification. we also simulated the zero-shot setting on the training data to adjust the confidence threshold and better estimate the performance of our model on the surprise languages. furthermore, we experiment with additional configurations, such as translating the data to a single language (english) and using it to fine-tune a monolingual model, applying various text pre-processing strategies, or layer freezing. however, these configurations did not lead to improvements. although our system is based on a rather simple concept, it still achieved exceptional results. we ranked 1st for 6 languages (italian, russian, german, polish, greek and georgian), 2nd for the spanish, 3rd for the french and 4th for the english language. in the zero-shot setting introduced by unseen languages, our system also performs exceptionally, achieving the best performance on two languages (greek and georgian) and second on the remaining unseen language (spanish). based on our experiments and official ranking on the test set, we make the following observations: together with the system description, we also release its source code foot_0 , as well as the fine-tuned model used for submitting the final results foot_1 . ## background the train and dev sets provided by the organizers consisted of 26 663 samples in 6 languages. each sample consist of a paragraph of news article and zero, one or multiple persuasion techniques (out of 23 possible classes) present in such a paragraph (with the exact span identified). by performing exploratory data analysis of the provided dataset, we observed a high data imbalance in both classes (persuasion techniques) as well as languages (some combinations of classes and languages contain no samples at all) -see appendix a. the research on the computational propaganda/persuasion techniques detection is still in its early stages, despite its potential importance and utilization for credibility evaluation or disinformation detection @xcite . many existing works closely relate to the semeval tasks in 2020 @xcite and 2021 @xcite , which preceded the current se-meval 2023 task 3. the approaches evolved from a simple binary classification of propaganda being present in a document @xcite , through a fine-grained detection of 18 propaganda techniques @xcite to detection of 23 persuasion techniques introduced in this task. moreover, while the methods proposed so far are trained solely on monolingual data, the introduced multilingual data allows to research true multilingual approaches. ## system overview the main principle used for development of our system is fine-tuning of a large language model using the data provided within the semeval task. in similar fashion to other fine-tuning approaches, we add a classification layer at the end of the pretrained model, while also including a dropout layer to prevent overfitting. as input, the language model takes the paragraph, potentially truncated if its length is higher than the maximum input size. no other processing of input is performed (i.e., we are working on paragraph level only). as the task is a multi-class multi-label problem, the predicted output label is not determined based on the maximum probability, but instead by specifying a confidence threshold. all classes that have their probability higher than this confidence threshold are predicted as output label. to develop the best configuration of the language model fine-tuning solution, we performed multiple experiments that can be organized into following five steps (which are summarized in figure 1 ): the final solution utilizes a single multilingual model fine-tuned on all languages at once, with slightly lowered confidence threshold, with no preprocessing and no layer freezing. ## experimental setup the only data we use for our system is the official dataset provided for the task @xcite . we also use the default training-development split provided for the task. during the development of our solution, we use the development set only for evaluating the different steps and experimental configurations. for the final submission, the language model is fine-tuned on both sets of data. for evaluation purposes we use the f1 micro score, which is the default for this subtask, even though it emphasizes the majority classes over the minority ones. the different pretrained language models used in our system are chosen from the ones available at hugging face. we use the pytorch deep learning library, version 1.13.1. we have also created a custom pipeline for efficient combining paragraphs and their labels from all articles into a single object, for running all the preprocessing and data translation, and the training of the models. for each language model, we add a dropout layer with dropout rate of 0.3, followed by a classification layer with output size of 23 (one per persuasion technique). before all the experiments, we perform a hyperparameter optimisation for each language model, mainly focusing on the number of epochs, batch size and learning rate. as starting point for the hyperparameters, we use the values that were determined to perform well in related work (e.g., @xcite ) and then searched close to these values. the best hyperparameters used for all the language models are: batch size of 16, adam optimizer with 1e -05 learning rate and fine-tuning for 5 epochs with early stopping using cross-entropy loss. ## results the same time, the nuances needed to correctly detect the persuasion techniques may be lost in translation. however, we can see that both monolingual and multilingual models have their own strengths and weakness, as their ensemble produces the best results overall. we can also observe significant impact of the confidence threshold calibration. although the ensemble of monolingual and multilingual models performs best without the calibration, the multilingual model quickly outperforms the ensemble when the best confidence threshold is used for all models (confidence threshold of 0.29). finally, we observe that impact of preprocessing has negligible impact on the overall performance, achieving similar, although slightly lower f1 score. on the other hand, the different layer freezing strategies have significant negative effects on the model, lowering the performance by ∼ 12%. ## conclusion in this paper, we have presented the implementation of the solution proposed by kinitveraai team for the subtask 3 within the semeval 2023 task 3. our rather simple, yet powerful, solution utilizes fine-tuning of multilingual language model. in a challenging multi-label task with 23 classes, it achieves very promising performance (f1 micro) of 36-55% for languages seen during the training, and 26-45% for unseen languages (zero-shot setting). in future, we plan to investigate the potential of prompting and in-context learning on the top of large pre-trained language models (like gtp-3 or chatgpt). our hypothesis is that the large size of these models may allow even deeper understanding of the input text. nevertheless, it will be critical to design appropriate prompts as a part of prompt engineering process, address a potential bias towards majority classes, and also overcome well-known issues with instability of these approaches.