Unisa at S em E val-2023 Task 3: A SHAP -based method for Propaganda Detection (2023)

Abstract: This paper presents proposed solutions for addressing two subtasks in SemEval-2023 Task 3: “Detecting the Genre, the Framing, and the Persuasion techniques in online news in a multi-lingual setup. In subtask 1, “News Genre Categorisation, the goal is to classify a news article as an opinion, a report, or a satire. In subtask 3, “Detection of Persuasion Technique, the system must reveal persuasion techniques used in each news article paragraph choosing among23 defined methods. Solutions leverage the application of the eXplainable Artificial Intelligence (XAI) method, Shapley Additive Explanations (SHAP). In subtask 1, SHAP was used to understand what was driving the model to fail so that it could be improved accordingly. In contrast, in subtask 3, a re-calibration of the Attention Mechanism was realized by extracting critical tokens for each persuasion technique. The underlying idea is the exploitation of XAI for countering the overfitting of the resulting model and attempting to improve the performance when there are few samples in the training data. The achieved performance on English for subtask 1 ranked 6th with an F1-score of 58.6% (despite 78.4% of the 1st) and for subtask 3 ranked 12th with a micro-averaged F1-score of 29.8% (despite 37.6% of the 1st).

## introduction recent technology improvements undoubtedly promote freedom of speech but leave society defenseless against potential news tampering @xcite . the main drive behind the semeval 2023 task 3 @xcite is to foster the development of methods and tools to support the analysis of online media content to understand what makes a text persuasive: which writing style is used, what critical aspects are highlighted, and which persuasion techniques are used to influence the reader. persuasion attempts and propaganda (through different ways), appeal to the user's sentiment to influence his/her opinions. unfortunately, detecting propaganda and persuasion techniques are trending research topics that still perform poorly, especially when labeled data is limited @xcite . our approach tries to overcome problems related to the limited number of training instances by generalizing data through the shapley additive explanations (shap) (lundberg and lee, 2017) method, an explainable artificial intelligence (xai) technique. shap is a framework trying to explain the prediction of an instance x by computing the contribution of each feature to the prediction. the shap explanation technique uses coalitional game theory to compute shapley values. in subtask 1, shap is employed for feature selection @xcite . the idea is to leverage features to understand how to pre-process documents before sending them to the model. in subtask 3, shap re-calibrates the attention mechanism for each persuasion technique. the idea consists of identifying words characterizing each type of persuasion and helping the learning model to focus on them. the paper is organized as follows. section 2 gives an overview of existing approaches and theories, and section 3 describes our solutions. sections 4 and 5 detail experimentation, and finally, section 6 concludes the manuscript. ## background in recent years, deep learning-based methods have become increasingly popular in natural language processing and information retrieval. transformers are a type of deep learning architecture that has gained popularity in natural language processing tasks, especially for tasks involving contextualized word embedding. the news genre categorization task includes satire detection, which is also approached using transformer architectures @xcite @xcite . in (alhindi et al., 2020; blackledge and atapour-abarghouei, 2021) authors present the classification of opinion-based news articles to finally classify fact or fake news pieces. regarding persuasion detection, most of the works in the literature focus on text analysis. iyer et al. leveraged patterns identified for each considered persuasion technique @xcite . hidey et al. demonstrated the ordering of arguments is crucial to persuasion @xcite . similarly, an lstm-based approach for persuasion detection in social media conversations is proposed in @xcite . persuasion detection has been a goal of recent semeval competitions @xcite . inspired by @xcite , the system implementing subtask 3 proposes a bert-based text classification model via constructing auxiliary sentence to turn the classification task into a binary sentence-pair one, aiming to address the limited training data problem and task-awareness problem. to the best of our knowledge, this is the first attempt to employ an xai-based approach for persuasion techniques identification. ## system overview the system adopts the distilbert transformer @xcite and the shap method. the following subsections detail how they are combined to address the subtasks 1 and 3 included in semeval-2023 task 3: "detecting the genre, the framing, and the persuasion techniques in online news in a multi-lingual setup". more in detail, subsection 3.1 points out that for subtask 1, shap was used to understand what tokens drive the classification of the article. in contrast, in subsection 3.2, for subtask 3, shap was adopted to re-calibrate the attention mechanism by leveraging words that distinguish each persuasion technique and help the model to generalize better. ## experimental setup methodologies described in section 3, have been applied to the dataset provided by the semeval-2023 task 3 organizers, specifically for the english subtask. the following subsections describe the experimentation setups and results for both considered subtasks. in particular, results have been evaluated through official measures: macro-f1 and micro-f1 for subtasks 1 and 3, respectively, as described in @xcite . furthermore, the number of epochs used in each subtask was defined with a maximum of 6 as we train lms; if no improvements are made, fewer epochs are used. ## results in this section, obtained results for both considered subtasks are described. ## conclusion this work derives from the contribution of our team to the semeval-2023 task 3: "detecting the genre, the framing, and the persuasion techniques in online news in a multi-lingual setup". specifically, it faces english subtasks 1 and 3. in the first case, the objective is to classify a news article as an opinion, a report, or a satire; in the second case, to identify one or many persuasion techniques in each paragraph of a given article. proposed solutions leverage the shapley additive explanations (shap) method. in particular, in subtask 1, shap was used as a feature detection method by filtering tokens bring to wrong classifications, while in subtask 3, a re-calibration of the attention mechanism (typically implemented by transformer models) was realized by extract- the objective is to exploit explainable artificial intelligence techniques to improve classification performance when the availability of labeled data for training is scarce through a generalization attempt. experimental results demonstrate promising, encouraging further investigation in this sense. in particular, in the future, regarding subtask 3, it could be interesting to better address the shap vocabularies construction by adding spans contain-ing the persuasion attempt to the adopted binary classifiers. this could allow focusing on a specific substring of the paragraph, improving the classification performance of the single binary classifiers.