S em E val-2023 Task 3: Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multi-lingual Setup (2023)

Abstract: We describe SemEval-2023 task 3 on Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multilingual Setup: the dataset, the task organization process, the evaluation setup, the results, and the participating systems. The task focused on news articles in nine languages (six known to the participants upfront: English, French, German, Italian, Polish, and Russian), and three additional ones revealed to the participants at the testing phase: Spanish, Greek, and Georgian). The task featured three subtasks: (1) determining the genre of the article (opinion, reporting, or satire), (2) identifying one or more frames used in an article from a pool of 14 generic frames, and (3) identify the persuasion techniques used in each paragraph of the article, using a taxonomy of 23 persuasion techniques. This was a very popular task: a total of 181 teams registered to participate, and 41 eventually made an official submission on the test set.

## introduction the widespread use of internet and the advances in internet-related technologies paved the way to easily create direct communication channels between information producers and consumers, potentially leaving the latter exposed to manipulative, propagandistic, and deceptive content. given the potentially huge audience that can be reached through online channels, major public events and debates revolving around relevant topics could be influenced as a result. therefore, there is an ever-growing need to develop automated tools supporting experts in analysing the news ecosystem and identifying large-scale manipulation attempts, and facilitating the study of how different events, global topics, and policies are being embraced by media in various countries, in order to carry out cross-country analysis and to gather knowledge on the ways how media informs public opinion, i.e., what aspects are being highlighted and linked to a topic, what pros and cons are mentioned, the way opinions are conveyed, and what rhetorical devices, i.e., logical fallacies and appeal to emotions, are used to support flawed argumentation, potentially leading to manipulation. to foster research in this direction, there have been several shared tasks asking to detect the use of specific propaganda techniques in text, as well as the specific span of each instance. this includes the nlp4if-2019 shared task on fine-grained propaganda detection @xcite , semeval-2020 task 11 on detection of persuasion techniques in news articles @xcite , semeval-2021 task 6 on detection of persuasion techniques in texts and images @xcite , and wanlp-2022 shared task on propaganda detection in arabic @xcite . our task is an extension of the above ones and introduces several novelties. first, it is multilingual, covering nine languages. second, it adds additional dimensions for better news understanding, i.e., framing and news genre detection. finally, our taxonomy of persuasion techniques is an extension compared to previous inventories, and it contains 23 techniques organised in a two-tier hierarchy. ## the tasks the shared task comprises three subtasks: subtask 1 (st1): news genre categorization. given a news article, determine whether: (a) it is an opinion piece, (b) aims at objective news reporting, or (c) is satirical. 1 this is a multi-class classification task at the article level. subtask 2 (st2): framing detection. given a news article, identify one or more frames used in the article from a pool of 14 generic framing dimensions (introduced in @xcite ): economic, capacity and resources, morality, fairness and equality, legality, constitutionality and jurisprudence, policy prescription and evaluation, crime and punishment, security and defense, health and safety, quality of life, cultural identity, public opinion, political, external regulation and reputation. this is a multi-class multi-label classification task at the article level. subtask 3 (st3): persuasion techniques detection. given a news article, identify the persuasion techniques used in each paragraph of the article. the pool of persuasion techniques consists of 23 techniques, and is an extension of the taxonomy introduced @xcite ; @xcite 2 . this is a multi-class multilabel classification task at the paragraph level. the persuasion techniques are grouped into six main categories: attack on reputation: the argument does not address the topic, but rather targets the participant (personality, experience, deeds) in order to question and/or to undermine their credibility. the object of the argumentation can also refer to a group of individuals, an organization, an object, or an activity. justification: the argument is made of two parts, a statement and an explanation or an appeal, where the latter is used to justify and/or to support the statement. simplification: the argument excessively simplifies a problem, usually regarding the cause, the consequence or the existence of choices. distraction: the argument takes the focus away from the main topic or argument to distract the reader. call: the text is not an argument, but an encouragement to act or to think in a particular way. manipulative wording: the text is not an argument per se, but uses specific language, which contains words or phrases that are either non-neutral, confusing, exaggerating, loaded, etc., in order to impact the reader emotionally. figure 1 gives an overview of the two-tier persuasion techniques taxonomy. ## related work this section discusses prior work related to each of the subtasks of the shared task. ## the dataset this section provides a brief description of the dataset, whereas detailed guidelines, definitions and examples are provided in a separate technical report @xcite . we collected articles in nine languages: english, french, german, georgian, greek, italian, polish, russian, and spanish published in the period between 2020 and mid-2022, and revolving around various globally discussed topics, including the covid-19 pandemic, abortion-related legislation, migration, russo-ukrainian war, some local events such as parliamentary elections, etc. we considered both mainstream media and "alternative" media sources that could potentially spread mis-/disinformation. for the former, we used various news aggregation engines, e.g., google news foot_1 and europe media monitor foot_2 , etc., which cover sources with different political orientation, whereas for the latter, we used online services such as mediabias-factcheck foot_3 and newsguard. 6 we extracted the article texts either using trafilatura @xcite or, in few cases, ad hoc procedures. we annotated each text for genre, framing, and persuasion techniques using the taxonomy described in section 2. while genre and framing were annotated at the document level, we annotated the persuasion techniques at the span level. we had about 40 annotators, who were either media analysts, disinformation specialists or nlp experts, most of which had prior experience in performing linguistic annotations. all annotators were either native or near-native speakers of the language they annotated for. we used the inception @xcite platform for carrying out the annotations. the annotation interface for an example document using inception is shown in figure 2 . as regards english, we exploited the texts from @xcite , but the annotations for persuasion techniques have been slightly modified in order to match the extended taxonomy, most notably whataboutism included two meanings: distracting from the main argument and calling the hypocrisy of the speaker; the latter meaning is now covered by the technique appeal to hypocrisy. moreover, we added annotations for framing and news genre. each document was annotated by at least two annotators. once the individual annotations for a document have been accomplished, a curator (an experienced annotator) with the help of annotators consolidated the final annotation. the consolidation consisted of: (a) merging complementary annotations (tagged only by one annotator), (b) deciding whether overlapping annotations are to be kept as they are (multi-labels) or joined into a singlelabeled annotation, and (c) carrying out global consistency analysis. the detailed description of the annotation and the consolidation process are described in a detailed technical report @xcite . in order to assess the annotation quality, we computed the inter-annotator agreement (iaa) using krippendorf's α: the value was 0.342, which is lower than the recommended threshold of 0.667, but we should note that this value represents the agreement before the consolidation, and as such, it is more representative of the consolidation difficulty rather than of the quality of the final consolidated annotations. actually, we used iaa to allocate consolidation roles and to eliminate low-performing annotators. we further studied the iaa by ranking the annotators by their performance with respect to the ground truth on the subset of documents they annotated. we then split them into two groups, top and low, based on the median micro-f 1 scores. their respective α scores were 0.415 and 0.250. finally, we considered the value of α of the group of annotators, based on italian, the only language with two curators, achieving 0.588, which is lower but close to the recommended value. the annotated data, consisting of 2,049 documents in total, were divided into train, dev, and test datasets, whose high-level statistics are provided in ## conclusions and future work we presented semeval-2023 task 3 on detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup. the task attracted a lot of attention: 181 teams registered for the task, 41 teams eventually madevan official submission on the test set, and 32 teams also submitted a task description paper. in future work, we plan to further increase the data size, cover additional languages, and explore different ways of evaluation of the persuasion technique detection, e.g., by changing the focus (sentence-and text span-level evaluation). team name transformers other representations additional data data augmentation ensembles preprocessing trained on all languages knowledge base chunk processing data translation biases human data annotation involves some degree of subjectivity. to mitigate this, we created a comprehensive 60-page guidelines document, which we updated from time to time to clarify newly arising important cases during the annotation process. we further had quality control steps in the data annotation process, and we have been excluding low-performing annotators. despite all this, we are aware that some degree of intrinsic subjectivity will inevitably be present in the dataset and will eventually be learned by models trained on it.