ACCEPT at S em E val-2023 Task 3: An Ensemble-based Approach to Multilingual Framing Detection (2023)

Abstract: This paper describes the system and experimental results of an ensemble-based approach tomultilingual framing detection for the submission of the ACCEPT team to the SemEval-2023 Task 3 on Framing Detection (Subtask 2). The approach is based on an ensemble that combines three different methods: a classifier based on large language models, a classifier based on static word embeddings, and an approach that uses external commonsense knowledge graphs, in particular, ConceptNet. The results of the three classification heads are aggregated into an overall prediction for each frame class. Our best submission yielded a micro F1-score of 50.69% (rank 10) and a macro F1-score of 50.20% (rank 3) for English articles. Our experimental results show that static word embeddings and knowledge graphs are useful components for frame detection, while the ensemble of all three methods combines the strengths of our three proposed methods. Through system ablations, we show that the commonsenseguided knowledge graphs are the outperforming method for many languages.

## introduction the semeval 2023 task 3 by @xcite (subtask 2) consisted in detecting different types of frames in newspaper articles in six different languages (english, french, german, italian, polish, and russian). beyond these seen languages, three more languages were added in the testing phase @xcite . different framings represent different perspectives on one and the same reported event. being able to categorize them helps to cluster such news articles and guides the analyses of different aspects @xcite . in the particular task at hand, the applied generic mediaframes set @xcite categorizes articles into 14 frame classes: "economic", "capacity and resources", "morality", "fairness and equality", "legality, constitutionality and jurisprudence", "policy prescription and evaluation", "crime and punishment", "security and defense", "health and safety", "quality of life", "cultural identity", "public opinion", "political" and "external regulation and reputation". in this paper, we describe the system and experimental results for submitting team accept to the shared task. our approach consists of an ensemble of three different methods, each relying on a fully connected classification layer on top of i) a large language model (lm), ii) static word embeddings and iii) a graph neural network (gnn) architecture exploiting a commonsense knowledge graph (cskg), conceptnet in particular. to handle inputs in multiple languages, as required by the task, we explore language-specific and languageagnostic methods, and determine the best configuration for each language. as an interesting result, we found that the overall best configuration is language-specific by using commonsense knowledge graphs for inference, while lms do not perform well without further guidance. we publish our highly adjustable ensemble framework, including the three proposed methods at https://github.com/phhei/ semeval233framingpublic . ## related work the challenge of detecting and tracing frames in newspaper articles goes back to seminal work of @xcite and @xcite , who provided a dataset with articles annotated with frames. one of the first automatic approaches to inferring frames for articles was proposed by @xcite , who relied on recurrent networks based on static word embeddings. this architecture performed with an accuracy of up to 71% in distinguishing between the five most frequent mediaframes on the text-span level. building on these first encouraging results, the task of frame detection also received attention in the field of argument mining @xcite and even aspect-tailored language generation with pretrained large language models @xcite that were also successfully applied for frame classification @xcite . beyond exploiting text only, some methods in argument mining have integrated external knowledge and found it to improve performance on tasks such as the prediction of the quality of argumentative conclusions @xcite , or the prediction of sentiment-related human needs categories following the hierarchy of human needs @xcite and basic motives @xcite , in the analysis of narratives @xcite . however, to the best of our knowledge, the inclusion of background knowledge for framing prediction, as proposed in our work, has not been investigated before. inspired by the success of combining different approaches to the task of frame detection @xcite and related shared tasks @xcite @xcite , we experiment with various method selections and hyperparameter settings to explore advantages and disadvantages across different methods for detecting frames in newspaper texts. ## dataset we used the provided dataset for the shared task by @xcite , containing 2,069 propaganda or political news article published between 2020 and 2022, split into 1,251 articles for training, 361 articles for development, and 457 for training. each article is annotated with 1-10 out of 14 different frame classes. while the annotators select some frame classes frequently (e.g. "political" and "security and defense"), other frame classes are about four times less selected (e.g.: "cultural identity" and "public opinion"). the articles are multi-lingual, including six languages in all splits (english, french, german, italian, polish, and russian) and three languages (spanish, greek, and georgian) only in the test split. the average number of tagged frames classes per article differs across languages (e.g., 1.2 frame classes per russian article and 5.9 frame classes per polish article, both averaged on the test split). 4 system overview we present an ensemble approach that combines three different multilabel classification models to predict the frames of a given news article. the methods are based on large language models (lms), static word embeddings (we) and gnns using commonsense knowledge subgraphs (kg), as illustrated in figure 1 . they are described in detail below. ## experimental setup and results we participated in subtask 2: framing detection and experimented with various configurations. our ensemble framework is implemented in python 3.9 based on the pytorch-library @xcite . in training, we used a standard learning rate of 5e-4 or 5e-5 in case of lm-based method. we trained up to eight epochs using an early-stopping criterion to load the best-performing checkpoint with respect to the product of micro-f 1 and macro-f 1 on the development data. to aggregate the frame predictions of our ensemble, we average the predictions made by the applied methods. additionally, in the case of our final submissions, we experimented with weighting the frame prediction of each method by introducing a trainable weight vector. in inference, we determine optimal probability thresholds to decide when to select a frame as observable in an article. we compute one threshold for each frame class separately using the development data as well. however, to avoid an empty or unrealistically large set of finally predicted frames, we enforce selecting between 1 and 8 frames per article based on a ranked list of frames, sorted by their prediction likelihood. ## conclusion predicting the occurring frame classes in long multilingual articles is challenging due to the variety of articles and differences between languages and cultures. this paper describes our participation in the semeval-2023 task 3 for which we proposed an ensemble combining three different methods (large language models, static word embeddings, and knowledge graphs). in order to handle inputs in all relevant languages, we leveraged machine translation software as a preprocessing step or alternatively relied on pre-trained multilingual language models that map texts into a language-agnostic semantic space. ablation analysis of our ensemble suggests that using a graph neural network on extracted commonsense knowledge subgraphs provides the strongest results for most of the nine languages. this shows the strength of knowledge-based methods for frame detection in multilingual settings. guillaume lample, alexis conneau, marc'aurelio ranzato, ludovic denoyer, and hervé jégou. 2018. word translation without parallel data. in 6th international conference on learning representations, iclr 2018, vancouver, bc, canada, april 30 -may 3, 2018, conference track proceedings. openreview.net. xiang luo, yanqing niu, and boer zhu. 2022. tcu at semeval-2022 task 8: a stacking ensemble transformer model for multilingual news article similarity. in proceedings of the 16th international workshop on semantic evaluation (semeval-2022), pages 1202-1207, seattle, united states. association for computational linguistics. abraham harold maslow. 1943. a theory of human motivation. psychological review, 50(4):370. nona naderi and graeme hirst. 2017. classifying frames at the sentence level in news articles. in proceedings of the international conference recent advances in natural language processing, ranlp 2017, pages 536-542, varna, bulgaria. incoma ltd. adam paszke, sam gross, francisco massa, adam lerer, james bradbury, gregory chanan, trevor killeen, zeming lin, natalia gimelshein, luca antiga, alban desmaison, andreas kopf, edward yang, zachary devito, martin raison, alykhan tejani, sasank chilamkurthy, benoit steiner, lu fang, junjie bai, and soumith chintala. 2019. pytorch: an imperative style, high-performance deep learning library. in advances in neural information processing systems 32, pages 8024-8035. curran associates, inc. debjit paul and anette frank. 2019. ranking and selecting multi-hop knowledge paths to better predict human needs. in proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 3671-3681, minneapolis, minnesota. association for computational linguistics. jeffrey pennington, richard socher, and christopher manning. 2014. glove: global vectors for word representation. in proceedings of the 2014 conference on empirical methods in natural language processing (emnlp), pages 1532-1543, doha, qatar. association for computational linguistics. jakub piskorski, nicolas stefanovitch, giovanni da san martino, and preslav nakov. 2023. semeval-2023 task 3: detecting the category, the framing, and the persuasion techniques in online news in a multi-lingual setup. in proceedings of the 17th international workshop on semantic evaluation, semeval 2023, toronto, canada. moritz plenz, juri opitz, philipp heinisch, philipp cimiano, and anette frank. 2023. similarity-weighted construction of contextualized commonsense knowledge graphs for knowledge-intense argumentation tasks. in proceedings of the 61th annual meeting of the association for computational linguistics, toronto, canada. association for computational linguistics. mayank raj, ajay jaiswal, rohit r.r, ankita gupta, sudeep kumar sahoo, vertika srivastava, and yeon hyang kim. 2020. solomon at semeval-2020 task 11: ensemble architecture for fine-tuned propaganda detection in news articles. in proceedings of the fourteenth workshop on semantic evaluation, pages 1802-1807, barcelona (online). international committee for computational linguistics. nils reimers and iryna gurevych. 2019. sentence-bert: sentence embeddings using siamese bertnetworks. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 3982-3992, hong kong, china. association for computational linguistics. nils reimers and iryna gurevych. 2020. making monolingual sentence embeddings multilingual using knowledge distillation. in proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pages 4512-4525, online. association for computational linguistics. steven reiss. 2004. multifaceted nature of intrinsic motivation: the theory of 16 basic desires. review of general psychology, 8(3):179. mattes ruckdeschel and gregor wiedemann. 2022. boundary detection and categorization of argument aspects via supervised learning. in proceedings of the 9th workshop on argument mining, pages 126-136, online and in gyeongju, republic of korea. international conference on computational linguistics. benjamin schiller, johannes daxenberger, and iryna gurevych. 2021. aspect-controlled neural argument generation. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 380-396, online. association for computational linguistics. michael schlichtkrull, thomas n kipf, peter bloem, rianne van den berg, ivan titov, and max welling. 2018. modeling relational data with graph convolutional networks. in the semantic web: 15th international conference, eswc 2018, heraklion, crete, greece, june 3-7, 2018, proceedings 15, pages 593-607. springer. robyn speer, joshua chin, and catherine havasi. 2017. conceptnet 5.5: an open multilingual graph of general knowledge. in proceedings of the thirty-first aaai conference on artificial intelligence, aaai'17, page 4444-4451. aaai press.