Hierarchical Multi-Instance Multi-Label Learning for Detecting Propaganda Techniques (2023)

Abstract: Since the introduction of the SemEval 2020 Task 11 (CITATION), several approaches have been proposed in the literature for classifying propagandabased on the rhetorical techniques used to influence readers. These methods, however, classify one span at a time, ignoring dependencies from the labels of other spans within the same context. In this paper, we approach propaganda technique classification as aMulti-Instance Multi-Label (MIML) learning problem (CITATION) and propose a simple RoBERTa-based model (CITATION) for classifying all spans in an article simultaneously. Further, we note that, due to the annotation process whereannotators classified the spans by following a decision tree,there is an inherent hierarchical relationship among the differenttechniques, which existing approaches ignore. We incorporate these hierarchical label dependencies by adding an auxiliary classifier for each node in the decision tree to the training objective and ensembling the predictions from the original and auxiliary classifiers at test time. Overall, our model leads to an absolute improvement of 2.47% micro-F1 over the model from the shared task winning team in a cross-validation setup and is the best performing non-ensemble model on the shared task leaderboard.

## introduction the development of the web and social media has amplified the scale and effectiveness of propaganda @xcite . automatic propaganda detection through text analysis enables social science researchers to analyze its spread at scale @xcite . within text analysis, two broad approaches include the identification of rhetorical techniques used to influence readers and document-level propagandistic article classification. the former is more promising because propaganda techniques are easy to identify and are the very building blocks @xcite . hence, it is the focus of this paper. recent research on fine-grained propaganda detection has been spurred by the nlp4if-2019 shared task @xcite and its follow-up semeval 2020 task 11 @xcite . in this paper, we focus on its technique classification subtask where, given a news article text and spans identified as propagandistic, systems need to classify each of the spans into one or more of 14 different common propaganda techniques. all top systems submitted to the task @xcite @xcite employed a pretrained roberta-based model @xcite which was trained to classify one span at a time. however, labels of different spans within the same article clearly depend on each other. thus, we approach the task within a multi-instance multi-label framework @xcite , where we model each article as an object with multiple instances (spans), each with its own labels. this allows us to model the dependencies between different labels within the same article. we show that this miml roberta observes a 1.98% micro-f1 improvement over the replicated applicaai system (referred to as baseline) @xcite . besides, as a decision tree was used to guide annotations @xcite , we explore incorporating this hierarchical relationship among the labels into classifiers. to do so, we add 7 more auxiliary classifiers on top of the span representations from roberta, one for each intermediate node in the tree, and train these classifiers to predict the path to a leaf node and hence the corresponding label (see figure 1 ). we show that incorporating the label hierarchy in this manner improves both the single-instance and the miml versions of the roberta model (referred to as hierarchical baseline and hierarchical miml roberta respectively). ## related work the miml framework was first introduced by @xcite aiming for better representing complicated objects composed of multiple instances, e.g., an image with several bounding boxes each with its own label. since then, it has been applied to many tasks, such as relation extraction @xcite and aspect-category sentiment analysis @xcite . the latter work uses a bi-lstm architecture which aggregates over the words in a sentence (instances) to classify the sentiments of different aspects (labels). miml has also been applied to bert-based models in the biomedical text analysis @xcite . to our best knowledge, our work is the first one to apply it for propaganda technique classification. given the decision tree used for annotations (reproduced in figure 4 in appendix), this task can also be viewed as a hierarchical text classification problem, with mandatory leaf node prediction and a tree-structured hierarchy @xcite . exploiting hierarchical information has been useful in significantly enhancing the performance of the system for medical image annotations @xcite and presents us with an opportunity to apply to the propaganda method detection. similar to @xcite , we use the multiplicative rule to combine probabilities along the different paths in the hierarchy, leading to a distribution across the leaf nodes which can be combined with the distribution predicted by the non-hierarchical baseline. different approaches to hierarchical classification across multiple domains can be found in @xcite . ## methods 1 task description. given an article d with propagandistic spans s 1 , ..., s m identified by their start and end indices, the task is to identify the sets of techniques y 1 , ..., y m where y i ⊆ y ∀@xmath0, ..., m} and y represents the set of 14 techniques. following the shared task, we assume that the number of labels |y i | for each span is known. ## conclusion we propose two simple extensions to a robertabased model for propaganda technique classification, which lead to notable improvements. our approach to incorporating hierarchical information about the labels into training could also be useful for other tasks where the annotation procedure involves making a series of decisions about instances. future work can also explore other methods to incorporate the hierarchical information, e.g., via regularizing the label embeddings.