Findings of the Shared Task Multilingual Bias and Propaganda Annotation in Political Discourse (2025)

Abstract: The Multilingual Bias and Propaganda Annotation task focuses on annotating biased and propagandist content in political discourse across English and Tamil. This paper presents the findings of the shared task on bias and propaganda annotation task. This task involves two sub tasks, one in English and another in Tamil, both of which are annotation task where a text comment is to be labeled. With a particular emphasis on polarizing policy debates such as the US Gender Policy and India’s Three Language Policy, this shared task invites participants to build annotation systems capable of labeling textual bias and propaganda. The dataset was curated by collecting comments from YouTube videos. Our curated dataset consists of 13,010 English sentences on US Gender Policy, Russia-Ukraine War and 5,880 Tamil sentences on Three Language Policy. Participants were instructed to annotate following the guidelines at sentence level with the bias labels that are fine-grained, domain specific and 4 propaganda labels. Participants were encouraged to leverage existing tools or develop novel approaches to perform fine-grained annotations that capture the complex socio-political nuances present in the data.

## introduction social media platforms have become important medium for communication, enabling the widespread exchange and access to information from diverse sources @xcite . however, this open ecosystem is increasingly filled with harmful content, including various forms of misinformation such as propaganda, conspiracy theories, and biased narratives @xcite . the rapid scale and sophistication of such content demand solutions beyond manual fact checking @xcite . consequently, developing automated methods to detect and mitigate biased and propagandist content has become an urgent research priority @xcite . the term bias is defined as "bias is a disproportionate weight in favor of or against an idea or thing, usually in a way that is inaccurate, closed-minded, prejudicial, or unfair" @xcite . biases can be innate or learned @xcite . propaganda can take many forms, including political speeches, advertisements, news reports, and social media posts @xcite . its goal is usually to influence people's attitudes and behaviors, either by promoting a particular ideology or by persuading them to take a specific action @xcite . hence our task foot_0 , 2 , addresses the critical need for analyzing bias and propaganda in multilingual political discourse. developing annotation guidelines for complex data is a challenging task. in our tasks, we have identified the ideological bias related to support or against the government decisions on transgender rights, three language policies. we consider this as biases because it highly judgmental on sensitivity concerns. annotating such bias is crucial for understanding how regional and linguistic identities are influenced in political discourse @xcite . the purpose of this annotation is to examine how political narratives shapes the public opinion by favoring or attacking specific policies and identities. to overcome these limitations, annotation efforts should incorporate diverse human perspectives: involving annotators from multiple cultural and linguistic backgrounds has been shown to reduce bias and capture nuanced interpretation. addressing this challenge requires incorporating diverse perspectives and multi cultural insights during annotation, which can significantly enhance the robustness and fairness of nlp systems. ## related works understanding and detecting bias in political discourse has become main concerns in computational social science @xcite . earlier research have explored the linguistic features of biased content and propaganda tactics in news articles, speeches and online comments @xcite . @xcite analyzed linguistic pattern across different types of biased text, including fake news and political fact-checks. the work identified the subtle forms of bias through lexical and syntactical structures. @xcite offered a fine-grained taxonomy for identifying propaganda techniques in news articles. this work emphasized on detecting 18 specific propaganda techniques in news articles, such as appeal to fear, flag-waving, and loaded language. @xcite presented study on predicting the political ideology of news articles. with a comprehensive dataset of with 34,737 news articles yielded the model's robustness. the authors suggested novel modeling approaches, such as a specially modified triplet loss function and adversarial media adaptation to deal with propaganda tactics in cultural contexts. multilingual bias detection presented by @xcite created two large-scale datasets, mwik-ibias and mwnc in eight indian languages. the authors propose techniques for the detection of neutrality bias in politically and socially sensitive articles through models such as mdeberta and mt5. @xcite proposed a method for multi label propaganda detection using llm. the wanlp 2022 shared task, which called for recognizing several propaganda strategies in a single text, inspired the development of their system. they achieved a micro-f1 score of 59.73% by using an ensemble of five models to handle the complexity of detecting 21 different propaganda techniques. @xcite conducted fignews shared task as a component of the ara-bicnlp 2024 conference, which was held concurrently with acl 2024. this work used the early stages of the israel war on gaza as a case study to examine bias and propaganda annotation in multilingual news posts. their findings highlights the importance of clear guidelines and collaborative efforts in advancing nlp research on sensitive opinion analysis tasks. ## task description the shared task, addresses the crucial need for analyzing bias and propaganda in multilingual political discourse. this task aligns with the nlp community growing efforts to create datasets and guidelines for complex opinion analysis through collaborative shared tasks. there are two tasks in this shared task ## dataset description the dataset has been carefully curated from the youtube platform by collecting comments from videos discussing the political concern. to identify relevant videos, we utilized a combination of hashtags such as 'national education policy', 'three language policy', 'us gender policy', 'ukraine russia war' alongside manual keyword searches including terms like 'zelensky', 'putin' and 'trump us gender policy'. after gathering the comments, we applied preprocessing steps to remove unrelated or noisy content. this included removing usernames, urls, and comments containing fewer than three words to ensure data quality and relevance. the dataset statistics is shown in team scalar also developed a transformerbased nlp model to detect both propaganda techniques and bias using the annotated data. they trained the model using transfer learning on a specially annotated dataset label encoding bias categories and tagging six propaganda techniques while adding extra nonpropaganda examples to reduce class imbalance. the model was optimized with adam and trained for four epochs (batch size 32) using sparse categorical crossentropy, achieving roughly 47.9 % accuracy. ## conclusion this shared task aims to enhance the annotation process of bias and propaganda in multilingual political discourse, focusing on english and tamil texts. the shared task highlighted the role of clear guidelines, examples, and collaboration in advancing nlp research on complex, sensitive, and opinion analysis tasks. the resulting dataset and insights contribute valuable resources and direction for future work in this important area. despite limited submissions, the task underscored the challenges in multilingual annotation and the importance of culturally-informed guidelines. future efforts will focus on expanding the dataset, refining the annotation schema, and encouraging broader participation to build more generalizable models.