H oward U niversity- AI 4 PC at S em E val-2025 Task 10: Ensembling LLM s for Multi-lingual Multi-Label and Multi-Class Meta-Classification (2025)

Abstract: This paper describes our approach and submission to the SemEval 2025 shared task on “Multilingual Characterization and Extraction of Narratives from Online News”. The purpose of this task was to assign primary and fine-grained roles to named entities in news articles from five different languages, on the topics of Climate Change and Ukraine-Russia War. In this paper, we explain how we approached the task by utilizing multiple LLMs via Prompt Engineering and combining their results into a final task result through an ensemble meta-classification technique. Our experimental results demonstrate that this integrated approach outperforms the provided baseline in detecting bias, deception, and manipulation in news media across multiple languages.

## introduction today, there is unprecedented access to information for audiences, largely due to direct channels between content producers and audiences. however, this also exposes readers to deception and manipulation, particularly during crisis events. to address these challenges and foster research on analytical tools for text analysis and disinformation detection @xcite @xcite , semeval 2025 task 10 is aptly titled multilingual characterization and extraction of narratives from online news. this task involves automatically identifying narratives, classifying them, and determining the roles of key entities. subtask 1 of the shared task focuses specifically on entity framing, which this paper and our experiments tackled. the provided dataset comprises news articles in five languages-english, portuguese, russian, bulgarian, and hindi-and covers two domains: the ukraine-russia war and climate change @xcite . entity framing is formulated as a multi-label, multi-class classification problem in which each entity mention is evaluated based on a fine-grained taxonomy that distinguishes between roles such as protagonists, antagonists, and innocents, and further differentiates each entity into specific finegrained roles corresponding to its primary classification @xcite . this subtask challenges models to handle multilingual content and subtle contextual cues. in our work, we use a diverse set of advanced language models: llama3.1 (8b parameters), mistral (7b parameters), phi4 (14b parameters) and gemma2 (9b parameters). their predictions are combined using an ensemble meta-classification strategy to yield robust, consistent role assignments. through this integrated approach, our objective is to demonstrate the effectiveness of llms in detecting bias, deception, and manipulation in the news media. multiple authors have provided a broad evaluation of llm-based text classification methods across various datasets and languages, comparing zero-shot, few-shot, and synthetic data approaches @xcite @xcite . although their work primarily targets general text classification rather than news entity framing, their findings on performance disparities across languages and the benefits of synthetic data generation are highly relevant. their study underscores the challenges of working with limited labeled data and complex multilingual settings, issues that our work also addresses through prompt engineering with advanced llms for entity role classification in news. ## system overview the system begins with a robust data preprocessing module that ingests news articles from the dataset in multiple languages: english, portuguese, russian, bulgarian, and hindi, and extracts entity mentions along with the article texts. each extracted entity is converted into a structured dictionary containing key metadata such as article id, article text, and character indices. this structured representation is then passed to our classification engine. the classification engine employs an llm of choice, through the ollama api. each model is prompted with a carefully engineered instruction set that enforces strict classification rules, ensuring that every entity is assigned one primary role (protagonist, antagonist, or innocent) along with corresponding predefined fine-grained roles chosen from the provided lists. given the variability in model outputs, we observed that a single model could occasionally generate erroneous tokens or roles not included in the prompt. to counteract this, our pipeline includes a cleaning step that filters out irrelevant characters and incorrect classifications. the final stage of our system is a metaclassification process. here, we aggregate the predictions of all llms we used and use phi4 to merge these outputs into a consensus classification. it is important to note that our initial system used only llama3.1:8b, but was later upgraded to this ensemble method. it emphasizes role frequency and consistency across models, proving effective at boosting overall performance compared to a singlemodel baseline. the resulting outputs are then reformatted into tab-separated files for submission, ensuring both readability and compliance with the shared task requirements. foot_0 ## experimental setup our experiments were carried out in a cloud-based google colab environment, which provided a v2-8 tpu runtime with 334.6 gb of system ram and 225.3 gb of disk storage. this configuration was vital for handling the computational demands of processing multilingual news articles and running multiple large language models (llms). training data provided by semeval 2025 task 10 organizers was stored on google drive with full permissions, allowing seamless access and file management during experiments. ## evaluation the initial single-model approach, which solely relied on llama3.1:8b and was used for processing english articles, performed worse than the baseline model. however, by integrating outputs from multiple llms through this ensemble metaclassification strategy, we observed a significant enhancement in performance. the ensemble approach effectively consolidated individual predictions, leading to more robust and consistent classifications across various languages. however, each llm generated irrelevant roles or erroneous characters for multiple entities, at every iteration. for example, the most frequently generated erroneous characters include ':', ']', ' @xcite similarly, the most frequently generated irrelevant roles were 'propagandists' and 'aggressor'. in case of deepseek-r1, the results were ex-tremely unreliable, with multiple instances of entities not being classified at all or erroneous characters populating the fields for primary and/or finegrained roles. for example, a correctly formatted entity role assignment versus deepseek-r1 results can be compared at ## limitations although our system demonstrates an improved performance over the baseline, we must acknowledge the several limitations in our system and solution. due to resource constraints, we opted for llms that primarily were compatible with our available hardware and time limitations. this decision prevented us from experimenting with larger models that might have yielded even more accurate classifications with better overall metrics compared to the baseline. we were also unable to test, observe, and improve the metrics of each llm separately for the subtask for the same reasons, hence resorting to ensemble meta-classification from the get-go. our experiments were conducted using limited paid google colab compute units, which provided the necessary computational power for heavy inference throughout. however, our reliance on highperformance resources for this system may limit the reproducibility of our results for researchers with less access to such hardware and resources. our approach focused primarily on prompt engineering to guide model outputs, without exploring additional methods such as fine-tuning or retraining on the provided datasets. while prompt engineer-ing proved effective, further improvements might be achieved through more advanced training techniques, which we were not able to pursue in this work due to resource and time constraints. lastly, our system also poses some inherent risks, including the possibility of algorithmic bias in the llms we have employed and the ethical implications of automated content evaluation. ## conclusion in this work, we presented a comprehensive approach to how we utilized multiple llms in a unified system for multilingual entity framing. our experiments on the semeval 2025 task 10 subtask 1 dataset demonstrate that a set of modelscombined with targeted prompt engineering and rigorous output cleaning -can achieve substantially higher performance. while our results do indicate promising improvements in metrics such as exact match and micro f1, our system also highlights critical limitations. resource constraints, hardware limitations, and the inherent challenges of prompt engineering for structured output still remain significant hurdles towards replicability. future research should explore the capabilities of each of the llms used individually and also experiment with advanced fine-tuning strategies and adaptive prompt mechanisms to further enhance model consistency and reliability. overall, our work underscores the potential of leveraging llm ensembles for nuanced tasks such as narrative extraction and entity role classification in a multilingual context.