N lp U ned at S em E val-2025 Task 10: Beyond Training: A Taxonomy-Guided Approach to Role Classification Using LLM s (2025)

Abstract: The paper presents a taxonomy-guided approach to role classification in news articles using Large Language Models (LLMs). Instead of traditional model training, the system employs zero-shot and few-shot prompting strategies, leveraging structured taxonomies and contextual cues for classification. The study evaluates hierarchical and single-step classification approaches, finding that a unified, single-step model with contextual preprocessing achieves the best performance. The research underscores the importance of input structuring and classification strategy in optimizing LLM performance for real-world applications.

## introduction the semeval-2025 task 10 on fine-grained role classification (subtask 1) @xcite aims to assign one or more sub-roles to named entity mentions in news articles @xcite . the task provides a structured taxonomy covering three main role types, each of which is further divided into fine-grained subroles. as a multilabel, multi-class text-span classification challenge, this task holds significant potential for developing objective and transparent ai systems capable of classifying human behavior in news narratives @xcite . the ability to accurately identify and classify entities' roles in news stories can support media monitoring, misinformation detection, and conflict analysis while reducing human subjectivity in information processing @xcite . our approach to this task centered on structuring the input and modeling strategy for optimal performance. the system first generates a summary of the news article to capture the broader context and then extracts the paragraph where the entity mention appears. unlike a hierarchical approach that classifies a main role first and then predicts the subroles associated to that main role, the best-performing system uses a unified prompt that presents all possible subroles at once. this non-hierarchical approach allows the model to directly assign the most suitable fine-grained role without an intermediate classification step. the unified structure ensures that the model considers all possible subroles equally rather than being constrained by an initial high-level role decision, potentially capturing more nuanced role assignments foot_0 . our results highlighted the critical role of signal processing and contextual representation in optimizing classification accuracy. specifically, we found that the way information is structured-whether through hierarchical or unified approaches-significantly may impact performance @xcite . contrary to expectations, breaking down the classification process into multiple stages and assigning explicit sub-role agents based on roles led to poorer results. instead, models performed better when using a single-step decisionmaking process, selecting among all possible subroles simultaneously. this finding underscores the importance of input representation and classification strategy in complex multi-label tasks, where contextual nuances are key to accurately determining entity behavior. ## background the semeval25 task 10 focused on fine-grained role classification in news articles, where the goal was to assign one or more sub-roles to named entity mentions within a given article. the task is framed as a multi-label, multi-class text-span classification problem depending on the provided context. the task relates to existing research in event extraction @xcite , named entity recognition (ner) @xcite , and stance detection @xcite . unlike standard ner, which primarily focuses on entity categorization (e.g., person, organization, location), this task introduces a context-dependent role classification framework, requiring systems to infer entity intent and alignment from the article's discourse. several studies have explored the role of large language models (llms) in legal and justice-related nlp tasks @xcite . recent research has demonstrated llms' ability to analyze legal contracts @xcite , assist in judicial decision-making @xcite , and extract roles in court case narratives @xcite . similarly, ai has been applied to bias detection in judicial decisions @xcite and factchecking in legal and political discourse @xcite . these studies highlight the increasing relevance of ai systems in understanding narrative roles, power dynamics, and responsibility attribution, which aligns closely with the fine-grained role classification required in semeval25. ## system overview our system tackles the task of fine-grained role classification of named entities in news articles. the objective is to classify each entity mention into corresponding sub-roles based on its portrayal in the text. this problem is challenging due to the subjectivity inherent in role perception, the contextual nature of entity portrayal, and the need for fine-grained semantic understanding. to address this, we explored three distinct classification strategies, each varying in preprocessing, hierarchical decision-making, and classification methodology. these approaches were evaluated based on their classification accuracy, robustness, and computational efficiency. ultimately, the single-step role and sub-role classification approach without majority voting emerged as the best-performing system, achieving a performance of 0.33 in the reference metric, exact match ratio (emr) @xcite , in our experiments. ## experimental setup the evaluation of our system is conducted under subtask 1 of the semeval 2025 task 10, which involves multiclass multi-label classification. given the subjectivity of role perception and the contextdependent nature of entity portrayal, this classification task presents significant challenges. an entity may take on multiple sub-roles within a single article, requiring the model to make multi-label predictions while maintaining high precision. ## conclusion this study highlights the growing relevance of large language models in the classification and assessment of human behaviour, particularly in news narratives and structured socio-political taxonomies. our work in the semeval-2025 fine-grained role classification task demonstrates that llms, when properly aligned with human-defined taxonomies, can effectively infer entity roles and sub-roles within complex textual contexts. by structuring input signals, leveraging contextual processing, and optimizing classification strategies, llms can be guided to generate taxonomically coherent and interpretable outputs, reinforcing their potential for media analysis, governance, and ethical ai applications. a key insight from our findings is the critical role of input structuring and system design in aligning llm-based classification with human-defined taxonomies. our results underscore that carefully designed input representations, contextual enrichment, and decision pipelines significantly impact performance-even in zero-shot settings where no explicit model training is conducted. this insight is particularly valuable for ai applications in legal, political, and ethical decisionmaking, where interpretability and alignment with human cognitive frameworks are essential, and data privacy could represent an obstacle to the creation of training datasets.