Fane at S em E val-2025 Task 10: Zero-Shot Entity Framing with Large Language Models (2025)

Abstract: Understanding how news narratives frame entities is crucial for studying media’s impact on societal perceptions of events. In this paper, we evaluate the zero-shot capabilities of large language models (LLMs) in classifying framing roles. Through systematic experimentation, we assess the effects of input context, prompting strategies, and task decomposition. Our findings show that a hierarchical approach of first identifying broad roles and then fine-grained roles, outperforms single-step classification. We also demonstrate that optimal input contexts and prompts vary across task levels, highlighting the need for subtask-specific strategies. We achieve a Main Role Accuracy of 89.4% and an Exact Match Ratio of 34.5%, demonstrating the effectiveness of our approach. Our findings emphasize the importance of tailored prompt design and input context optimization for improving LLM performance in entity framing.

## introduction news is produced and consumed at an unprecedented scale, yet the expectation of neutrality and objectivity in journalism @xcite often contrasts with the reality that media selectively determines what is newsworthy @xcite . through inclusion, omission, or emphasis of specific details, journalists shape public interpretation of events @xcite , promoting particular recommendations @xcite , which in turn influences public opinion and policy decisions @xcite . when such selective framing is used to shape perceptions of an entity, it is known as entity framing @xcite . semeval-2025 task 10 introduces a benchmark for analyzing entity framing @xcite , examining how linguistic choices from word selection to narrative structure affect perception. example in figure 1 demonstrates how strategic language choices portray the entity bill gates as gates claimed that because he continues to spend billions of dollars on climate change activism, his carbon footprint isn't an issue [. . . ] elsewhere during the carefully constructed interview, gates said he was surprised that he was targeted by 'conspiracy theorists' for pushing vaccines during the pandemic [. . . ] while the bbc interview was set up to look like gates was being challenged or grilled, he wasn't asked about his close friendship with the elite pedophile jeffrey epstein figure 1 : an example document excerpt from the dataset. the author strategically uses loaded language, contrastive framing, and selective emphasis to shape the reader's perception of bill gates as corrupt and deceptive. by highlighting contradictions in his public advocacy, casting doubt on the authenticity of his media appearances, and referencing controversial associations, the text reinforces a negative framing. an antagonist. despite mentioning his substantial investments in climate change initiatives, the narrative emphasizes his high carbon footprint, highlighting hypocrisy. phrases such as "carefully constructed interview" and references to his connections with epstein further reinforce an image of deception and evading accountability. this selective emphasis shapes the reader's perception of gates as a deceiver and corrupt figure, demonstrating how framing subtly influences interpretation. recent advances in large language models (llms) have demonstrated superior performance in text classification, frequently outperforming traditional machine learning approaches on complex tasks @xcite . in this paper, we explore if this extends to entity framing. relying solely on prompting, we ask whether llms accurately identify the framing roles assigned to entities in news articles. given the sensitivity of llm predictions to prompt variability @xcite , we assess how prompt engineering strategies such as role-based prompting, incorporating task related information such as label definitions, and rationale generation shape classification performance. to this end, we develop a modular zero-shot approach that relies exclusively on prompting. our method decomposes the task into two stages, first predicting broad narrative roles (protagonist, antagonist, innocent), and then refining into finegrained roles. we systematically vary the input context comparing full articles, entity-centered excerpts, and framing-preserving summaries and design prompt templates incorporating expert personas and task related information. the main contributions of this paper are: we report a main role accuracy of 89.4% and an exact match ratio of 34.5% on the semeval 2025 task 10 subtask 1 for english, placing sixth on among the participating teams. we make our codebase and associated prompts publicly available. foot_0 2 background: task & dataset the semeval 2025 task 10: multilingual characterization and extraction of narratives from online news introduces three subtasks for analyzing narrative elements in news articles across five languages: bulgarian, english, hindi, european portuguese, and russian. this paper focuses on subtask 1: entity framing for english. the goal of entity framing is to determine how a named entity is framed within a news article. framing is represented at two levels: main role and fine-grained roles. the main role broadly categorizes the entity as a protagonist, antagonist, or innocent, while fine-grained roles provide more detailed labels. 2 an entity may be framed with multiple fine-grained labels, making this a multiclass, multi-label classification problem. for further details on the dataset, including domain coverage, corpus statistics, and annotation methodology, see @xcite . we now describe our approach to leveraging large language models (llms) for entity framing classification. ## related work language shapes interpretation by selectively emphasizing aspects of reality @xcite . prior research identifies two key linguistic mechanisms underlying framing effects: naming conventions, where respectful or formal titles correlate strongly with positive sentiment @xcite ; and narrative framing, reflecting partisan differences in media portrayals of events (ziems and yang, 2021). however, these studies primarily treat framing as an implicit phenomenon correlating with other tasks, rather than explicitly classifying or predicting specific framing roles or labels. beyond text, in the multimodal domain, @xcite introduce hvvmemes, a meme dataset covering covid-19 and us politics, annotated with coarse-grained framing roles: hero, villain, victim, or none. in contrast, our work focuses on news text and classifies entities at a more fine-grained, hierarchical roles. large language models (llms) have recently achieved strong performance in text classification tasks, often surpassing traditional machine learning methods on complex datasets @xcite . however, significant challenges remain, particularly concerning sensitivity to prompt phrasing and input structure @xcite . structured prompting strategies, such as role conditioning @xcite and instruction reframing @xcite , have been proposed. nevertheless, the effectiveness of these strategies remains highly task-dependent and often requires careful adaptation to specific problem settings @xcite . guided by these insights, we develop and systematically experiment with multiple prompting strategies for hierarchical entity framing, aiming to improve both coarse and fine-grained entity framing classification. {expert_phrasing} you will be provided with {input_format_phrase}. your task is to {task_definition}. {task_instructions} {output_format_with_example} {input_context} figure 2 : structure of the prompt template. curlybracketed content is dynamically replaced based on the experimental setting (see appendix d for exact phrasing). the template is designed to minimize variability, ensuring that observed differences arise solely from targeted prompt modifications. ## prompting for framing classification to systematically evaluate the impact of different prompting strategies on framing classification, we adopt a structured, template-based approach (figure 2 ). llm decision-making is often opaque. by using a fixed template, we ensure any variation in outputs comes from our controlled prompt modifications rather than randomness.the template consists of modular components that adapt to different experimental conditions. our approach is structured around three key components: (1) input context which determines the textual context provided for classification, specified in input_context. (2) prompt design, which examines prompting strategies, such as persona prompting, structured within task_definition, task_instructions, and output_format_with_example. (3) inference strategy, which defines whether classification is performed as a single task or decomposed into two steps, reflecting the hierarchical nature of framing. the following sections provide a detailed discussion of each component. ## experimental setup we first discuss a key assumption in our study, followed by evaluation metrics, and llm setup. single-label approximation for fine-grained role classification although the semeval task defines entity framing as a multi-label classification problem, we observe that entity mentions typically receive only one fine-grained role, averaging 1.08 fine-grained roles per mention in the english training split, a trend consistent across languages and main roles (appendix b). additionally, in our experiments, we observed that the model in our study consistently predicts two main roles even when allowed to predict only one. given these patterns, we adopt a single-label approximation, assigning each mention a single fine-grained role. ## results we present results from our prompting experiments on the development set, followed by the official semeval results and post-semeval analyses. key findings are discussed in section 7. ## insights we present key findings from our experiments, examining the impact of different approaches. framing and emphasis: selective highlighting shapes interpretation as shown in ## conclusion in this paper, we explore the effectiveness of large language models (llms) for entity framing classification in a zero-shot setting. while llms perform well in broad role classification, fine-grained classification remains challenging. our multi-step approach, incorporating distinct input contexts and prompt strategies at each stage, significantly improves overall performance. document:{document_type_str}. the corresponding document_type_str for each input context type is provided in the table below: input context type document_type_str full text news article entity-sentences or entity-sentences neighbors excerpt of a news article neutral summary or framing-preserved summary news article summary