Team INSAL yon2 at S em E val-2025 Task 10: A Zero-shot Agentic Approach to Text Classification (2025)

Abstract: We present Team INSALyon2’s agentic approach to SemEval-2025 Task 10 Subtask 2, which focuses on the multi-label classification of narratives in news articles across five languages. Our system employs a zero-shot architecture where specialized Large Language Model (LLM) agents handle binary classification tasks for individual narrative/subnarrative labels, with a meta-agent aggregating these decisions into final multi-label predictions. Instead of fine-tuning on the dataset, we leverage AutoGen to orchestrate multiple GPT-based agents, each responsible for detecting specific narrative/subnarrative types in a modular framework. This agent-based approach naturally handles the challenge of multi-label classification by enabling parallel decisions across the two-level taxonomy. Experiments on the English subset demonstrate strong performance with our system achieving F1_macro_coarse = 0.513, F1_sample = 0.406, securing third place in the competition. Our findings show that zero-shot agentic approaches can be competitive in complex classification tasks.

## introduction the rapid spread of online news and user-generated content has increased exposure to deceptive narratives and manipulation attempts. major crisis events, such as geopolitical conflicts and climate change discussions, are particularly susceptible to the dissemination of disinformation. to support research in identifying and analyzing these narratives, subtask 2 (task) of the semeval-2025 task 10 @xcite focuses on narrative classification, aiming to automatically categorize news articles into predefined narratives and subnarratives. the goal is to assign multiple subnarrative labels from a two-level taxonomy to news articles. to address this problem, traditional machine learning techniques such as binary relevance (training separate classifiers for each label ignoring potential correlations between labels) @xcite , classifier chains (a sequence of classifiers where predictions are based on previous classifications and original features) @xcite @xcite , and label powerset methods (treating each unique label combination as a single class thus transforming the multi-label problem into a multi-class problem) @xcite @xcite have been explored in the state-of-the-art. more recently, deep learning models leveraging transformer architectures, such as bert @xcite and its multilingual variants (mbert, xlm-roberta @xcite , camembert @xcite ), have proven effective in capturing contextual nuances in text classification. such models are typically fine-tuned on specific datasets to enhance their performance @xcite @xcite . besides, strategies like hierarchical classification models @xcite @xcite and graph-based methods @xcite @xcite @xcite have been employed to account for label dependencies within structured taxonomies like the one used in the challenge. in the task, participants get plain-text news articles in multiple languages (bulgarian, english, hindi, portuguese, and russian). they are sourced from web portals, including alternative media platforms identified by fact-checkers as potentially spreading misinformation. the documents are annotated with a two-level taxonomy of narrative labels @xcite . the goal is to develop systems assigning the appropriate narrative and subnarrative labels to each article. performance is evaluated on coarse (narrative) and fine (subnarrative) levels and as the official measure the sample-averaged f 1 score foot_0 is used. it measures how accurately predicted labels (narrative_x : subnarrative_x) match the ground truth. to solve this task, we introduce an agentic approach 2 where each large language model (llm) agent handles a binary classification task for a single label. these binary decisions are then aggregated using another meta-agent to form the final multi-label output. our method leverages the specialization of individual agents while combining their strengths to improve overall classification performance. on the english language, our approach has achieved f 1_macro_coarse = 0.513 and f 1_sample = 0.406, securing third place. ## problem definition the task is structured as a multi-label, multi-class text classification problem, where each article must be assigned one or more narrative labels from a twolevel taxonomy. the first level consists of broader narratives, while the second level contains more specific subnarratives (e.g. see ## related work to address a multi-label multi-class document classification problem, several techniques have been proposed in the state-of-the-art such as traditional machine learning (bag of words), deep learning approaches (word embeddings, cnns) or even transformer based approaches (bert model family). given the advancements in llm capabilities for various nlp challenges, we propose to incorporate them into our approach. llms can serve as zeroshot classifiers, enabling text classification without explicit training on task-specific datasets. few-shot learning, a prompting method where models are given minimal examples, further enhances their adaptability and performance @xcite . however, while llms demonstrate high accuracy in text classification, their performance can vary based on the task and dataset. fine-tuning strategies, such as enhanced discriminative fine-tuning, can significantly improve their performance, especially in non-generative text classification tasks @xcite . the final methodological choice involved determining whether to employ a single model specialized in multi-label classification @xcite or to utilize multiple binary classifiers, followed by an aggregation of their outputs. while a single multilabel classifier might capture label dependencies, addressing issues like class imbalance more effectively @xcite , having multiple binary classifiers presents a modular and flexible framework optimizing individual classifiers per class pair, potentially enhancing overall classification performance @xcite . techniques such as error-correcting output coding (ecoc) improve generalization by leveraging relationships between classifiers @xcite . additionally, in certain scenarios, binary classifiers, particularly when used with ensemble methods (e.g., one-vs-one, one-vs-all), can achieve superior performance compared to traditional multi-class classifiers @xcite . thus, in the context of llm-based zero-shot classification, using multiple binary classifiers aligns well with the inherent strengths of llms. ## system architecture we propose to adopt an agentic framework, where each agent functions as a specialized binary classifier. a general overview of our architecture is given in figure 1 . each agent is responsible for detecting whether a given text belongs to a specific narrative or subnarrative. we based this decision on the growing ecosystem of llm-based agent frameworks, such as autogen (microsoft, 2024), crewai (crewai, 2024), swarm (openai, 2024), and smolagent @xcite , which provide mechanisms for structuring llms into specialized roles. our classification system is structured around autogen (microsoft, 2024), an agent-based framework to coordinate multiple llm agents. in this setup, each agent processes input independently and returns a binary decision, with some agents dedicated to higher-level narratives and others focused on finer subnarrative distinctions. we provide the prompts for different kinds of agents in appendix b. an example of the functioning of our approach is provided in appendix c. group chat mechanics the system is organized as a group chat consisting of the user proxy agent, the manager agent, and multiple narrative (and subnarrative) agents. the manager agent limits each narrative agent to a single query per classification task, mitigating the risk of extended conversational history that could lead to context length issues in llm-based systems. the user proxy agent initiates the group chat for each new text sample by providing the manager agent with the document to be classified. the manager then selects up to six narrative agents, requesting a binary decision from each. once all relevant agents have responded, the manager collects the answers and produces a multi-label classification output for the text. narrative level classification each narrative agent is created with a system prompt that defines the narrative in question, using the taxonomy file given by the organizers and instructs the agent to respond with either 1 (if the text is clearly related to the assigned narrative) or 0 (if not). additionally, each agent provides a short description, introducing itself and specifying the narrative it detects. it is presented to the manager agent within the group chat when the session is initiated. moreover, llm agents tend to give many false positives due to the semantic similarity of the classes. this is why we specified explicitly that the agent classifies negatively a text that is slightly ambiguous. ## results the main results are reported in on the dev set, the climate change narratives generally have been predicted with higher recall than urw. among the best performing narratives, we can list: "cc: climate change is beneficial", "urw: discrediting ukraine", and "urw: blaming the war on others rather than the invader". in contrast, 9 narratives (41% of total) have zero true positives (@xmath1, meaning the model failed to identify any positive instances of these narratives: 3 cc narratives (e.g., "amplifying climate fears"), and 6 urw narratives (e.g., "russia is the victim"). we may also note the high false positive rate for "cc: criticism of climate policies", "cc: criticism of institutions and authorities" and cc: criticism of climate movement and misclassification of similar narratives. thus, confusion occurred between related categories such as "cc: criticism of climate policies" and "cc: criticism of institutions and authorities" indicating limitations in distinguishing subtle semantic differences. providing more detailed agent prompts focusing on discriminative features between similar categories can be explored for potential improvement. we provide confusion matrices for narratives and subnarratives on the dev set in appendix d. the issues such as class imbalance and the narratives with t @xmath2. another error pattern that can be observed is due to hierarchical error propagation. errors at the narrative level invariably propagated to the subnarrative level, highlighting the importance of highquality initial classification. although our primary focus was on english texts, after the official challenge, we conducted additional experiments on portuguese and russian subsets to address the multilingual nature of the original task. to do so, we translated all texts into english using the deepl translation model @xcite to ensure consistency across linguistic sources. no further pre-processing or data augmentation was applied. the results are given in ## discussion our agent-based classification framework offers several advantages, including ease of implementation, scalability, and model flexibility. its modular design enables parallelization, making it suitable for large-scale classification tasks. additionally, the approach is model-agnostic, meaning it can be used with any model that exposes an api. despite these strengths, the system faces several limitations. the system's primary limitation is latency, as classification depends on multiple api calls, leading to slow processing times. this bottleneck is particularly problematic in real-time applications or large-scale datasets. future improvements could include local model inference to reduce dependence on external apis and caching mechanisms to optimize efficiency. combining our zero-shot approach with fine-tuned components could balance flexibility with performance in a computationally efficient manner. the framework's effectiveness diminishes notably in non-english languages, limiting its applicability in truly multilingual settings without significant adaptation. developing language-specific agent configurations with culturally adapted prompts and examples could improve performance across languages. another directions could be a use of multilingual models like xlm-roberta instead of gpt. in this case, an adjustment of the architecture will be required. the binary decision approach sometimes fails to capture implicit narrative elements that require reading between the lines or understanding cultural context. despite these limitations, the framework remains robust by incorporating a structured two-step classification process and an "other" class to handle ambiguous inputs. future work could explore contextaware classification and cross-agent communication to further improve accuracy and efficiency. adding capabilities for agents to justify their decisions would enhance system transparency and facilitate targeted improvements. ## conclusion this paper introduced an agentic framework for multi-label multi-class text classification, leveraging specialized llm agents to handle narratives and subnarratives. despite hardware constraints preventing local fine-tuning and cost limitations linked to advanced api-based models, the proposed approach demonstrated competitive performance, achieving third place in subtask 2 of semeval-2025 task 10. future work could explore more sophisticated reasoning models and expanded fine-tuning strategies, potentially enhancing classification accuracy while balancing the practical trade-offs between computational resources and model complexity.