DEMON at S em E val-2025 Task 10: Fine-tuning LL a MA -3 for Multilingual Entity Framing (2025)

Abstract: This study introduces a methodology centred on Llama 3 fine-tuning for the classification of entities mentioned within news articles, based on a predefined role taxonomy. The research is conducted as part of SemEval-2025 Task 10, which focuses on the automatic identification of narratives, their classification, and the determination of the roles of the relevant entities involved. The developed system was specifically used within Subtask 1 on Entity Framing. The approach used is based on parameter-efficient fine-tuning, in order to minimize the computational costs while maintaining reasonably good model performance across all datasets and languages involved.The model achieved promising results on both the development and test sets. Specifically, during the final evaluation phase, it attained an average accuracy of 0.84 on the main role and an average Exact Match Ratio of 0.41 in the prediction of fine-grained roles across all the five languages involved, i.e. Bulgarian, English, Hindi, Portuguese and Russian. The best performance was observed for English (3rd place out of 32 participants), on a par with Hindi and Russian. The paper provides an overview of the system adopted for the task and discusses the results obtained.

## introduction the way entities are presented within a text plays a crucial role in shaping the narrative and influencing public opinion. entity framing refers to the process by which a text assigns specific roles to the actors involved in an event, based on a predefined set of roles. this phenomenon is far from neutral, as defining a subject as a "victim" rather than a "perpetrator," for example, can significantly influence how an event is perceived by the reader. the study of the dynamics of mis/disinformation and information manipulation has received growing attention, both within the nlp community and beyond @xcite , and understanding how language structures reality and public debate has become a key task. several efforts have been made in this direction, both in developing computationalthough theoretically-grounded-frameworks @xcite and in creating linguistic resources and label taxonomies aimed at thoroughly analyzing the phenomenon @xcite . these resources, in turn, contribute to advancing the state of the art in automatic approaches. in this paper, we present a computational approach to this challenging issue in the context of our participation in subtask 1 of semeval-2025 task 10 on multilingual characterization and extraction of narratives from online news @xcite . subtask 1 on entity framing precisely focuses on assigning one or more roles to entities mentioned within an article, based on a predefined taxonomy. this task is proposed as a span classification problem and presents both multi-class and multi-label challenges: multiple roles can be assigned to a single entity, and the number of possible roles is extensive. our approach relies on parameter-efficient fine-tuning using qlora and llama 3 (8b parameters) as the reference model. during training, we experimented with different prompting strategies in order to assess the impact on results of two key factors: the paper provides an overview of the task addressed and the approach followed in developing our system, finally discussing the results obtained on the datasets released for the competition in the different setups. ## background given a news article and a list of entity mentions (including their span offsets), the task of entity framing consists in assigning one or more roles to each entity based on a predefined taxonomy. the taxonomy provided for this task covers three main roles: protagonist, antagonist and innocent. protagonists represent entities with a positive role in society, who actively strive for the common good. in contrast, antagonists oppose the good deeds of the protagonists by performing cruel acts against people or things. in between these two opposing factions are the innocents. they represent the outcasts and people who are victims of injustice. a detailed list of fine-grained roles is associated with each one of these main roles. 1 the dataset made available by the organizers consists of articles dealing with two main topics, ukraine-russia war and climate change, and it includes five languages: english, bulgarian, hindi, russian and portuguese. each article in the dataset includes a title and its content, while the annotations specify entity classifications. for each entity, the dataset provides its position within the article, the assigned main role along with the associated fine-grained roles. concerning the task evaluation criteria, in addition to metrics assessing main role accuracy and measures such as micro-precision, micro-recall, and micro-f score, the primary evaluation metric used to determine the ranking is the exact match ratio (emr), which is computed as follows: next section describes the main characteristics of the system, which takes into account the task's peculiarities just outlined. ## dataset overview the dataset used for the model training phase consists of approximately 5500 annotations, in which in 47% of the cases the annotated entities are classified as antagonists, in 32% of the cases as protagonists, and in 21% of the cases as innocents. an imbalance towards the antagonist class is therefore immediately apparent, and is also visible in the evaluation dataset, which justifies more annotations on this class in order to optimize the classifier. with regard to subclass labels, the distribution is also uneven. in fact, some labels have a significantly low number of annotations. in the case of "spy" or "martyr", for instance, the number of annotations is less than 1%. more details on the dataset development and composition are provided in the main task report @xcite . ## system overview the pipeline followed for the system development included three main steps of data pre-processing, prompt definition and actual fine-tuning. as further detailed below, the former two steps aimed at properly addressing the challenges posed by multi-label classification and the impact of different context windows, while the fine-tuning process was set so as to reduce the computational cost deriving from the use of large language models. data pre-processing this phase begins with the segmentation of the input articles into individual sentences using a pre-trained model tailored to each source language. afterward, the tokenized text is processed by a module that identifies the relevant portion of the article containing the entity to be classified, alongside its annotation (entity name and character offset). a variable parameter, the context size, is specified, to determine the number of sentences included in the final passage to pass to the prompt. furthermore, non-english data was translated into english. at this stage, a comparison was made against a number of python libraries for the automatic translation of sentences, from which it emerged that googletrans foot_1 offered a good trade-off between response time and translation quality. the last operation performed in this phase is the augmentation of the training dataset, in order to extend the distribution of underrepresented sub-roles. as mentioned in section 3, the training dataset is characterized by a strong imbalance in the distribution of sub-class labels. new examples were generated from existing ones, through the use of the llama 3-8b model and the definition of a simple prompt. the instructions given to the model include the replacement of some terms with synonyms, leaving the name of the entity and the general meaning of the sentence unchanged.during this phase, a series of tests are carried out to ensure good quality in the examples generated via llm. in particular, from a set of elements generated by llama via the starting prompt, a manual analysis of the results is performed to identify potential frequent errors from which additional prompts can be generated. one of the most common errors concerns the fact that the model tries to achieve the result by substituting names of persons or things. to try to refine the result, synonyms are specified in the prompt. during the actual data augmentation process, a check is performed on the output produced by the model to ensure the presence of the entity to be classified within the sentence. if this check fails, the generated sentence is discarded in order to avoid the addition of noise within the final dataset. ## experiment setup all the experiments carried out for this task were performed only using the data made available by the organizers. as mentioned in section 4, during the training phase, we augmented the data, thus passing from an overall amount of around 5500 to 6750 annotated entities across the five languages. for the data pre-processing step, we used stanza @xcite for the sentence splitting (as this library supports all the five languages included in the dataset) and the googletrans library to translate the data in english. as regards the context window, we observed that increasing the number of sentences to include in the input did not necessarily lead to better predictions. as a matter of fact, this often introduced conflicting annotations for the same entity. after several testings, we finally opted for a span of three sentences, as this allowed to provide a reasonable amount of information to get accurate predictions. the model used is meta-llama-3-8b-instruct 3 . for its fine-tuning, the model was loaded with four-bit quantization, and the lora parameters were configured as follows: @xmath0, @xmath1. adamw was used as optimizer. the model was trained over five epochs due to the small dataset size. with particular reference to the qlora, the training settings used are: @xmath2, @xmath3, @xmath4.3, @xmath5.03, @xmath6, @xmath7. all experiments were performed in a google colab environment, with a a100 gpu (40 gb vram). ## results this section aims to provide an overview not only of the final results obtained during the evaluation phase, but also of the preliminary results obtained in the development phase while experimenting with the different prompting strategies. ## error analysis to complete our description, we carried out an exploratory error analysis of the model configuration that obtained the best results during the development phase and was finally employed for the evaluation phase; specifically, the configuration is the one featuring 3 context sentences in the prompt and using s2 as prompting strategy. the analysis was performed on the development data itself, due to the absence of the gold labels for the test set. as a case study, we opted for the hindi section of the task dataset, since it provides a larger number of annotated entities (280) compared to the other languages, thus allowing for a more reliable basis for observing the model's behavior. the full confusion matrix is shown in appendix b. the sub-roles for which the model was completely unable to make correct predictions are 'guardian', 'traitor', 'rebel', 'scapegoat', 'bigot' and 'spy'. conversely, the three sub-classes with the highest number of correct predictions are 'sabo-teur', 'exploited' and 'peacemaker'. among these three, 'peacemaker' stands out with an accuracy of around 0.7. notably, 'peacemaker' appears only in 6% of the training dataset, suggesting that label frequency alone does not determine classification success. this is further supported by the case of 'foreign adversary', which, despite having a frequency of over 10% in the training set, was classified with an accuracy of only 0.38. a significant source of error involves the 'virtuous' sub-role. as the matrix shows, the model frequently confuses this sub-class with others. in particular, the number of false positives for this sub-role is 16. a recurrent misclassification is between 'exploited' and 'virtuous', with the model incorrectly predicting the latter for the former 10 times. this type of error is particularly critical as it affects not only the sub-role but also the broader classification of the main role (since 'exploited' is a specification of the 'innocent' role, while 'virtuous' falls under the 'protagonist' category). the 'exploited' sub-role proved to be quite problematic indeed, as despite achieving 27 correct predictions, it also exhibited a high number of false positives (19) and false negatives (28), indicating substantial confusion in distinguishing it from similar categories. certainly the main factor related to the low emr score concerns the nature of the metric itself, as it does not distinguish partially right answers from wrong answers. this makes the correct prediction of multi-subclass annotations particularly complex. in addition, the choice of the s2 prompting strategy for training the model results in the inability of the model to predict annotations from two or more sub-classes. in spite of this, it is preferred over the other two strategies because of the greater accuracy in annotations from only one sub-class. from a grammatical point of view, the model makes several errors that can be traced back to certain writing techniques commonly used within articles; these are not correctly understood by the model. for example, passive forms often lead the model to treat entities as active subjects, despite the fact that they are the patients, i.e. the entities undergoing the action. in example 1 below, the model mistakes ukraine for an active subject by misclassifying it as conspirator. in some cases, one can see how the irony used by the writers leads the model to a semantic misreading. in example 2, the model literally interprets the sentence by classifying the entity as deceiver, whereas the correct classification is tyrant. in addition, the size of the context significantly influences the correct prediction of the entities. in fact, although three article sentences are extracted as context for each classification, in some cases the sentences are short and of little meaning, thus making the classification more challenging. ## conclusions the paper described an approach based on llama 3 fine-tuning to tackle subtask 1 on entity framing. the results we obtained especially within the final evaluation phase indicate that while the model generally classifies entities' main roles quite effectively, it struggles more with exact sub-role matching, as seen in the moderate scores obtained in terms of emr, which was also the primary metric used for this task and determining its ranking. another general remark concerns the fact that performance greatly varied by language and especially between the development and test sets, suggesting that factors such as dataset composition, but also translation effects could have had an impact on results. future improvements could focus on enhancing sub-role differentiation, possibly through better prompting strategies or alternative fine-tuning approaches.