Tuebingen at S em E val-2025 Task 10: Class Weighting, External Knowledge and Data Augmentation in BERT Models (2025)

Abstract: The spread of disinformation and propaganda in online news presents a significant challengeto information integrity. As part of the SemEval 2025 Task-10 on Multilingual Characterization and Extraction of Narratives from Online News, this study focuses on Subtask 1: Entity Framing, which involves assigning roles to named entities within news articles across multiple languages.We investigate techniques such as data augmentation, external knowledge, and class weighting to improve classification performance. Our findings indicate that class weighting was more effective than other approaches

## introduction the internet has opened new ways for communication but has also made consumers more vulnerable to misleading content and manipulation (semeval2025-task-10 (2025)). recognizing propaganda strategies is crucial for combating disinformation, particularly in media analysis, politics, and online discussions. the semeval 2025 multilingual characterization and extraction of narratives from online news task aims to automate the identification and classification of narratives, assisting analysts in addressing disinformation. this task comprises three subtasks: entity characterization, narrative classification, and narrative extraction. it is available in five languages: bulgarian, english, hindi, portuguese, and russian. more information can be found in the task description document @xcite ). this paper discusses our experimental approach in subtask 1. we evaluated several transformer-based models with minimal hyperparameter tuning. we experimented with data augmentation, external knowledge integration, and class weighting to improve performance. our model performed better than the baseline, ranking 21st out of 32 participants in the final submission. however, our results from the development set were better, suggesting that our models were likely overfitting and leading to overly optimistic results. our code and setup are available at: https://github.com/ cicl-iscl/semeval25-task10 . ## background the task covers news articles from two domains: the ukraine-russia war and climate change (semeval2025-task-10 (2025)). in this paper, we focus on subtask 1, entity-framing. the goal is to assign one main role and one or more sub-roles to a preidentified named entity (ne) in a news article, using a fine-grained entity role taxonomy @xcite ). the task is formulated as a multi-label, multiclass text-span classification problem and does not require named entity recognition (ner) @xcite ). an example of a system response to a news article is provided in appendix a. ## system overview and experimental setup this study evaluated transformer-based models with minimal hyperparameter tuning. we adopted a nonhierarchical approach, first classifying sub-labels and then assigning the main label within the algorithm. we employed bert-family models, specifically roberta @xcite ) and distilbert @xcite ). ## conclusion our results demonstrate the effectiveness of various approaches for entity role classification in multilingual 1251 data. the exact match ratio shows that while our models performed competitively, there is room for improvement. before the closing of the official submission leaderboard, augmented data combined with roberta achieved the highest emr (0.13190). data augmentation significantly enhanced performance for underrepresented classes, as seen in f1 scores. however, its failure to boost the emr could be caused by backtranslation-label noise, loss of contextual integrity, or inconsistencies in entity role assignments. especially thinking our best results came from an intermediate data augmentation, in which we did not "over-augment" the data. however, a key oversight in our process was that we did not record the submission results at various stages. in short, the complexity of this task may require a more refined augmentation technique to reduce noisy data. another observation is that the validation results were likely inflated due to the similarity of training and validation data, leading to overly optimistic performance estimates. test results revealed that our models were likely overfitted. also, the results of external knowledge also indicate potential overfitting, shown by high main role accuracy but low metrics. other factors could be differences in dataset distribution, improved feature representation, or random initialization effects. further examination with additional models is needed. beyond these, we explored prompting techniques at the entry level, which showed some promise in improving accuracy. however, due to time constraints, we could not further investigate prompting. in closing, classifying with imbalanced datasets remains crucial. future research should explore alternative models, such as gpt-based architectures, prompting or ensemble learning techniques @xcite ), as they combine multiple models to leverage their strengths. "it is "abundantly clear" that the met office cannot scientifically claim to know the current average temperature of the u.k. to a hundredth of a degree centigrade, given that it is using data that has a margin of error of up to 2.5°c, notes the climate journalist paul homewood. his comments follow recent disclosures in the daily sceptic that nearly eight out of ten of the met's 380 measuring stations come with official 'uncertainties' of between 2-5°c. in addition, given the poor siting of the stations now and possibly in the past, the met office has no means of knowing whether it is comparing like with like when it publishes temperature trends going back to 1884. there are five classes of measuring stations identified by the world meteorological office (wmo). classes 4 and 5 come with uncertainties of 2°c and 5°c respectively and account for an astonishing 77% of the met office station total. class 3 has an uncertainty rating of 1°c and accounts for another 8.4% of the total. the class ratings identify potential corruptions in recordings caused by both human and natural involvement. homewood calculates that the average uncertainty across the entire database is 2.5°c. in the graph below, he then calculates the range of annual u.k. temperatures going back to 2010 incorporating the margins of error. the blue blocks show the annual temperature announced by the met office, while the red bars take account of the wmo uncertainties. it is highly unlikely that the red bars show the more accurate temperature, and there is much evidence to suggest temperatures are nearer the blue trend. but the point of the exercise is to note that the met office, in the interests of scientific exactitude, should disclose what could be large measurement inaccuracies. this is particularly important when it is making highly politicised statements using rising temperatures to promote the net zero fantasy. as homewood observes, the met office "cannot say with any degree of scientific certainty that the last two years were the warmest on record, nor quantify how much, if any, the climate has warmed since 1884". the u.k. figures are of course an important component of the met office's global temperature dataset known as hadcrut. as we noted recently, there is ongoing concern about the accuracy of hadcrut with large retrospective adjustments of warming in recent times and cooling further back in the record. in fact, this concern has been ongoing for some time. the late christopher booker was a great champion of climate scepticism and in february 2015 he suggested that the "fiddling" with temperature data "is the biggest science scandal ever". writing in the telegraph, he noted: "when future generations look back on the global warming scare of the past 30 years, nothing will shock them more than the extent to which official temperatures records -on which the entire panic rested -were systematically 'adjusted' to show the earth as having warmed more than the actual data justified."