Oath Breakers at S em E val-2025 Task 06: P romise E val (2025)

Abstract: SemEval Task 6: Promise Eval, was designed to evaluate a company’s adherence to its ESG commitments. Using Natural Language Processing (NLP) and Deep Learning techniques, the task involves analyzing ESG reports to identify, classify, and verify corporate promises. The verification process follows a structured pipeline with four subtasks: Promise Classification, Evidence Verification, Evidence Classification, and Timeline Verification. These subtasks ensure that identified promises are well-defined, supported by credible evidence, and time-bound.For model implementation, BERT was initially used for most of the classification tasks but was later replaced with DeBERTa, which improved performance due to its superior contextual understanding. To enhance model generalization, contrastive learning was applied alongside standard classification loss, helping the model differentiate between positive and negative examples. Oversampling techniques were used to address class imbalance issues, particularly for the Misleading evidence category. For timeline verification, BART was chosen initially but then shifted to DeBERTa again, as it better captures sequential dependencies in text.The dataset consists of ESG reports containing labeled promise statements, evidence snippets, and timeline information. The data was preprocessed by tokenizing text, handling imbalanced classes through oversampling, and incorporating domain-specific embeddings to improve understanding.By implementing these techniques, the research aims to provide a transparent and accountable framework for assessing corporate promises, ensuring that companies are held accountable for their ESG commitments.

## introduction recognizing the critical role of transparency and accountability, semeval-2025 task 6: promiseeval aims to assess a company's commitment and adherence to its environmental, social, and governance (esg) promises. to this end, the organizers have compiled a diverse collection of esg-related texts from company reports and news articles. this task aims to enhance enhance transparency and compel organizations and public figures to uphold their commitments. the insights derived would empower consumers, investors, and the broader public to make informed decisions grounded in verifiable actions and stated objectives. ultimately, it aims to seek tangible progress on global sustainability, social justice, and ethical governance. additional details about the task and dataset can be found at the official project page: https://sites.google. com/view/promiseeval/promiseeval . * corresponding author ## problem definition the primary objective of this research is promise verification. given a report or a part of a report from a company, the goal is to identify and verify promises made within that report. specifically, we aim to determine whether a statement qualifies as a promise based on three key criteria: the promise verification process follows a pipeline approach, with multiple subtasks: ## data description the dataset used for the promise verification task consists of company reports, primarily focusing on environmental, social, and governance (esg) commitments. each entry in the dataset provides detailed information regarding specific esg-related statements. below is an outline of the dataset structure, including key fields and preprocessing steps undertaken. out of the 600 records given in the dataset, each record in the dataset includes the following fields: this dataset provides a structured approach to assess esg promises by capturing essential attributes related to promises, timelines, and evidence, which are critical for the promise verification pipeline. ## related work in recent years, several research efforts have focused on developing robust methodologies for classification and verification tasks in deep learning and natural language processing (nlp). a particularly comprehensive study by @xcite categorizes a wide range of techniques aimed at addressing class imbalance in nlp. their analysis spans sampling strategies, data augmentation, staged learning, and the application of instance-level weighting, all of which are highly relevant to our work. in the context of evidence verification, we adopt oversampling methods inspired by these strategies to mitigate imbalance, especially concerning the misleading class. another pertinent line of research by @xcite explores the classification of implicit negative intentions in questions-an area often overlooked in mainstream nlp research. by introducing the question intention dataset, they provide a framework for detecting both explicit and implicit negative intentions using a tf-idfbased dictionary and transformer models such as roberta. their emphasis on polarity classification and nuanced intention detection has informed our understanding of subtle linguistic cues, which we incorporate into the task of evidence classification within promise verification. complementary to these efforts, @xcite and @xcite demonstrate the effectiveness of contrastive learning in multilingual, multi-label framing detection tasks. their models learn to differentiate between similar and dissimilar frame representations by employing contrastive loss functions, which draw semantically close instances together while pushing apart unrelated ones. inspired by this technique, we integrate contrastive loss into our own model to enhance the semantic separation of misleading and accurate evidence, thereby improving verification accuracy. collectively, these studies lay the groundwork for our approach, which builds upon class imbalance handling, intention-aware representation, and contrastive learning. our method synthesizes these components to address the unique challenges posed by the promise verification task, particularly in the classification and interpretation of evidence. ## methodology the methodology section provides a detailed outlook on the progression of the task and how the baseline approach was complemented via the new methods and techniques. ## evaluation we measure macro-averaged f1 on both our development split and the official test set. for all experiments, the dataset was split into 80% training, 15% validation, and 5% testing, ensuring a robust evaluation of the models while maintaining a sufficient amount of data for generalization. the ## analysis and insights promise evidence clarity timing f1 0.739 0.770 0.669 0.465 clearer signal patterns (presence/absence of explicit evidence markers), so additional representation power translates to more accurate binary judgments. further improvements might be achievable with larger datasets and better context understanding. ## limitations despite the improvements achieved, this study has several limitations. first, the dataset's size and diversity may have constrained the model's ability to generalize, particularly in subtasks such as evidence quality and verification timeline. limited training samples and potential annotation inconsistencies could have introduced biases, affecting performance. the small size of the dataset restricted our ability to train models on a fully representative dataset that was large enough to capture all the nuances. second, while advanced techniques like contrastive learning and few-shot learning improved results, their effectiveness was not uniform across all subtasks. promise classification showed marginal gains, suggesting that additional refinements or task-specific adaptations might be necessary. finally, computational constraints limited the exploration of more complex architectures, such as large-scale transformers or graph neural networks. this hindered our experimentation and we could not try out more resource-intensive models like llama. ## conclusion our participation in semeval-2025 task 6 showcased the effectiveness of a carefully curated methodology that integrated contrastive learning, data augmentation, and semantic-aware representations to tackle the multifaceted challenge of promise verification. achieving the 2nd position overall in the leaderboard in the english dataset. the performance gains observed in the more complex subtasks affirm the importance of addressing class imbalance and leveraging semantic alignment through contrastive loss. beyond competitive results, our model architecture and training strategy provide a strong foundation which we speculate can be used for generalizing across multilingual and multi-domain verification tasks. in summary, our system demonstrates promising strides in automated evidence verification, and we hope our insights contribute meaningfully to the broader research community working at the intersection of fact-checking, framing, and nlp-based reasoning.