WC Team at S em E val-2025 Task 6: P romise E val: Multinational, Multilingual, Multi-Industry Promise Verification leveraging monolingual and multilingual BERT models (2025)

Abstract: This paper presents our system developed for SemEval-2025 Task 6: PromiseEval: Multinational, Multilingual, Multi-Industry Promise Verification. The task aims at identifying “promises” made and “evidence” provided in company ESG statements for various languages. Our team participated in Subtasks 1 and 2 for the languages English, French, and Japanese. In this work, we propose using BERT and finetuning it to better address the task. We achieve competitive results, especially for English and Japanese.

## introduction corporate environmental, social, and governance (esg) statements often contain forward-looking promises -commitments to sustainability, social responsibility, or ethical governance -that significantly influence public trust and corporate reputation. in recent years, there has been increasing emphasis placed on companies' esg commitments @xcite . ensuring the integrity of esg promises is not only vital for transparency, but also for upholding stakeholder trust and holding organizations accountable for their commitments. however, the complexity and lack of standardization in these statements pose a challenge to innovate new natural language processing (nlp) approaches to assess their strength and verifiability. in this context, promise verification has emerged as an important task: systematically checking if a stated promise is made and supported by evidence in company reports in the form of semeval task 6 @xcite . the task we address here, is titled promiseeval: multinational, multilingual, multi-industry promise verification and is divided into the following sub-tasks: given that the data implies a classification task, one significant approach is to fine-tune pre-existing language models for each language. this is the methodology taken in this paper. advances in natural language processing offer a promising path toward automating promise verification. in particular, transformer-based language models such as bert have proven highly effective at modeling context and meaning in text, especially against traditional machine learning text classification methods such as tf-idf @xcite . bert's flexible architecture enables finetuning for custom classification tasks by adding only a simple output layer, yet achieves robust performance. verifying esg promises is inherently a multifaceted and multilingual challenge. as will be discussed earlier, results performed better for singlelanguage bert models, where learning in multilingual models did not transfer to other languages. promise evaluation in natural language processing is an important area of research which focuses on evaluating if promises are made in certain statements and if they are supported by evidence. this paper systematically explores methodologies for training language models to identify and evaluate these things. recent developments have led to several promising approaches for evaluating multilingual text, which will be explored in background. ## background the training data sets were provided by the promiseeval2025 organizers, covering five languages, including english, french, japanese, chinese, and korean @xcite . each data set, covering one of the five languages, was created by extracting texts from esg reports released by various corporate organizations headquartered in countries where the language is the native tongue. for example, the japanese data set used japanese esg reports published by companies primarily operating or originating from japan, while the french data set used those from france. following extraction, the texts were segmented into paragraphs or sentences. they were later annotated with labels related to the four subtasks set by the organizers. our team focused on subtasks 1 ("identifying promises") and 2 ("linking evidence to the promise") for data sets pertaining to the languages english, french, and japanese. for the goal of each subtask, "identifying promises" required creating models that could determine whether each segmented text contained promise statement(s), while "linking evidence to the promise" focused further on being able to identify whether each text (assuming they contained promise statements) have any evidence statements to support their claim. additionally, for asian languages (including japanese), the subtasks further required models to be able to extract the exact sentences. however, given the complexity and challenge of accurately extracting exact promise and evidence texts, we had limited our model to verify whether a dataset contains promise and evidence text. as for the labels, we used "promise_status" and "evidence_status" to create models for subtasks 1 and 2, respectively. note that in the asian language data sets (including japanese), annotators further included "promise_string" and "evidence_string," which referenced the specific sentences within the data where promise and evidence statements were made. concerning esg specifically, there has been growing interest in methods to assess the credibility of esg promises. early nlp research in this domain involves analyzing esg reports for insights, such as shi bowen (2023) analyzing similarities in sustainability reports through the application of latent dirichlet allocation, and support vector machine models to analyze esg scores against a list of company mergers and acquisitions, and petridis et al. (2022) utilizing a retrieval-augmented generation approach to idenfity sustainable development goals in environmental impact assessments. aside from the aforementioned methods, some of the algorithms that can be found in the literature for identification and classification tasks are naive bayes @xcite , stochastic gradient descent @xcite , k-nearest neighbors @xcite , decisions trees @xcite , convolutional neural networks @xcite and support vector machines @xcite . in 2018, however, a revolutionary language model was released by google, titled bidirectional encoder representations from transformers (bert) @xcite . utilizing an encoder-only transformer architecture, it dramatically improved over previous state-of-the-art models, and remains to perform exceptionally well. an example in utilizing bert in classifying text to quantify esg ratings, @xcite trained roberta and distilroberta, while pasch and ehnes (2022) utilized transformer-based models to train an esg sentiment model based on esg ratings and text documents. relatedly, an annotated environmental claim detection dataset was presented by @xcite , focusing on identifying statements about environmental actions or impacts in corporate communications. this work, along with efforts by @xcite further highlight the importance of identifying esg claims. our work builds on these efforts, thus contributing to the field of promise and claim detection and verification. ## system overview given our team's focus on creating robust models for promise-verification tasks by fine-tuning existing state-of-the-art llm models, our strategy for detecting promise and evidence statements in both subtasks revolved around leveraging the base bert model and its derivatives for french, japanese and multilingual. the decision to adopt bert models for our task is primarily associated with its flexible architecture. it enables researchers to train task-specific models with only another output layer added to the base model, producing highly satisfactory results @xcite . the successful application of bert has also been noted in text classification tasks for esg-related research. addi-tionally, bert's transfer learning feature further allows models to be created from relatively small datasets with limited computational resources. this was another reason to adopt bert models, as the datasets for each language contained only 400 rows of data, which is even smaller than the traditional number of what is considered a 'small dataset.' our team's approach to creating accurate models that can verify promise and its associated evidence was to use monolingual models explicitly dedicated to each language, alongside a multilingual one encompassing all three languages. for english, we adopted the base bert model for finetuning, while french and japanese used languagespecific bert models. for the french model, we chose to fine-tune camembert, a bert model dedicated to french texts. camembert's architecture is based on roberta, an improved iteration of the original bert model, and its training was done using french datasets extracted from the oscar corpus @xcite . in terms of performance, it has produced superior results in downstream tasks, such as part-of-speech tagging and natural language inference, against mbert. in addition, research by @xcite , further found that camembert and in text-classification tasks on verbatim transcripts alongside online posts compared to flaubert (another bert model finetuned on french dataset). regarding the japanese models, we used tohoku bert foot_0 , a well-known bert model that was finetuned for japanese data and is one of the widely adopted models in japanese nlp research. in particular, research by @xcite , found that fine-tuning tohoku bert led to creating a japanese-based sentence-bert that demonstrated higher performance than other models they created. finally, we also utilized mbert foot_1 to create multilingual model for both promise and evidence verification tasks. previous researches that compare performances of monolingual and multilingual models in various classification tasks often provide differing results on which strategies are overall better than the other. nonetheless, in many of the same studies, researchers have also highlighted how multilingual models offer competitive results with similar performance metrics to monolingual models @xcite @xcite . most critically, however, multilingual models have been found to produce superior results in complex classification and retrieval tasks @xcite . for example, in the paper by @xcite , they found models like xlm-r perform well in natural langauge inference tasks. another research by @xcite which is directly related to evidence verification, showcased models trained on cross-lingual fake news evidence datasets can yield advantageous results in evidence classifications. ## conclusion in this paper, we introduce our results for the semeval task. our scores on evaluation data show that models fine-tuned on general corpora can obtain competitive results, especially for english. bert performed well on promise identification, and less well on evidence identification. however, a more thorough comparison between multilingual and language-specific bert is expected to yield more definitive answers as to which methodology is superior overall. while we addressed label imbalance by adjusting class weights during training, our model may still be more biased towards 'yes' data. as such, techniques such as synthetic minority over-sampling technique (smote), undersampling the majority class, or implementing alternative loss functions such as focal loss may further improve model robustness, especially in tasks with severe imbalance in the training dataset. the introduction and successful implementation of the promise verification tasks are expected to have impacts on various fronts, including enhanced accountability and transparency, empowerment of stakeholders, policy and regulation shaping, and increased social and environmental impact. for example, by providing a systematic and scalable approach to verify promises, this task could significantly improve the transparency of organizations and public figures, compelling them to adhere more closely to their commitments. this, in turn, could foster greater trust and credibility among stakeholders and the general public. additionally, equipped with data and insights from the promise verification task, stakeholders, including consumers, investors, and the general public, could make more informed decisions based on the verifiable actions and commitments of organizations and leaders. ultimately, by holding entities accountable for their promises, especially those related to esg commitments, this task could contribute to more tangible progress in addressing environmental issues, promoting social justice, and ensuring ethical governance, leading to a more sustainable and equitable world.