S em E val-2025 Task 6: Multinational, Multilingual, Multi-Industry Promise Verification (2025)

Abstract: While extensive research exists on misinformation and disinformation, there is limited focus on future-oriented commitments, such as corporate ESG promises, which are often difficult to verify yet significantly impact public trust and market stability. To address this gap, we introduce the task of promise verification, leveraging natural language processing (NLP) techniques to automatically detect ESG commitments, identify supporting evidence, and evaluate the consistency between promises and evidence, while also inferring potential verification time points. This paper presents the dataset used in SemEval-2025 PromiseEval, outlines participant solutions, and discusses key findings. The goal is to enhance transparency in corporate discourse, strengthen investor trust, and support regulators in monitoring the fulfillment of corporate commitments.

## introduction in an era characterized by rapid information dissemination and increasing reliance on public statements from influential figures-such as corporate executives and political leaders-the balance between freedom of speech and ethical responsibility has become a critical societal concern. while freedom of speech empowers individuals to express opinions and make commitments, it also raises complex challenges when these statements impact public trust, financial decisions, or social stability. only a small number of political leaders, such as the president, may be tracked manually @xcite . 1 however, when this extends to legislators or even corporate-level individuals, the number of targets increases significantly, making automated tracking a necessary and inevitable approach. although there has been extensive discussion on disinformation @xcite @xcite and misinformation @xcite @xcite , there is relatively little focus on statements related to the future. while some analyses of forward-looking statements @xcite address future events, promises represent visions for the future that are more abstract in nature and difficult to verify. for example, a forward-looking statement might forecast the next quarter's earnings per share (eps), which can be verified once the quarter concludes. however, a promise may be less specific, such as "the company will continue to strive to reduce carbon emissions in the coming years." such a promise can create a positive impact for the company but also poses regulatory challenges. public figures often make promises that shape societal expectations and influence critical decisions. these promises are typically based on the information available at the time. however, when such commitments go unfulfilled, the consequences extend beyond personal accountability to affect broader public trust and market stability. the challenge lies in discerning whether these unfulfilled promises result from unforeseen circumstances, representing legitimate changes in strategy, or if they were knowingly misleading from the outset. in the financial field, the regulatory framework governing corporate disclosures has long established sophisticated guidelines for forward-looking statements, yet remains underdeveloped in addressing esg commitments and promises. this regulatory asymmetry persists despite esg commitments increasingly resembling financial forward-looking statements in their market impact. more companies now disclose climate transition plans with quantified milestones, and some institutional investors use esg forward-looking metrics in capital allocation decisions. yet, unlike financial forward-looking statements, esg-related promises face significant legal and ethical risks. some climate-related statements have been alleged as cases of "greenwashing." as an early step in assessing the integrity and fulfillment of the company's esg promises, we introduce a new task: promise verification, which considers multinational, multilingual, and multiindustry aspects. we propose leveraging natural language processing (nlp) techniques to automatically detect esg commitments made by companies, identify supporting evidence, and evaluate the alignment between the stated commitments and the corresponding evidence. furthermore, this approach aims to infer or detect potential time points at which these commitments can be verified. this fine-grained analysis ensures transparency and accountability in corporate esg discourse. through multilingual and multi-industry scalability, the system is designed to operate across geopolitical boundaries and sector-specific nuances, making it adaptable to diverse regulatory environments and cultural expectations. this research aims to foster a more informed and accountable public sphere. in this paper, we present the details of the dataset used in semeval-2025 promiseeval, the solutions from task participants, and the findings of this round's shared task. we believe that this dataset and the proposed systems can not only assist regulators but also help companies prepare reports that are more trustworthy to investors, as well as support companies in detecting any incompleteness in the information disclosed through natural language. a clear label indicates that the provided evidence is both specific and sufficient to substantiate the promise. a not clear label reflects ambiguity or insufficient detail, making it difficult to confirm the company's commitment. the misleading label is applied when the evidence appears intentionally deceptive or when it misrepresents the actual fulfillment of the promise. both the quantity and quality of evidence are critical in this assessment. ## participants and methods the semeval-2025 task 6 attracted various innovative approaches from participating teams, reflecting the diverse strategies used to tackle multilingual esg promise verification. ten teams shared their experimental results on the kaggle leaderboard, and seven teams submitted a method description paper. some teams participated only in the promise and evidence verification tasks, while others focused on specific languages. we summarize their methods in this section. please refer to the paper for more details. csecu-dsg (hossain and chy, 2025) proposed a bi-lstm-based model, which leverages team name method csecu-dsg (hossain and chy, 2025) bi-lstm (laser and universal embedding) cscu @xcite gpt-4o, svm, distilbert (data augmentation) cyut @xcite llama-3.1 (structured prompt) & rag oath @xcite deberta (data augmentation) qm-ai @xcite ensemble bert (data augmentation) wc team @xcite bert (camembert, tohoku-bert) ynu-hpcc @xcite bert (r-drop) cscu @xcite focused on data augmentation, particularly through paraphrase augmentation and synthesis augmentation generated by gemini-2.0-flash. they compared multiple classifiers, including gpt-4o (zero-shot and six-shot), support vector machine (svm) using multilingual e5 embeddings, and fine-tuned distilbert. their results highlight that synthetic data augmentation significantly boosts classification performance, especially for identifying promises and supporting evidence. cyut @xcite adopted a structured prompting with retrieval-augmented generation (rag) approach, using llama 3.1 to systemati-cally evaluate promises. their framework employs structured definitions, examples, and step-by-step reasoning, combined with retrieval of relevant context, to enhance esg promise verification across multiple languages. oath @xcite introduced a deberta-based pipeline, enhanced with contrastive learning and data augmentation to handle class imbalance and subtle differences between promise-related and non-promise text. this approach significantly improved evidence classification and timeline verification performance. qm-ai @xcite proposed an ensemble bert framework, combining four different bert variants (original, augmented, translated, and mixed-language versions). this ensemble approach leverages both multilingual training and language-specific fine-tuning to improve robustness, particularly for non-english data. wc team (nishi and takagi, 2025) adopted a language-specific bert strategy, training individual bert models for each language-bert-baseuncased for english, camembert for french, and tohoku-bert for japanese. this monolingual fine-tuning approach emphasizes capturing each language's unique syntactic and semantic characteristics, achieving strong performance for highresource languages. finally, ynu-hpcc (deng et al., 2025) introduced a bert model regularized with r-drop, a regularization technique that forces consistent predictions across different dropout applications. this method stabilizes the model's predictions in lowresource and noisy environments, particularly benefiting smaller language datasets within the task. these methods collectively demonstrate the importance of data augmentation, multilingual embeddings, structured prompting, and regularization techniques when developing robust promise verification systems for esg reports across diverse languages. ## evaluation results each team was allowed to submit multiple entries. the public leaderboard reflected performance on 70% of the test set, while the remaining 30% was kept private by the organizers. the values presented in as shown in the participants experimented diverse approaches, ranging from transformer architecture to llm-based framework, revealing important insights about effective multilingual nlp strategies. most results demonstrate the effectiveness of models designed with multilingual capabilities and the potential of advanced llm techniques such as rag, cot, got structured prompting when properly implemented. additionally, team wc achieved encouraging results with a monolingual approach using separate models for each language, while team oath experimented with different approaches for each subtask, showing promising scores compared to the multilabel classification approach. the varying effectiveness of data augmentation and ensemble methods across languages and tasks highlights the need for language/task-specific strategies rather than one-size-fits-all approaches. for english and french, data augmentation showed minimal benefits contrary to the japanese, korean and chinese datasets for which data augmentation using an llm like gpt-4o provided meaningful improvements. the best-performing approaches combined targeted data augmentation, fine-tuning on large models like bert, deberta and xlm-roberta and an llm framework, which is obviously a viable alternative to traditional fine-tuning. for korean, the highest performance was achieved by team qm-ai, which utilized an ensemble of multilingual bert models directly trained on original texts without english translation @xcite . a notable challenge specific to the korean dataset was that texts were provided as pdf pages rather than extracted snippets, requiring additional preprocessing to extract clean textual inputs. although data augmentation using gpt-4o generally improved results for korean, excessive synthetic augmentation negatively affected performance, as observed with team csecu-dsg @xcite . this indicates the importance of carefully balancing data augmentation and preserving linguistic nuances specific to korean. for japanese, three teams-qm-ai, csecu-dsg, and wc team-participated. their approaches included: • qm-ai: bert-based ensemble models with data augmentation and machine translation ## conclusion this paper presents semeval-2025 task 6, a multilingual, multi-industry shared task on verifying corporate esg promises-a shift from misinformation research to future corporate commitments critical for public trust. we introduced a novel dataset in five languages annotated for promise identification, evidence detection, clarity, and timing. results from diverse methods, including multilingual transformers and llm-enhanced prompting, highlight challenges like data imbalance and linguistic variation. we propose expanding research to greenwashing risk and stakeholder impact, aiming to strengthen esg transparency and accountability.