QM - AI at S em E val-2025 Task 6: an Ensemble of BERT Models for Promise Identification in ESG Context (2025)

Abstract: This paper presents our approach and findings in the SemEval-2025 Task 6: Multinational, Multilingual, Multi-industry Promise Verification (PromiseEval), which focuses on verifying promises in the industrial Environmental, Social, and Governance (ESG) reports. Specifically, we participate in the first subtask of the PromiseEval shared task, promise identification. We tackle this subtask by building an ensemble of four BERT models trained in different experimental configurations, and deploying logistic regression as meta-model. Each configuration has a different combination of two variables: whether augmented data is used, and whether English translation is used. We find out that the BERT model trained without augmented data or English translation not only has the best evaluation results on the test data in most languages, but also has higher robustness than the meta-model. We submitted results from the meta-model to the leaderboard, and rank the first place in Japanese and Korean, the second place in French and Chinese, and the seventh place in English.

## introduction as the public's emphasis on the environment protection and the importance of environmental, social, and governance (esg) aspects of industries grow, a strong implementation of esg framework creates value for companies in various ways @xcite . in order to gain such benefits without paying the cost, it is revealed that companies with environmental violations try to appear more environmentally friendly by producing more frequent, abundant reports with less readability to deflect reader's attention from their violations @xcite . to verify the promises made in the industrial esg reports, @xcite proposes a four-step approach: 1. identifying promise 2. linking supporting evidence to the promise, 3. assessing clarity of the promise-evidence pair, and 4. inferring timing for verifying the promise. these four steps correspond to the four subtasks in the semeval-2025 task 6: multinational, multilingual, multi-industry promise verification (promiseeval) @xcite . in this paper, we describe our submission to the first subtask of the semeval-2025 task 6: promise identification. the subtask 1 is a multi-lingual binary classification task. the used dataset is the proposed multilingual dataset, ml-promise, that includes esg reports from various industries in english, french, japanese, korean, and chinese @xcite . each instance in the ml-promise dataset contains the origin of a pdf and a page number. additionally, a text snippet is included in each instance for the english, french, and japanese languages. these text snippet can be used directly as input for classification. for the korean and chinese languages, the participants need to either extract the text from the whole page for classification or use the given page of the pdf as direct input. the output of the task 1 is a boolean label indicating whether the input contains any promise. we approach this task by building an ensemble of bert models @xcite . each bert model is exposed to a different experiment configuration. our hypothesis is that with each bert model trained differently, the robustness of our ensemble will improve. after evaluating all base bert models and the meta-model on the test data, we find out that the bert model trained with original data without data augmentation or english translation has the highest performance and robustness than the other bert model and the meta-model. ## background bert models have been proven to have promising performance in other esg related tasks. in the multi-lingual esg issue identification shared task, where multi-lingual esg news articles are to be classified into 35 key esg issues, bert-like language models with data augmentations by llms have leading performance in all languages @xcite . in the multi-lingual esg impact type identification shared task, a classification task with three classes, fine-tuned roberta model @xcite is proven to have the best results in english and french, while fin-bert model @xcite with english translation achieves the highest performance in chinese @xcite @xcite . in the multi-lingual esg impact duration inference shared task, a classification task with three classes, deberta-v3 @xcite is the bestperforming model for english, while for korean and japanese, the xlm-roberta model @xcite is among the best models @xcite . based on the findings in the previous tasks, we introduce english translation as a variable into our experimental configurations. furthermore, we pick the deberta-v3 model to be the model receiving english translation, and the xlm-roberta model to receive the original multi-lingual input. more details regarding our classification system and experimental setup is presented in section 3 and 4. the promiseeval shared task has one separate leaderboard for each language. the leaderboard only accepts submission files containing results for all four subtasks. although our focus is only to solve the subtask 1, we utilize the gpt-4o-mini to gain labels for the other three subtasks in order to submit our results to the leaderboard. we present our approach with gpt-4o-mini to generate predictions in section 4. the leaderboard scores are aggregated over labels from all subtasks. this means that the leaderboard scores do not reflect the ranking and performance of participant's system in one individual subtask. to showcase our system's performance in subtask 1, we evaluate our system on the test data and show the results in section 5. ## experimental setup in the process of data augmentation, we exclude pages or sentences containing fewer than 16 words to prevent the dataset from being populated with simple word phrases or page headings. the training datasets for english, french, and japanese languages include the text for classification directly. in contrast, the chinese and korean datasets only list the page number and pdf source. hence, we utilize the pypdf2 library foot_1 to retrieve text from specified pages of the chinese and korean pdfs, serving as the input for the bert models. there are two experimental variables: with augmented data or without, and either using multilingual or monolingual text input (english translation), yielding a total of four configurations. as described in section 3.2, one bert model is trained for each configuration. when a model is trained using monolingual data, it will likewise be evaluated on the english translations of the input text in both the development and test datasets. all bert models share the same hyper-parameters: they are fine-tuned with a batch size of 16, and a learning rate of 6e-6 over 20 epochs. the model with the best macro f1 score is selected as the checkpoint. each of the four fine-tuned bert models is evaluated on the development data split, and the resulting labels and probabilities are recorded. a logistic regression model, built using the scikit-learn library 3 , is then trained on the full development data split, including bert models' outputs and additional metadata as detailed in section 3.2. the positive promise status is given the label id 0 and the negative promise status is 1. the trained logistic regression model is our meta-model. finally, we deploy our fine-tuned bert models on the test dataset, and use the meta-model on the bert mod-els' outputs and other meta data to generate final predictions for the test data. to generate labels for subtasks 2-4, we used the gpt-4o-mini model with retrieval augmented generation as a classifier. in all cases, we set the model's temperature value to 1.0. first, we encoded all provided data points with the openai text-embedding-3-small embedding model. for each subtask, we devised a system prompt that described the problem and the expected model output. for each test sample, we retrieved the three most similar examples from the training data, included them in the sample-specific prompt, and generated the output label. the chinese and japanese datasets included two additional subtasks. for data points containing a promise or evidence, we needed to extract the corresponding string that included the promise or evidence text. to do this, we split the input text of each sample into a list of sentences and then individually classified each sentence to determine whether it contained a promise or evidence. lastly, we concatenated the relevant sentences for the final output. ## results a similar data distribution pattern is observed in the test data. the augmented data helps to dampen the positive class bias, thereby enhancing the model's performance for chinese. this proves that data augmentation is beneficial in reducing the impact of class biases. when comparing the outcomes of using english translations versus not using them, it is noticeable that in most cases, experiments with monolingual text perform worse than those with multi-lingual text. this suggests that translating multi-lingual content into a single language might not enhance the model's learning capabilities. however, this difference could also stem from the variation in model selection between multi-lingual and monolingual contexts. in the future, a more detailed investigation could be conducted using the same bert model across all experimental conditions to examine the helpfulness of english translations. in addition to the base model's predictions and probabilities, we incorporate two metadata variables into the logistic regression: the language of the data instance and whether it is augmented. both of these variables exert minimal to no influence on meta-model's output. the coefficient for the language variable is 0.03. being a logistic regression model, our meta-model treats each variable independently, and thus, the language information contributes little to the final result. similarly, as anticipated, the coefficient of the augmented variable rounds to 0 when rounded to three decimal places. as the meta-model is trained using the development data split that consists solely of original data, this paper describes our contribution to the first subtask of semeval-2025 task 6, promise identification. we deploy an ensemble of four bert models trained in different experimental configurations and use logistic regression as meta-model. our results show that the bert model trained without augmented data or english translation has the best performance in most languages. for future work, one can try out other metamodels that take relations between base model predictions and meta data into account. on the base model side, we can use multi-modal models to take pdf page directly as input to improve the varieties of base models. furthermore, we believe a more sophisticated data cleaning pipeline for extracted text from pdfs can also potentially improve the base bert model's performance.