Exploring Knowledge Composition for ESG Impact Type Determination (2023)

Abstract: In this paper, we discuss our (Team HHU’s) submission to the Multi-Lingual ESG Impact Type Identification task (ML-ESG-2). The goal of this task is to determine if an ESG-related news article represents an opportunity or a risk. We use an adapter-based framework in order to train multiple adapter modules which capture different parts of the knowledge present in the training data. Experimenting with various Adapter Fusion setups, we focus both on combining the ESG-aspect-specific knowledge, and on combining the language-specific-knowledge. Our results show that in both cases, it is possible to effectively compose the knowledge in order to improve the impact type determination.

## introduction the substantial rise in environmental, social, and governance (esg) research over the past few years underscores the growing significance of sustainability in the corporate world @xcite . both investors and companies have a growing interest in esg issues, as it becomes clearer that they are vital for a company's brand and, consequently, also value @xcite , @xcite . studies show that investor interest in a company's stock depends significantly on whether an estimation of their esg-practices is available or not @xcite . different agencies have developed a variety of esg-scoring mechanisms in order to quantify the sustainability practices of companies which offers investors an easier way to determine the esg-related risk a company represents in their portfolio. popular examples of this are msci foot_0 and sustainalytics foot_1 . however, inconsistencies in the scoring practices and discrepancies between different score-providers cast doubt on the available esg-scores @xcite , @xcite . this doubt could be alleviated by providing a more detailed overview of the various aspects of sustainability a company is involved in. researchers in the field of natural language processing (nlp) have recently made efforts to enhance conventional esg scoring methodologies by incorporating alternative data sources, aimed at offering a clearer understanding of a company's sustainability practices. @xcite underscores the impact of esg-risk events found in social media data, illustrating how they can result in adverse financial returns. meanwhile, @xcite focuses on analyzing news articles to construct a predictive model for determining esg scores. the ml-esg series of shared tasks aims to advance research in this area of using natural language processing to provide more transparency in esg topics. ml-esg-1 focused on determining the esg-aspects of news articles @xcite . in the next step of the ml-esg series, ml-esg-2 aims to find the impact type of the information in news articles, meaning detecting if they represent opportunities or risks @xcite . in this paper, we present our submission for the french and english part of the ml-esg-2 shared task. we train adapters on different parts of the dataset and experiment with several approaches in order to combine their knowledge using adapter fusion. we show that this knowledge composition yields a significant improvement of the performance compared to simply finetuning an adapter for the whole dataset. our submitted approaches achieve fourth place for french and fifteenth place for english, however we investigated additional approaches after the task deadline which improve on those results. ## experimental approach the basis of our approach consists of mbert @xcite , a multilingual bert-based model. for fine-tuning, we use bottleneck-adaptermodules @xcite , which are a small set of weights inserted into the layers of the basemodel. in order to account for the specific language we are training in, we implement an approach described in @xcite , where a pre-trained language adapter is inserted before the adapter that is being fine-tuned (known as the task-adapter). we use the language adapters from the adapterhub @xcite , which are available for all task-languages for the mbert architecture. adapter-modules allow us to be more flexible in training our model, as we will describe in subsection 3.2. to train the adapters, we split the dataset and use 10% as the evaluation set. we use a learning rate of 5e -5. finally, to account for the label imbalance in the training data we can observe in figure 1 , we use a weighted loss function. ## results the test data is shown in figure 1 in blue. for english, the label imbalance in the training set is also present in the test set. for french, while there is still no clear imbalance, the "risk" label now occurs more often as opposed to the "opportunity" label in the training set. in ## conclusion in this work, we present multiple approaches using adapters to determine the esg impact type of news articles. we show that it is possible to gain significant performance increases by using adapter fusion, which allows us to combine the knowledge present in the separate adapter modules, by experimenting with implementing adapter fusion in two different ways: firstly, using the prior knowledge about esg aspects, we train an adapter for each aspect and fuse them to determine the esg impact type. secondly, we fuse the adapters trained on the different languages in order to combine the knowledge learned from the different language datasets. further studies could focus on how to combine the two adapter fusions, which we did not attempt in this work due to time constraints. in addition, it might be interesting to use the japanese and chinese datasets to try to improve the language-fusion approach even further. finally, putting more focus on augmenting the training data, e.g. by using paraphrases, could bridge the gap between this study and the top-placed systems.