Using Contextual Sentence Analysis Models to Recognize ESG Concepts (2022)

Abstract: This paper summarizes the joint participation of the Trading Central Labs and the L3i laboratory of the University of La Rochelle on both sub-tasks of the Shared Task FinSim-4 evaluation campaign. The first sub-task aims to enrich the ‘Fortia ESG taxonomy’ with new lexicon entries while the second one aims to classify sentences to either ‘sustainable’ or ‘unsustainable’ with respect to ESG (Environment, Social and Governance) related factors. For the first sub-task, we proposed a model based on pre-trained Sentence-BERT models to project sentences and concepts in a common space in order to better represent ESG concepts. The official task results show that our system yields a significant performance improvement compared to the baseline and outperforms all other submissions on the first sub-task. For the second sub-task, we combine the RoBERTa model with a feed-forward multi-layer perceptron in order to extract the context of sentences and classify them. Our model achieved high accuracy scores (over 92%) and was ranked among the top 5 systems.

## introduction financial markets and investors can support the transition to a more sustainable economy by promoting investments in companies complying to esg (environment, social and governance) rules. today there is growing interest among investors in the performances of firms in terms of sustainability. therefore, the automatic identification and extraction of relevant information regarding companies' strategy in terms of esg is important. the use of nlp (natural language processing) methods adapted to the field of finance and esg could help identify and process related information. taxonomies are important nlp resources, especially for semantic analysis tasks and similarity measures @xcite . in this context, the finsim4-esg shared task proposed the tasks of enrichment of esg taxonomy and sentences classification. finsim-4 is the fourth edition of a set of evaluation campaigns that aggregate efforts on textbased needs for the financial domain @xcite @xcite . this latest edition is particularly challenging due to the continuously evolving nature of terminology in the domain-specific language of the esg which leads to a poor generalization of pre-trained word and sentence embeddings. several studies addressed the problem of taxonomy generation for different domains @xcite . deep learning based embedding networks, such as bert @xcite have proven to be efficient for many nlp tasks. @xcite used bert for knowledge base completion and showed that bert performs well for this task. @xcite used bert to complete an ontology by inserting a new concept with the right relation. @xcite used sentence bert @xcite to measure semantic relatedness in biomedical concepts and showed that sentence bert outperforms corresponding bert models. @xcite used sentence bert to build a knowledge graph for the biomedical domain and showed that it obtains the best results. for the shared task finsim-4, we proposed several strategies based on bert language models. for the first sub-task, we proposed a model based on pre-trained sentence-bert models to project sentences and concepts in a common space in order to better represent esg concepts. for the second sub-task, we combined the roberta model with a feed-forward multi-layer perceptron to extract the context of sentences and classify them. official results of our participation show the effectiveness of our models over the shared task finsim-4 benchmark. in terms of accuracy, our best runs respectively ranked 1 st and 4 th for the sub-tasks 1 and 2 with scores 0.848 and 0.927, respectively. the remainder of this paper is organized as follows. in section 2, we present the shared task finsim-4 and the datasets for both sub-tasks. our proposed models are detailed in section 3. the setup and official results are described in section 4. finally, section 5 concludes this paper. the finsim 2022 shared task aims to spark interest from communities in nlp, ml/ai, knowledge engineering and financial document processing. going beyond the mere representation of words is a key step to industrial applications that make use of natural language processing. the 2022 edition proposes two sub-tasks. ## conclusion this paper described the joint effort of the l3i laboratory of the university of la rochelle and the trading central labs in the shared task finsim-4 evaluation campaign for the task of esg in financial documents. for this task, we developed bertbased models. our model based on siamese sentence analysis achieved the best results for the first sub-task. for the second sub-task, our approach based on the roberta model got the fourth position.