Automatic Term and Sentence Classification Via Augmented Term and Pre-trained language model in ESG Taxonomy texts (2022)

Abstract: In this paper, we present our solutions to the FinSim4 Shared Task which is co-located with the FinNLP workshop at IJCAI-2022. This new edition of FinSim4-ESG is extended to the “Environment, Social and Governance (ESG)” related issues in the financial domain. There are two sub-tasks in the FinSim4 shared task. The goal of sub-task1 is to develop a model to predict correctly a list of given terms from ESG taxonomy domain into the most relevant concepts. The aim of subtask2 is to design a system that can automatically classify the ESG Taxonomy text sentence into sustainable or unsustainable class. We have developed different classifiers to automatically classify the terms and sentences with augmented term and pre-trained language models: tf-idf vector, word2vec, Bert, Distill-Bert, Albert, Roberta. The result dashboard shows that our proposed methods yield a significant performance improvement compared to the baseline which ranked 1st in the subtask2 and 2rd of mean rank in the subtask1.

## introduction natural language processing (nlp) is a kind of computational techniques which processes and analyzes large volume of natural language data, such as document text. in the last decade, term frequency-inverse document frequency (tfidf) [wikipedia, ] word vector and word embedding such as word2vec @xcite and glove @xcite are widely used in the nlp tasks which became the default standard features in many nlp tasks, such as text classification. recently, transformer model which utilizes the mechanism of self-attention is considered as a breakthrough for nlp @xcite and computer vision field @xcite . the transformers model has caused the paradigm shift in nlp domain such that pre-trained language models which are applied widely in nlp tasks. pre-trained language model has been gained wide attention after bert achieved state-of-the-art results on a variety of nlp tasks @xcite . openai gpt @xcite , @xcite , @xcite , roberta @xcite , @xcite and xlm @xcite are examples of pre-trained language model (plm) that could be applied to a wide range of nlp tasks. in the finance domain, there is a large volume of document texts to be processed for analysing financial markets, investment support, trading and so on. one of tasks is to classify these document text sentences into proper classification. the word vectors and plms are implemented widely for text classification in the finance text field @xcite @xcite . recently "environment, social and governance (esg)" related issues in the financial domain are gained more and more attention with the goal of building sustainable environment. the aim of the finsim4 shared task @xcite is to elaborate an esg taxonomy (esg related concepts representations) based on the document data like companies' annual reports, sustainability reports, environment reports, etc. and utilizes them to analyse how the economic activity complies with the taxonomy. there are two sub-tasks in the finsim4 task. regarding the sub-task1, there is a number of terms which are selected from esg taxonomy texts. for example, the given terms: "low-carbon", "carbon footprint" et al., these terms are related to the "carbon factor" concept. the goal of sub-task1 is to develop a model to classify correctly a list of given terms from esg taxonomy domain into the most relevant concepts. regarding the sub-task2, there are selected sentences texts from the sustainability reports and other documents about sustainable or unsustainable activities. the aim of sub-task2 is to design a system that can automatically classify the esg taxonomy text sentence into sustainable or unsustainable class. as sub-task1, we make use of on-line data such as wikipedia data to augment the financial terms with terms' definition to be term sentence. moreover, we combine the given dataset composed of financial and non-financial reporting documents files with augmented terms' sentences to train word2vec with the context-free word2vec model. the logistic regression and deep attention model @xcite by inputting word2vec and tf-idf vectors are implemented to predict the concepts of the test terms. as sub-task2, the bert, albert, distil bert, roberta, xlnet are applied to this task. based on the results of the experiments, the proposed models have achieved good performance for each task. section 2 describes the task data and the term augmentation method. section 3 describes our proposed methods for two tasks. section 4 shows experimental configurations and discusses the results. then, we conclude this paper in section 5. ## data description and augmented terms as the sub-task1, the number of training and test set data are 647 and 145 respectively. there are 24 categories of concepts in the training data. the number of each concept in training data is listed as based on the above table, the concept of "energy efficiency and renewable energy" has the largest number of label data. some concepts like the "injury frequency rate"," share capital" and "board independence" have just 2 label data. we found that some concepts are very similar, such as the "injury frequency rate for subcontracted labor" and "injury frequency rate" concepts, "waste management and water" and "waste-water management" concepts which have common words. moreover, some terms are composed of a single word like "strikes", "contraceptives" terms which are not easy to understand the meaning of terms. we augment the terms with terms' definition in the training and test set data. the wikipedia terms' definitions are utilized to describe the meaning of the terms. we take the "recycle" as an example to describe how we augment the term with term's definition. the definition of "recycling" in the wikipedia is "recycling is the process of converting waste materials into new materials and objects". we merge the term word "recycle" and definition of "recycling" with a space as a new sentence: "recycle recycling is the process of converting waste materials into new materials and objects". some terms like "tobacco 5% revenues" which have intuitive meaning, we keep the term words as sentence without adding additional definition. since the number of provided training and test data is still limited for training a good word embedding. we combine the given dataset composed of financial and nonfinancial reporting documents files with augmented term sentences to train the word embedding. there are a total of about 196615 sentences for training word2vec. as the task 2, there are 2265 rows in the training data and 205 rows in the test data. the number of sustainable and unstainable sentences are 1223 and 1042 respectively. ## conclusion this paper mainly presents kaka team how to tackle the fin-sim4 shared tasks. we approach the two tasks using different modes. as sub-task1, we implemented logistics regression and deep attention model with different word vectors. as sub-task2, several plm are implemented with fine tuning to classify the sentences into sustainable or unsustainable class. the experimented result show that our methods could effectively solve the goal of the two tasks. however, our method still needs to be improved to achieve better performance in the following direction. firstly, it is better to do more parameter tuning in each model to improve the accuracy. secondly, as sub-task1, there is significant gap between our score and the best result in the final test leaderboard, we could make more efforts in feature engineer like text similarity for models.