TCS WITM 2022@ F in S im4- ESG : Augmenting BERT with Linguistic and Semantic features for ESG data classification (2022)

Abstract: Advanced neural network architectures have provided several opportunities to develop systems to automatically capture information from domain-specific unstructured text sources. The FinSim4-ESG shared task, collocated with the FinNLP workshop, proposed two sub-tasks. In sub-task1, the challenge was to design systems that could utilize contextual word embeddings along with sustainability resources to elaborate an ESG taxonomy. In the second sub-task, participants were asked to design a system that could classify sentences into sustainable or unsustainable sentences. In this paper, we utilize semantic similarity features along with BERT embeddings to segregate domain terms into a fixed number of class labels. The proposed model not only considers the contextual BERT embeddings but also incorporates Word2Vec, cosine, and Jaccard similarity which gives word-level importance to the model. For sentence classification, several linguistic elements along with BERT embeddings were used as classification features. We have shown a detailed ablation study for the proposed models.

## introduction sustainability disclosures have started gaining traction in financial world. sustainability disclosures reflects the procedures that an organization follow to fulfill its commitment towards environment, social and governance (esg) factors. investors are increasingly considering the esg commitments of organizations to aid their investments decisions. esg information is not commonly part of financial reporting but organizations have started making esg disclosures either as a part of their annual report or as a separate sustainability report altogether. recently, european union has proposed new guidelines making esg disclosures mandatory for companies providing esg driven investment products. companies in eu must comply with increasing obligations related to esg in order to have their businesses qualify as sustainable and to be noticed by investors on the eu market. several institutions, such as the sustainability accounting standards board (sasb), the global reporting initiative (gri), and the task force on climate-related financial disclosures (tcfd) are also working to form standards and define standards to facilitate incorporation of esg factors into the investment process. organization need to align their disclosures to various esg taxonomies available to facilitate effective utilization of the disclosures. since, esg domain and the allied taxonomies are still evolving, there is a need to have automated systems that can elaborate the existing taxonomies and also aid the organization to align their disclosures to the existing standards. the finsim4-esg shared task is one of the few attempts that primarily focused on automatically learning effective and precise semantic models adapted to the esg domain. specifically it addressed the task of automatic categorization of esg terms into pre-defined categories to enhance the existing taxonomy and to create a language model that can understand the language of sustainability and segregate sustainable sentences from unsustainable ones. the specific details of the shared task is described in section 3. in this paper, we propose to augment the transformer based bert architecture with semantic similarity features to address the esg term classification task. the base bert model is first fine-tuned on a set of esg document to facilitate capture of esg language context. the embedding obtained from the resultant model is then augmented with semantic similarity features like word2vec, cosine and jaccard similarity to perform the classification task of segregating domain terms into fixed number of class labels. in a similar way, for sentence classification task, various lexical features are used along with bert embeddings. dimensionality reduction techniques are also incorporated into the experiments. for training the model, we have used the bert base architecture. we have conducted multiple experiments with the model architecture such as taking combination of proposed features with different classifiers in both sub-tasks. our experiments shows that a combination of esg fine tuned bert with word2vec, cosine and jaccard similarity with a logistic regression classifiers gives the best result for term classification. for sentence classification we exploited linguistic and nlp-based features like presence of specific named entities (like organisation, date), word-count in the document, etc. our experimental results showed that when we combine these features along with finetuned bert-base classifier, it lead to an increase of about 2% in system's accuracy. ## related work rising sustainability awareness amongst consumers, investors and regulators is forcing organizations to pay attention towards environment, social or governance (esg) factors. researchers around the world have conducted experiments to study the effect of esg factors on financial performance of organizations. the last three editions of the finsim proposed the challenge to automatically learn effective and precise semantic models for the financial domain @xcite @xcite . @xcite solve the financial hypernym detection task by using the logistic regression classifier trained on a combination of word embeddings, semantic and string similarity metrics and won the challenge. @xcite approached the task as a semantic textual similarity problem. they used a siamese network with pre-trained language model encoders to derive term embeddings and computed similarity scores between them while @xcite leverage the use of given documents by extracting sentences corresponding to each term and then used a transformer based bert architecture to perform a classification task. a number of approaches like use of publicly available knowledge graphs to generate explicit features @xcite , customize word embeddings @xcite , word ontology (tian and chen, 2021), pre-trained sentence embedding extracted from universal sentence encoder along with cosine similarity (anand et al., 2020) and use of static word embeddings @xcite have also been proposed to automatically map financial concepts with its most relevant figure 2: sub-task2 data distribution hypernym. ## shared task detail and dataset description the finsim4-esg 2022 task focused on elaboration of an esg taxonomy (esg related concepts representations) based on the data like companies sustainability reports, annual reports, and environment reports, and make use of them to analyze how an economic activity complies with the taxonomy. the current edition proposed two sub-tasks : ## proposed approach the proposed approach can be divided into three main stages namely feature extraction stage, dimensionality reduction stage and classification stage as shown in figure 3 . for both sub-tasks, the input text is first subjected to feature extraction module where different nlp and text mining techniques are applied to obtain various linguistic and semantic features along with bert embeddings. a combination of these features are then subjected to dimensionality reduction where a subset of features are selected as final input variables for classification module. the detailed description of the activities in each stage is explained next. 4.1 feature extraction for sub-task1 • esg bert -bert (bidirection encoder representation from transformers) is a state of the art model for language modeling (devlin et al., 2018). the official bert repository foot_0 contains various pre-trained models that can be further used for the downstream task. in this work, esg-bert model foot_1 pre-trained on sustainability corpora has been utilised to achieve better capability of understanding domain specific vocabulary. the model is fine-tuned further on the downstream task of concept classification as shown in figure 4. experiments show that the fully connected (fcn) layer added between the classification layer and bert output layer benefited the model in learning the representation for the downstream task. the output of the fcn layer represents the term by an embedding vector. this vector has been used as one of the feature in our system for the classification. a tsne-plot of esg bert embedding of terms in training data is shown in figure 5. the cluster segregation for different concepts can be seen clearly in the figure which shows the potential of esg-bert based embedding in classifying esg concepts. dimensionality reduction with principal component analysis (pca) -the combination of embeddings and features used in our experiments results into large number of features as compared to number of samples. the performance of any machine learning model can suffer from the curse of dimensionality where the number of features becomes larger than the number of samples in the dataset. to tackle this issue, we have empirically reduce the dimensions of the features using principal component anaysis (pca) @xcite and found that reduction to 200 dimensions are giving the best results. ## metrics the most important part of any system is to choose the most accurate model. the metrics used for the evaluation is provided by the organizers. for sub-task1, the evaluation metric used is accuracy and mean rank. the model is required to predict the concepts in the ranking order (from the most probable to the least probable) for each term. the accuracy and mean rank is defined as follows - note that rank i corresponds to the rank of the correct label. for sub-task2, the evaluation metric considered is accuracy and recall. recall (also known as sensitivity) is the fraction of relevant (here sustainable) instances that were correctly retrieved. ## experiments and results sub-task1 -we were provided with 641 data samples which included terms and their corresponding concepts for model development and validation. we had split the data in 80:20 train-validation split using 5 fold cross validation technique which results into 5 sets containing 512 data points for training and 129 data points for validation. all proposed models were trained and validated on the generated training and validation dataset respectively. the test set provided contained a list of 145 terms without their concepts in json format. baseline 1 is a distance-based classifier using custom embeddings @xcite whereas baseline 2 uses logistic regression classifier over these custom embeddings @xcite given in finsim4-esg shared task. bert (pre-trained on general english corpora 7 ) based classification model is fine-tuned on the downstream task by adding a classification layer (cl). it can be seen from the ## conclusion as part of finsim4-esg shared task on learning semantic similarities for financial domain, we attempt to solve the problems of finding the most relevant esg concept for each given domain term and classify a given sentence into sustainable or unsustainable based on provided training data. in sub-task1, in order to utilize contextual word embeddings along with sustainability resources to elaborate an esg taxonomy, we proposed the use of semantic similarity features along with bert embeddings to segregate domain terms into a fixed number of class labels. for sub-task2, several linguistic and nlp features along with bert embeddings were used for classification. in our experiments we observed that linear classifier models like logistic regression performs the best. we have also compared our models with the given baseline accuracy and found that their performance is far superior to it. we have also shown a detailed ablation study for the proposed models. in future, we are planning to use better data augmentation techniques and explore the possibility of using the ontology hierarchy and definitions available as part of the eu taxonomy.