Using Transformer-based Models for Taxonomy Enrichment and Sentence Classification (2022)

Abstract: In this paper, we present a system that addresses the taxonomy enrichment problem for Environment, Social and Governance issues in the financial domain, as well as classifying sentences as sustainable or unsustainable, for FinSim4-ESG, a shared task for the FinNLP workshop at IJCAI-2022. We first created a derived dataset for taxonomy enrichment by using a sentence-BERT-based paraphrase detector (Reimers and Gurevych, 2019) (on the train set) to create positive and negative term-concept pairs. We then model the problem by fine-tuning the sentence-BERT-based paraphrase detector on this derived dataset, and use it as the encoder, and use a Logistic Regression classifier as the decoder, resulting in test Accuracy: 0.6 and Avg. Rank: 1.97. In case of the sentence classification task, the best-performing classifier (Accuracy: 0.92) consists of a pre-trained RoBERTa model (Liu et al., 2019a) as the encoder and a Feed Forward Neural Network classifier as the decoder.

## introduction taxonomies classify, categorize and organize information hierarchically and are typically designed and curated by domain experts. they require frequent manual and automated updates to capture a domain sufficiently and to be considered complete. however, it is not to feasible to manually edit taxonomies to reflect changing concepts and evolving human knowledge. the taxonomy enrichment task helps address this problem by developing methods to add new terms to an existing taxonomy. the finnlp shared task 1 defines this problem on a esg taxonomy. given a list of concepts and terms, the task is to rank the concepts given the term. in case of shared task 2, we are asked to classify a given sentence from sustainability reports and other documents as either sustainable or unsustainable. in approaching these problems, we leverage large-scale pre-trained language models for token and sentence representations. we explore transfer learning through transformer models like bert @xcite , distillbert @xcite , roberta @xcite as well as generative text to text transformers like t5 @xcite especially since training data is very limited for both tasks. like most nlp tasks in fintech, the task 1 has limited amount of data. we addressed this limitation by creating a dataset derived from the train set and used a paraphrase detector to create positive and negative instances @xmath0, @xmath1. we then fine-tune sentence-bert @xcite on this derived dataset and use it as the encoder in our model. the decoder is a logistic regression classifier. this gives us a tenfold cross-validated accuracy of 0.89 on the train set. however at test time, the performance varies and resulting accuracy is 60.6%. we describe the different approaches to modeling this problem that led to this final system and hypothesize reasons for the train-test performance discrepancy in the final system. shared task 2 is a binary sustainability classification task. we experimented with various models starting with a tf-idf based classifier to transformer based roberta @xcite based classifier. the roberta based model resulted in a ten-fold cross-validated accuracy of 0.96 and test-set accuracy of 0.92. ## data description the training dataset for sub-task 1 contains 646 annotated term-concept pairs. the total number of unique concepts are 25. the training dataset for sub-task 2 contains 2265 annotated sentences. ## sentence classification in sub-task 2, we holdout 20 percent of the data (463 instances of 2265) as validation set to evaluate performance of our various approaches and finetune the hyperparameters. we use rest of the data for training. we have built the following systems for sub task 02: ## test submission as part of the shared task, we have made submissions to both the subtasks. our team name is jetsons and we have presented the results of our systems from both sub tasks in the ## conclusion in this paper, we presented our submission to the sub tasks of finsim4-esg. we first present a system that addresses the taxonomy enrichment problem for "environment, social and governance" issues in the financial domain. we first created a derived dataset for taxonomy enrichment by using a sentence-bert-based paraphrase detector to create positive and negative term-concept pairs. we employ a logistic regression classifier as the decoder, resulting in test accuracy: 0.6 and avg. rank: 1.97. we then present our approach to the sub task of sentence classification. our best performing model, a finetuned version of roberta model achieves 96 percent on validation set and 92.3 on test set.