Knowledge informed sustainability detection from short financial texts (2022)

Abstract: There is a global trend for responsible investing and the need for developing automated methods for analyzing and Environmental, Social and Governance (ESG) related elements in financial texts is raising. In this work we propose a solution to the FinSim4-ESG task, consisting of binary classification of sentences into sustainable or unsustainable. We propose a novel knowledge-based latent heterogeneous representation that is based on knowledge from taxonomies and knowledge graphs and multiple contemporary document representations. We hypothesize that an approach based on a combination of knowledge and document representations can introduce significant improvement over conventional document representation approaches. We consider ensembles on classifier as well on representation level late-fusion and early fusion. The proposed approaches achieve competitive accuracy of 89 and are 5.85 behind the best achieved score.

## introduction in this work we develop a knowledge-backed approach for the detection of sustainability on premises of a given short textual document (i.e. a sentence). more specifically, we propose a solution to the shared task of the finsim4-esg workshop, where the task is to classify a given sentence extracted from a company financial report as either sustainable or unsustainable. investors have ever-increased interest in the assessment of the environmental, social and governance (esg) criteria, as non-financial factors describing the company's position on social well-being @xcite . esg criteria cover a company's environmental impact (environmental), their relationships with their community including employees, suppliers and customers (social), and their leadership structures including executive pay, shareholder rights, audits and controls (governance). these esg factors are usually reported as a structured output in the companies annual reports. the companies reporting these factors have moved from only a dozen in the 1990s to more than 6000 in 2014 @xcite . in a study by @xcite using survey data from mainstream investment organizations, the authors provide insights into why and how investors use reported esg information and highlight that relevance to investment performance is the most frequent motivation, followed by client demand, product strategy, and then ethical considerations. also social media impacted the way of interaction between companies, employees and potential customers. publishing posts that reveal a certain way of operation that leads to company's miss-behaviour can have a wild response from the customers. aula @xcite studied the impact of social media on the reputation risk and the ambient publicity. an instance of such event is the h&m's thrash-gate scandal -where a company producing and selling clothes was charged of damaging and disposing them as waste instead of reusing them. the story sparked a public outrage and a sever impact on the company. recently @xcite highlighted the high correlation of the company's volatility on the market based on the esg factors. these works showcase how social monitoring of posts can be a powerful asset in the way of achieving a more sustainable and environmentally friendly companies and market. the field of natural language processing (nlp) has seen increased interest in esg-related automated analysis. for example, @xcite propose fine-tuning a generic bert model @xcite on esg corpus (esg-bert), and use this model for detecting positive or negative change in companies stock values based on the related sections of their 10-q filings. @xcite showed that esg ratings can be used to predict market reactions to esg news, particularly when there is disagreement amongst raters. the remainder of this work is structured as follows: section 2 describes the related work, section 3 presents the dataset. section 4 presents the proposed method, followed by the results in section 5 and the final remarks and conclusions in section 6. to be able to learn to classify the documents, initial approaches focused on lexicons or used machine learning techniques. for the financial domain, the collection of financial dictionaries by @xcite has been widely used. later, machine learning approaches have been proposed where models need a numerical representation of documents as input. initial methods for document classification relied on hand-crafted features or on word frequency counts using various weighting schemes (e.g. tf-idf [sammut and webb, 2010]). for example, @xcite represent past annual reports with tf-idf weighted word stems and various feature selection methods in order to predict return on equity (roe) ratio classes with a linear svm classifier. weighted (tf, tf-idf and logarithm damp weighting) unigrams and bigrams are used as features in a study by @xcite , where a support vector machine for regression (svr) with linear kernel is trained to predict volatility of stock returns. @xcite use a linear svm classifier to predict subsequent performance based on narrative parts of 10-k reports, based on both word-level and document-level features. the democratisation of neural-networks introduced denser and more robust document representations, where the models from this paradigm are tasked to predict the next word or the missing word in the sequence. contemporary state-ofthe-art models such as bert @xcite are based on the transformer architecture @xcite . this model learns to generate document representations by being pre-trained on a big corpora from a general domain on the task of masked language modeling, where a portion of the corpora is masked and the model is tasked to predict the words missing. the pre-trained model is then fine-tuned on data from a downstream task such as document classification; this is the transfer learning setting. in this study, we utilize two different variants of the bert model family: finbert [yang et al., 2020], a model pre-trained on financial data, and @xcite , a model that modifies the initial bert learning paradigm by taking into account background knowledge. taxonomies and ontologies are increasingly used for machine reasoning over the last few years. in our study we use tax2vec @xcite which is based on knowledge derived from taxonomies, aiming at improving short documents classification. recently @xcite studied the inclusion of knowledge graphs as banks of large factual knowledge. in their work, they have proposed heterogeneous representation ensembles that are based on knowledge graphs and contextual and non-contextual document representations. these proposed representations achieve nearly state-of-the art results on various tasks such as classification of short texts in the scope of the depression detection from short documents (social media posts) @xcite . in the financial domain, automatic classification of a given list of financial terms against a domain ontology was proposed in the scope of finsim2 @xcite . in terms of esg-related nlp, in addition to esg-bert by @xcite , @xcite studied the effect of the environmental performance of a company on the relationship between the company's disclosures and financial performance, @xcite focused on automated esg scoring, while @xcite performed a diachronic analysis of esg terms in uk annual reports. ## data the shared task consisted of two phases: development of methods and official evaluation. in the first phase the organizers released 2265 training documents. for our internal evaluation purposes we created custom splits of the data into 1812 (80%) documents for training, 226 (10%) for development and 227 (10%) for testing. we give description of the data in ## methodology in this section, we present the different methods we used to generate sentence representations. we classify them into 3 categories: standalone, which are either knowledge or textbased, high-level, which are ensembles of representations and models learned on top of the standalone, and fine-tuned bert models. ## results in this section we report the results of our internal evaluation (with our own splits) together with the final evaluation using the test set of the shared task. ## conclusions and further work in this work we developed a system for classification of esg sentences. we used two representation paradigms: text-based and knowledge-based. in the text-based approaches we finetuned two bert variants: linkbert and finbert. on top of the standalone representations we built ensembles on two different verticals: at the representation level, where we concatenated the representations and transformed them into a new latent space via svd, and at the model level, where we stacked various models together for prediction of final labels. our models scored competitively good, achieving nearly 89% in terms of accuracy. for further work, we consider training deep neural networks on top of the sentence representations to obtain more expressive deep representations that would improve classification performance. we also consider performing feature importance analysis on the representation-level ensembles, to see how representations in the heterogeneous stacks affect the classification on instance level. we also want to include domain-specific knowledge graphs or ontologies and explore their impact on the performance of the models. we also consider using background knowledge as a source for data augmentation, since for various use cases it contributes to better performance @xcite @xcite . finally, we want to perform recursive dimensionality reduction to produce better fused document representations.