Transformers-Based Approach for a Sustainability Term-Based Sentiment Analysis ( STBSA ) (2022)

Abstract: Traditional sentiment analysis is a sentence level or document-level task. However, a sentence or paragraph may contain multiple target terms with different sentiments, making sentiment prediction more challenging. Although pre-trained language models like BERT have been successful, incorporating dynamic semantic changes into aspect-based sentiment models remains difficult, especially for domain-specific sentiment analysis. To this end, in this paper, we propose a Term-Based Sentiment Analysis (TBSA), a novel method designed to learn Environmental, Social, and Governance (ESG) contexts based on a sustainability taxonomy for ESG aspect-oriented sentiment analysis. Notably, we introduce a technique enhancing the ESG term’s attention, inspired by the success of attention-based neural networks in machine translation (Bahdanau et al., 2015) and Computer Vision (Bello et al., 2019). It enables the proposed model to focus on a small region of the sentences at each step and to reweigh the crucial terms for a better understanding of the ESG aspect-aware sentiment. Beyond the novelty in the model design, we propose a new dataset of 125,000+ ESG analyst annotated data points for sustainability term based sentiment classification, which derives from historical sustainability corpus data and expertise acquired by development finance institutions. Our extensive experiments combining the new method and the new dataset demonstrate the effectiveness of the Sustainability TBSA model with an accuracy of 91.30% (90% F1-score). Both internal and external business applications of our model show an evident potential for a significant positive impact toward furthering sustainable development goals (SDGs).

## introduction in 2015, the united nations (un) adopted the 2030 agenda and its 17 sustainable development goals (sdgs; nations (2015)), addressing global challenges including poverty, inequality, climate change, environmental degradation, peace, and justice. the secretary general's roadmap for financing this collective and transnational effort invites all stakeholders to consider environmental, social, and governance (esg) issues. esg matters have assumed relevance for investors, regulators, and industry participants, while esg criteria are increasingly used to measure the impact of investment activities on sustainable development. however, esg-integrated investing remains challenging, even for world-class asset managers, institutional investors, and pension funds, because of data gaps in coverage of emerging markets and a lack of analytical capacity. further, these markets present the greatest opportunities for investors to achieve impacts through the sdgs because their development needs are the most significant. at the same time, there is growing recognition of the fundamental role played by data, primarily structured data, in achieving the objectives set out in the sdgs @xcite @xcite . structured data and sdg metrics are essential to ensure the successful design of local projects but are often absent when required for insights into beneficiaries' needs and values @xcite . unstructured data can provide such insights. natural language processing (nlp) techniques can process such qualitative data to provide relevant facts and figures to project developers. expected benefits are time and cost reductions, higher operations efficiencies, due diligence improvements, and better sustainability impact assessments @xcite @xcite . recent progress in masked language modeling such as google bert (bidirectional encoder representations from transformers, @xcite , roberta (robustly optimized bert approach @xcite ), and deberta (decodingenhanced bert, @xcite )-combined with cloud computing, is unlocking the potential for creating analytical capacity to assess unstructured data at scale and is facilitating sdg-aligned financing for emerging markets to address the $4.2 trillion usd annual shortfall in investments needed to meet the sdgs (oecd, 2020). despite these advances, nlp research and applications that contribute to sustainable development are absent @xcite . this gap is attributed to the lack of high-quality sustainability data and the scarcity of relevant labeled data to train sustainability-domain language models. our work proposes a sustainability-domain adaptation of transformer-based models to perform various nlp tasks, such as esg term extraction and sentiment analysis. such a sustainability domain-specific language model is a significant advance; pre-trained models and commercial sentiment analysis solutions perform poorly at predicting esg sentiments because of differences in domain-specific vocabulary (these models are trained using datasets such as restaurant or movie reviews or tweets that are not relevant to sustainability analysis). domain-specific models are also necessary to process sustainability reporting documents which are typically lengthy, complex, and use terms that do not carry emotional connotations, unlike movie or restaurant reviews. hence the need to create a specific taxonomy for context-based esg sentiment analysis @xcite . development finance institutions have decades of archival sustainability data created from project due diligence and monitoring. we use examples of such data to create a unique esg taxonomy and human-annotated dataset. namely, we equip two pre-trained language models (roberta and de-berta) to understand esg context by fine-tuning and modifying the models into a sustainability termbased sentiment analysis (stbsa) model, thereby creating a new approach based on an esg taxonomy of more than 1,200 terms. we then train the models with human-annotated data to predict the context of esg terms in sentences and classify words by positive, negative, or neutral esg sentiment. significantly, our experiments find that the stbsa model (based on roberta) performs with 91.30% accuracy (90% f1-score) and outperforms the current state-of-the-art baseline models for sentiment analysis tasks. target-based or target sentiment analysis is another task that identifies polarity towards a target entity, as opposed to over an entire volume of text @xcite @xcite @xcite . @xcite were the first to propose targeted sentiment analysis on twitter. they demonstrated the importance of targets by showing that 40% of sentiment errors are due to not considering them in classification. however, this task only identifies the overall sentiment, and the existing corpora consist only of text with one single entity per unit of analysis. this task caters to more generic text by making fewer assumptions while extracting fine-grained information. esg-domain transformers-based models. in recent years, transformer-based models have become the default solution for nlp tasks such as search, machine translation, or sentiment analysis @xcite . only a few studies apply language models to the sustainability area. cli-matebert, proposed by @xcite , analyzes companies' climate risk using the task force on climate-related financial disclosures framework. another application, developed by @xcite , is an artificial neural network classifier for modeling environmental impact statement documents from the us environmental protection agency. finally, @xcite demonstrate that fine-tuning bert using large amounts of business and financial news data from the reuters news archive led to better results with classification tasks such as detecting esg controversies. terms-based sentiment analysis. term-based sentiment analysis is particularly valuable in domain-specific text, which very much resembles how a human domain expert comprehends this text content. domain-specific text such as sustainability reporting documents are very complex, often ambiguous, and may have multiple target terms in a single sentence. moreover, the same terms may have different meanings or polarity depending on the context in which they appear @xcite , demanding a different approach. @xcite show that previous methods for aspectbased sentiment models are unable to achieve the same performance as human-level sentiment understanding. additionally, @xcite argue that basic encoder-decoder architecture with a fixed-length vector is a bottleneck in improving those models' performance. inspired by the above research, both aspect-based sentiment and transformers-based architectures, we proposed a novel architecture that addresses the issue of long and complex sentences by expending the absa to emphasize parts of a source sentence that are relevant to predicting esg sentiment. ## methodology most aspect-based sentiment analysis methodologies comprise multi-grained nlp tasks and consist of two major subtasks: target term extraction and sentiment classification @xcite . accordingly, this section introduces our approach for esg terms selection and extraction and presents the model design for conducting esg sentiment classification. ## discussion and conclusion in this paper, we proposed a novel approach to realize a term-based sentiment analysis built on a unique esg taxonomy to address the limitations of the aspect-based sentiment analysis models and off-the-shelf sentiment analyzers for sustainabilitydomain applications. furthermore, using historical sustainability corpus data and expertise from a development finance institution (ifc), we produced an unprecedented human-annotated dataset of 125,000+ sentences for esg sentiment classification. the subsequent experiments demonstrated the effectiveness of this model with an accuracy of 91.3% and a 90% f1-score, outperforming the current state-of-the-art baseline models by over 20 points @xcite . our stbsa model addresses three challenges. first, it offers a new model design with capabilities to handle multiple target terms and different sentiments by leveraging an esg domainspecific taxonomy with more than 1,200 esg risk terms. recent studies underscored the difficulties of developing sustainability domain-specific taxonomies @xcite @xcite , which are blockers to building more efficient and better-performing models. second, it proposes an unprecedented sustainability domain nlp model, which yields a far higher performance (91.3% accuracy, 90% f1-score) than baseline models such as finbert @xcite or similar studies such as the ones presented by @xcite or @xcite with 70% and 75% accuracy respectively. our model fills a critical research gap in the nlp literature. third, for investors in emerging markets, it offers the potential to enhance esg due diligence and impact assessments resulting in a positive impact for green investments and contributing to achieving the un sdgs. these findings, while promising, have limitations and create opportunities for future research. first, the model can only understand and predict esg sentiment in english (about 75% of the corpus). there are obvious benefits to expanding its understanding to additional languages such as french, mandarin, portuguese and spanish. second, as our stbsa model is derived from "black box" systems, the explainability and transparency framework proposed in this paper needs to be fully implemented to enable users to understand its design, operation, and biases, and to trust its predictions. this paper emphasizes data-driven ai and keeping humans in the loop and proposes a new multi-stakeholder framework for operationalizing ai systems. it is essential to ensure that complex and computationally heavy models, such as illustrated in this paper, do not penalize developing countries with limited data, leading to model biases @xcite . this awareness may help mitigate underlying word embeddings biases of pretrained language models associated with specific demographics such as gender, ethnic minorities, and local communities @xcite . this paper provides a first but decisive step toward further research at the intersection of nlp and esg. we intend to partially release the model and esg-annotated data as a public good to enable a strong baseline for sustainability domain research, given its major value for the research community either to replicate our approach or to stimulate further research. we hope the results and dataset inspire the nlp and sustainability research communities to actively explore how advanced language modeling can be applied to esg and impact data to support creating solutions furthering the sdgs.