Identifying ESG Impact with Key Information (2023)

Abstract: The paper presents a concise summary of our work for the ML-ESG-2 shared task, exclusively on the Chinese and English datasets. ML-ESG-2 aims to ascertain the influence of news articles on corporations, specifically from an ESG perspective. To this end, we generally explored the capability of key information for impact identification and experimented with various techniques at different levels. For instance, we attempted to incorporate important information at the word level with TF-IDF, at the sentence level with TextRank, and at the document level with summarization. The final results reveal that the one with GPT-4 for summarisation yields the best predictions.

## introduction environmental, social, and governance (esg) factors have been deemed essential for a company's prosperity in the long run and emerged as a crucial consideration for investment and corporate operations @xcite . spontaneously, esg has garnered increased attention among the finnlp community. in 2023 finnlp, in conjunction with ijcai, has proposed a shared task of multi-lingual esg impact type identification (ml-esg-2), releasing a multi-lingual dataset that consists of news articles in four languages -(traditional) chinese, english, french, and japanese @xcite . the objective is to determine if the given news is an opportunity or a risk for the company from the esg aspect. ml-esg-2 presents itself as a text classification problem, which involves extracting features from raw textual data and predicting categories based on such features. research around this topic in recent years centers on the attention mechanism, among others (see @xcite . in particular, transformer models such as bert @xcite are widely exploited, and further encourage the trend of using more data and large language models for text classification tasks @xcite . in the case of long document classification where regular transformers fail, more effective methods have been proposed, mostly involving pre-training another language model for long sequences or extracting key information to feed into the model. for example, @xcite revised the attention mechanism in bert and developed a longformer that increases the input capacity up to 4, 096 tokens, and @xcite proposed cogltx that jointly trains two bert or roberta @xcite models -one for key sentence extraction and the other for the final task. however, a survey by @xcite suggests that complicated approaches don't necessarily bring better outcomes, meanwhile demanding more investment (e.g. longformer requires more gpu memories, and cogltx costs much more runtime). inspired by such findings, we also used pre-trained language models (plms) for the esg task. specifically, we also exploited the chatgpt series as a translation engine for data augmentation and to discern the important information for long document classification. ## related work in the last decade, text classification tasks have gradually embraced the deep learning approach, as it relieves the burden of feature designing. multilayer perceptions @xcite already outperform traditional models such as naive bayes, svm, etc., cnn (convolutional neural network) and rnn (recurrent neural network) further advance the performance in this area @xcite . the gnn (graph neural network) also takes a place but focuses on modeling the structural information within the text @xcite . the introduction of bert @xcite has especially promoted the fashion of ap-plying plms in text classification tasks. compared with previous methods such as tf-idf (rajaraman and ullman, 2011) and word2vec @xcite , plms capture more effective representations and boost performance text classification tasks. however, bert and its variants such as roberta @xcite , bart @xcite , etc., are intrinsically incapable of processing long sequences, and a brutal truncation does not necessarily provide benefits. the predicament sees the appearance of more plms tailored for long sequences. attention-based models like longformer @xcite and big bird @xcite employ sparse self-attention instead of full attention as in the bert series and expand the input capacity up to 4, 096 tokens. hierarchical transformers such as tobert @xcite produce chunk-level representations and thus can take input of any length. ernie-doc @xcite enhances the recurrence mechanism as employed in transformer-xl @xcite and xlnet @xcite , etc. and introduces a retrospective feed mechanism to directly model the text at the document level. another type of approach aims at selecting important sentences from the document for classification, e.g. cogltx @xcite and more traditional approaches such as textrank @xcite . techniques utilizing summarization for classification also fall within this category (e.g. @xcite . it should be noted that sophisticated models such as those described above do not guarantee better performances, as a regular transformer model may surpass them with simple augmentation (see @xcite . sparse attention cannot fully exploit the global information for each segment when modeling long documents; the recurrence mechanism introduces latency @xcite , and hierarchical transformers have the problem of context fragmentation @xcite . ## methods as evident in ## results and discussion the scores on the english set including ours have achieved a high level in general, which can be expected considering that the english task is relatively simple. the tally on the chinese task, on the other hand, shows that our models outperform the others by a notable margin. the models without ensemble learning (see regarding the chinese task, we also investigated other possibilities and reported their evaluation results on the dev set in ## conclusion to recap, we employed a simple architecture for the ml-esg-2 shared task on esg impact type identification and ended up with a fair result. particularly, we employed a summarising technique to address the document classification problems as in the chinese track with the widely popular ai bot - @xcite as a text summarizer. note that the summarization-based approach is a consequence of multiple experiments. before settling down on summarization, we investigated the influences of other components including news headlines, key sentences, and words. the results reveal that the key formation as such is useful for text classification. our method turns out to be effective in that gpt-4 captures the essential meaning of the texts. however, we failed to compare the summarization performance of gpt-4 and other possible methods, nor did we examine other approaches to keywords or key sentence extraction besides textrank @xcite and tf-idf (rajaraman and ullman, 2011). the evaluation results show that there is still room for improvement in the chinese task. a further and deeper investigation could produce some more sparkles and lead to more interesting findings.