The Risk and Opportunity of Data Augmentation and Translation for ESG News Impact Identification with Language Models (2023)

Abstract: This paper presents our findings in the ML-ESG-2 task, which focused on classifying a news snippet of various languages as “Risk” or “Opportunity” in the ESG (Environmental, Social, and Governance) context. We experimented with data augmentation and translation facilitated by Large Language Models (LLM). We found that augmenting the English dataset did not help to improve the performance. By fine-tuning RoBERTa models with the original data, we achieved the top position for the English and second place for the French task. In contrast, we could achieve comparable results on the French dataset by solely using the English translation, securing the third position for the French task with only marginal F1 differences to the second-place model.

## introduction esg factors have gained increasing prominence in recent years, not only among stakeholders but also in the decision-making processes of investors and financial institutions. as the awareness of esg risks grows, so does the need for precise and realtime classification of these risks. traditional esg risk analysis has largely relied on structured data, such as company disclosures, financial reports, and pre-defined esg metrics. however, these sources often provide an incomplete and lagging view of a company's esg footprint. moreover, they are subject to reporting biases and may lack granularity in capturing the diverse dimensions of esg risks. the proliferation of online news media offers a fertile ground for harvesting a more comprehensive set of data on esg issues. news articles, in particular, often capture real-time events, public sentiment, and expert opinions, providing a more immediate and multifaceted perspective on esg risks than can be obtained from traditional structured data. yet, leveraging this unstructured textual data to accurately classify esg risks presents computational challenges. these challenges include but are not limited to, natural language understanding, sentiment analysis, and the development of a robust taxonomy for esg risk classification. the finnlp-2022 workshop introduced a finsim4-esg foot_0 shared task that centres on esg issues. to deepen the understanding of these areas, finnlp@ijcai-2023 released a new dataset for the finnlp community. this dataset is designed to explore the task of identifying key esg issues in multiple languages, guided by the msci esg rating framework, which includes 35 key issues for categorisation. expanding on this discourse, a new task dubbed multi-lingual esg impact type identification (ml-esg-2) was introduced. the primary objective of this task is to identify the type of esg impact a given piece of news may have. specifically, models are tasked with determining whether the news presents an esg-related opportunity or risk. this aspect of impact identification is structured as a single-choice question. in this study, we present our methodology for tackling the ml-esg-2 shared task, using datasets in both english and french. we conducted our experiments using two primary methods: fine-tuning language models on the dataset and training a logistic regression model using sentence-bert (sbert) embeddings @xcite . our findings suggest that the optimal approach can be achieved solely by relying on the given datasets, without the need for any data augmentation. furthermore, we found that comparable results could be obtained on the french dataset by first translating the text into english and then employing a model pre-trained specifically on english text. using this strategy, our models secured first place in the english language dataset and second and third places in the french language dataset. ## datasets in ml-esg-2, a new task called "esg impact type identification" was introduced to advance discussions on esg ratings from the previous shared task @xcite . this task requires models to discern whether a given news article signifies an esg "opportunity" or "risk". each data point consists of the url, title, content, and the assigned impact type for a given article. an overview of the english and french datasets can be seen in ## methods we conducted experiments using two primary methods: fine-tuning language models on the dataset and training a logistic regression model using sbert embeddings @xcite . given the absence of development sets and the presence of imbalanced training data, we em-ployed 5-fold cross-validation to assess the effectiveness of our approaches. additionally, we observed that the news titles are not unique and two news contents with the same title can be both "opportunity" and "risk". thus, we decided to disregard the titles altogether and used only the content as the input. baseline. to demonstrate the performance improvements achievable through the two methods mentioned earlier, we initially established a simple baseline. this benchmark was created by employing tf-idf and logistic regression using scikitlearn @xcite . we retained the top 1000 features identified by tf-idf and conducted hyperparameter optimisation (see appendix a.3). fine-tuning language models. we experimented with three well-known pre-trained encoder models: distilbert @xcite , roberta @xcite , and deberta @xcite . specifically, the model names in hugging face @xcite were distilbert-base-uncased, roberta-large, microsoft/deberta-v3-large. we also experimented with the xlm-roberta (xlm-roberta-large) model @xcite for the french dataset. we fine-tuned the pre-trained models on the ml-esg-2 dataset and used optuna @xcite to find the optimal hyperparameters of each model. the final list of the hyperparameters is shown in ## results our experiments revealed several key findings in the context of model performance. firstly, roberta demonstrated outstanding performance on the english dataset, achieving an impressive f1 score of 85.19% ( when subjected to the test set, roberta continued to shine, outperforming st5-xxl by a significant margin. roberta achieved an f1 score of 92.00%, while st5-xxl scored 75.36% ( shifting our focus to the french dataset, our cross-validation results favoured st5-xxl (@xmath0, with an f1 score of 76.52% versus 72.17% ( furthermore, the test results between xlm-roberta and st5-xxl differ only slightly (<1%). note that xlm-roberta was trained on the original french dataset, but st5-xxl used the translated dataset. the results for st5-xxl raise the potential of translating other languages to english and used english-based models for text classification. ## related work in recent studies @xcite , it was discovered that relatively compact specialised clinical models exhibit significantly superior performance compared to all in-context learning approaches when applied to llms. this superior performance holds true even when these clinical models are fine-tuned on a limited amount of annotated data. additionally, their research revealed that pretraining on clinical tokens enables the development of smaller, more parameterefficient models that can either match or surpass the performance of much larger language models trained on general text. on a similar note, septiandri et al. ( 2020 ) found that classical nlp techniques, i.e. bag-of-words and tf-idf, could produce comparable results to the more advanced word2vec @xcite and bilstm with a fraction of the training time. the study focused on a binary classification task, similar to ml-esg-2. they suggested that even though the tiny improvement from complex models is crucial in a competition, one should consider allocating more resources to improve the quality of the dataset in practical settings. ## conclusion in summary, within the english sub-task of ml-esg-2, the roberta model fine-tuned only with the original data secured the top position, even without any particular strategy to address the class imbalance. our investigation revealed that data augmentation failed to improve f1 scores on the training set. similarly, adding translated french news for english models did not contribute to improved performance. from these findings, we deduce that increasing data quantity through augmentation and translation may not consistently benefit model performance. for the french sub-task, we observed that translating to english provided better results for the training set. however, the multi-lingual roberta model (xlm-roberta) fine-tuned on the original french dataset achieved higher f1 for the test set, albeit slightly. these results indicate an opportunity for future works: the potential of translating text to english as a preprocessing strategy in other languages.