Large Language Model Adaptation for Financial Sentiment Analysis (2023)

Abstract: Natural language processing (NLP) has recently gained relevance within financial institutions by providing highly valuable insights into companies and markets’ financial documents. However, the landscape of the financial domain presents extra challenges for NLP, due to the complexity of the texts and the use of specific terminology. Generalist language models tend to fall short in tasks specifically tailored for finance, even when using large language models (LLMs) with great natural language understanding and generative capabilities. This paper presents a study on LLM adaptation methods targeted at the financial domain and with high emphasis on financial sentiment analysis. To this purpose, two foundation models with less than 1.5B parameters have been adapted using a wide range of strategies. We show that through careful fine-tuning on both financial documents and instructions, these foundation models can be adapted to the target domain. Moreover, we observe that small LLMs have comparable performance to larger scale models, while being more efficient in terms of parameters and data. In addition to the models, we show how to generate artificial instructions through LLMs to augment the number of samples of the instruction dataset.

## introduction natural language processing (nlp) has become an increasingly important field in the financial industry, with applications ranging from sentiment analysis and named entity recognition to question answering. information retrieved using machine learning from financial reports, news or posts in social media can be used as indicators of companies' performance or as insights of a market. many industry actors are interested in extracting this information to use it as a resource that can provide them * corresponding author. with a competitive advantage, such as firms forecasting internal future benefits and losses, investors extracting differential information for trading purposes or any practitioner interested in tracking financial assets. nevertheless, some characteristics of financial text make these tasks especially challenging for models that have been trained on general domain data. the use of specific terminology along with the high complexity of the documents, leads these generalist language models to underperform on financial tasks, which suggests that domain adaptation might be required to improve accuracy of interpretation and analysis. furthermore, the rapid evolution of large language models (llms) and their proven capabilities for nlp tasks has made them stand out and become an interesting option to study. due to the fact that even the best general language models fall short for some financial tasks, some proposals have been recently presented for a financial domain adaptation of llms. these models tailored for finance, such as bloomberggpt @xcite , have been introduced as multitasking generative models specifically designed for financial text understanding and generation. however, these fine-tuned models still show room for improvement, both in performance and in the efficiency of the proposed training strategies. this paper tackles various aspects of adapting llms to the financial domain. in particular, we explore diverse strategies of domain adaptation and fine-tuning of llms for financial sentiment analysis, and conduct a series of experiments over two different foundation models. the study focuses particularly on smaller manageable models, up to 1.5b parameters, in order to explore the possibilities of models that can be accessible with relatively low hardware requirements. although the adapted models are smaller than the current state-of-the-art ones, results show that they achieve similar or higher performance. in addition, a curated data collection with two main datasets is also presented. one constructed with financial documents and reports, and the other a set of instructions for financial tasks. we show, step by step, the process of creating these datasets and particularly focus on the use of more powerful llms to generate synthetic instructions to fine-tune smaller llms. finally, apart from the main focus of the study which is on financial sentiment analysis, other tasks have also been evaluated to analyze the multitasking capabilities of our models. ## related work sentiment analysis is one of the most common use cases of nlp. in this task, a model classifies a text according to the sentiment detected, usually between positive, negative and neutral. however, while any sentiment analysis model would be capable of undertaking financial sentiment analysis, an adaptation is required. in this section, the evolution of sentiment analysis in the financial domain is studied using models based on transformers @xcite . finbert foot_0 @xcite is based on the idea of training a bert @xcite model in two steps to adapt it to the financial domain and the sentiment analysis task. the first step consists of further pre-training the model on financial documents, as this strategy has already be proven to be effective @xcite for domain adaptation. this step aims at helping the model to understand financial terminologies better than the base model. authors used a subset of reuters' trc24 dataset foot_1 , a collection of news articles published by reuters that was filtered with keywords related to finance, to fine-tune the model. in the second step, the model is prepared for the sentiment analysis task by adding a dense layer to the last hidden state of the classification token cls of the encoder-based architecture, a recommended practice for classification with bert. this task is finetuned using the financial phrasebank (fpb) @xcite , a financial sentiment analysis dataset. finbert presents remarkable results on financial sentiment analysis, outperforming the state-of-theart. nevertheless, the model is strongly limited to sentiment analysis and underperforms greatly on other tasks. ## methodology in this section, we describe the methods designed to conduct the experiments. first, we list the foundation models that are used as a part of this project. then, two new dataset collections are introduced, one with data based on documents and the second with instructions. we also give details of designing a data augmentation strategy for the instructions as well as the description of the training process carried out to fine-tune the foundation models. ## conclusion this project has covered a wide range of aspects of financial llms. through a series of experiments, using pythia-1.4b and opt-1.3b as base models, we studied the adaptation of relatively small llms for finance. the experiments we conducted first show that llms adapted to the financial domain through further pre-training followed by instruction fine-tuning perform better than some of the best current generalist llms (such as gpt-4) on financial tasks. second, it validates our training strategy since our llms obtain higher or similar scores than other financial llms that were trained with much more parameters and larger datasets. lowering the requirements to fine-tune llms for this specific industry can be key for the future of several companies, since it can enable smaller organizations to host their own llms or, at least, to make them more accessible. furthermore, it is worth mentioning that the models used for this project as well as most datasets, except for the inhouse subset (only 9.7% of the documents dataset), are open and publicly available. in addition to the findings related to domain adaptation of llms for financial tasks and the models presented, a strategy for the generation of samples for the instructions dataset is introduced. moreover, the two datasets used for the project are described with enough details to be reproduced by other researchers. finally, the paper also presented a comprehensive study that delves into the state-of-the-art and the evolution of approaches for financial sentiment analysis, ranging from traditional dictionary-based methods to the more recent advancements in llms. despite the fact that the results showed great performance of the small-sized models, in further research these fine-tuning strategies could be applied to larger models and study their impact on different scales and domains. an interesting option to study in the future are low-rank adapters or lora @xcite , a method that reduces the number of trainable parameters by freezing the foundation model weights and injecting trainable rank decomposition matrices into each layer of the llm.