Are C hat GPT and GPT -4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks (2023)

Abstract: The most recent large language models (LLMs) such as ChatGPT and GPT-4 have shown exceptional capabilities of generalist models, achieving state-of-the-art performance on a wide range of NLP tasks with little or no adaptation. How effective are such models in the finance domain? Understanding this basic question would have a significant impact on many downstream financial analytical tasks. In this paper, we conduct empirical studies and provide experimental evidences of their performance on a wide variety of financial text analytical problems, using eight benchmark datasets from five categories of tasks. We report both the strengths and limitations of the current models by comparing them to the state-of-the-art fine-tuned approaches and the recently released domain-specific pretrained models. We hope our study can help to understand the capability of the existing models in the financial domain and facilitate further improvements.

## introduction the advancement of llms is bringing profound impacts on the financial industry. through training with reinforcement learning from human feedback (rlhf) @xcite and masked language model objectives, the most recent models such as chatgpt 1 and gpt-4 2 have demonstrated exceptional capabilities in a wide range of natural language processing (nlp) tasks @xcite @xcite . these llms are trained on datasets that encompass a broad range of genres and topics. while their performance in generic nlp tasks is impressive, their applicability and effectiveness in specific domains like finance yet need a better understanding and can influence a wide range of applications. in general, in the financial domain, llms is playing an increasingly crucial role in tasks such as investment sentiment analysis, financial named entity recognition, and question-answering systems for assisting financial analysts. in this paper, we perform an empirical study and provide experimental evidence for the effectiveness of the most recent llms on a variety of financial text analytical problems, involving eight benchmark datasets from five typical tasks. these datasets are from a range of financial topics and sub-domains such as stock market analysis, financial news, and investment strategies. we report both the strengths and limitations of chatgpt and gpt-4 by comparing them with the state-of-the-art domain-specific fine-tuned models in finance, e.g., finbert @xcite and finqanet @xcite , as well as the recently pretrained model such as bloomberggpt @xcite . our main contributions are summarized as follows: ## related works chatgpt and related models. chatgpt, gpt-3.5 (text-davinci-003), and gpt-4 are generically trained llms and have shown impressive performance on a wide range of tasks. recent studies have shown that they outperform fine-tuned models on some tasks. but, they still fail in some other cases. @xcite evaluated chatgpt on multitasking, multilingual and multimodal tasks, highlighting addressing the failures to improve the overall performance. ## experiment setup tasks and datasets. our research utilizes a wide range of financial nlp tasks and challenges @xcite @xcite , enabling us to establish a testbed with different types of nlp problems ranging from basic sentiment analysis and text classification to information extraction and question answering (see the span of the tasks enables us to make observations along modeling complexity and different levels of financial knowledge required to perform the tasks. regarding the modeling complexity of tasks, sentiment analysis and text classification are often regarded to be more straightforward, compared to information extraction (ie) tasks such as named entity recognition (ner) and relation extraction (re). the latter often requires more understanding of syntax and semantics in the input contexts as well as the interactions of labels in the output space as the structured prediction problems. compared to sentiment analysis and text classification, question answering (qa) is often thought of as being harder as it often requires a model to understand the embedded internal logic and numerical operation/reasoning. regarding financial knowledge, the existing classification and sentiment analysis datasets are sourced from daily news and social media. on the other hand, ie and qa data are often from professional documents like financial filings and reports, which usually require more domain knowledge to comprehend. ## discussions comparison over llms. we are able to benchmark the performance of chatgpt and gpt-4 with four other llms on five tasks with eight datasets. chatgpt and gpt-4 significantly outperforms others in almost all datasets except the ner task. it is interesting to observe that both models perform better on financial nlp tasks than bloomberggpt, which was specifically trained on financial corpora. this might be due to the larger model size of the two models. finally, gpt-4 constantly shows 10+% boost over chatgpt in straightforward tasks such as headlines and fiqa sa. for challenging tasks like re and qa, gpt-4 can introduce 20-100% performance growth. this indicates that gpt-4 could be the first choice for financial nlp tasks before a more powerful llm emerges. prompt engineering strategies. we adopted two commonly used prompting strategies: few-shot and chain-of-thoughts. we constantly observe 1% to 4% performance boost on chatgpt and gpt-4 from few-shot over zero-shot learning across various datasets. chain-of-thoughts prompting is very effective in our test and demonstrates 20-30% accuracy improvement over zero-shot and few-shot as well. according our findings, we argue that these two strategies should always be considered first when applying llms to financial nlp tasks. llms vs. fine-tuning. one attractive benefit of using llms in business domains is that they can be applied to a broad range of nlp tasks without conducting much overhead work. it is more economical compared to fine-tuning separate models for every task. whereas, our experiments show finetuned models still demonstrate strong performance in most of the tasks except the qa task. notably, for tasks like ner and re, llms are less effective than fine-tuned models. in the qa tasks, llms illustrated the advantage over fine-tuned model. but the reasoning complexity of the tested qa tasks is still deemed as basic in financial analysis. although chatgpt and gpt-4 have proven to be able to perform multi-step reasoning, including numerical reasoning, to some extent, simple mistakes have still been made. ## conclusion this study is among the first to explore the most recent advancement of generically trained llms, including chatgpt and gpt-4, on a wide range of financial text analytics tasks. these models have been shown to outperform models fine-tuned with domain-specific data on some tasks, but still fall short on others, particularly when deeper semantics and structural analysis are needed. while we provide comprehensive studies on eight datasets from five categories of tasks, we view our effort as an initial study, and further investigation of llms on financial applications is highly desirable, including the design of more tasks to gain further insights on the limitations of existing models, the integration of llms in the loop of human decision making, and the robustness of the models in high-stakes financial tasks. authors are solely responsible for the contents of the paper and the opinions expressed in this publication do not reflect those of the funding agencies.