Is C hat GPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing (2023)

Abstract: The emergence of Large Language Models (LLMs), such as ChatGPT, has revolutionized general natural language preprocessing (NLP) tasks. However, their expertise in the financial domain lacks a comprehensive evaluation. To assess the ability of LLMs to solve financial NLP tasks, we present FinLMEval, a framework for Financial Language Model Evaluation, comprising nine datasets designed to evaluate the performance of language models. This study compares the performance of fine-tuned auto-encoding language models (BERT, RoBERTa, FinBERT) and the LLM ChatGPT. Our findings reveal that while ChatGPT demonstrates notable performance across most financial tasks, it generally lags behind the fine-tuned expert models, especially when dealing with proprietary datasets. We hope this study builds foundation evaluation benchmarks for continuing efforts to build more advanced LLMs in the financial domain.

## introduction recent progress in natural language processing (nlp) demonstrates that large language models (llms), like chatgpt, achieve impressive results on various general domain nlp tasks. those llms are generally trained by first conducting self-supervised training on the unlabeled text @xcite @xcite and then conducting instruction tuning @xcite or reinforcement learning from human feedback (rlhf) @xcite to let them perform tasks following human instructions. financial nlp, in contrast, demands specialized knowledge and specific reasoning skills to tackle tasks within the financial domain. however, for general language models like chatgpt, their selfsupervised training is performed on the text from various domains, and the reinforcement learning feedback they receive is generated by non-expert workers. therefore, how much essential knowledge and skills are acquired during the learning process remains uncertain. as a result, a comprehensive investigation is necessary to assess its performance on financial nlp tasks. to fill this research gap, we are motivated to evaluate language models on financial tasks comprehensively. for doing so, we propose a framework for financial language model evaluation @xcite . we collected nine datasets on financial tasks, five from public datasets evaluated before. however, for those public datasets, it is possible that their test sets are leaked during the training process or provided by the model users as online feedback. to eliminate this issue, we used four proprietary datasets on different financial tasks: financial sentiment classification (finsent), environmental, social, and corporate governance classification (esg), forward-looking statements classification (fls), and question-answering classification (qa) for evaluation. in the evaluation benchmark, we evaluate the encoder-only language models with supervised fine-tuning, with representatives of bert @xcite , roberta @xcite , finbert @xcite and flang @xcite . we then compare the encoder-only models with the decoder-only models, with representatives of chat-gpt @xcite , gpt-4 (openai, 2023), pixiu @xcite , llama2-7b @xcite and bloomberg-gpt @xcite by zero-shot prompting. besides, we evaluate the efficacy of in-context learning of chatgpt with different in-context sample selection strategies. experiment results show that (1) the fine-tuned task-specific encoder-only model generally performs better than decoder-only models on the financial tasks, even if decoder-only models have much larger model size and have gone through more pre-training and instruction tuning or rlhf; get response with prompt: perform financial sentiment classification. classify the following sentence into 'neutral',' positive', or 'negative' class. only provide the label in the output. {placeholder} the sentence: energy-related investments reported a 277% increase in quarterly earnings. the label: here are some examples: the sentence: {example 1} the label: {label 1} … the sentence: {example n} the label: {label n} the sentence: energy-related investments reported a 277% increase in quarterly earnings. the label: evaluating encoder-only models expected output: positive energy-related investments reported a 277% increase in quarterly earnings. zero-shot decoder-only models have more advantages than fine-tuned encoder-only models; (3) the performance gap between fine-tuned encoder-only models and zero-shot decoder-only models is more significant on private datasets than the publicly available datasets; (4) in-context learning is only effective under certain circumstances. to summarize, we propose an evaluation framework for financial language models. compared to previous benchmarks in the financial domain like flue @xcite , our evaluation includes four new datasets and involves more advanced llms like chatgpt. we show that even the most advanced llms still fall behind the fine-tuned expert models. we hope this study contributes to the continuing efforts to build more advanced llms in the financial domain. ## related works the utilization of language models in financial nlp is a thriving research area. while some general domain language models, like bert @xcite , roberta @xcite , @xcite and llama @xcite have been applied to financial nlp tasks, financial domain models like fin-bert @xcite @xcite , flang @xcite , pixiu @xcite , investlm @xcite and bloomberggpt @xcite are specifically designed to contain domain expertise and generally perform better in financial tasks. recent work such as flue @xcite has been introduced to benchmark those language models in the finance domain. however, the capability of more advanced llms, like chatgpt and gpt-4, has not been benchmarked, especially on proprietary datasets. in this work, in addition to the public tasks used in flue, we newly include four proprietary tasks in finlmeval and conduct comprehensive evaluations for those financial language models. ## methods we compare two types of models in finlmeval: the transformers encoder-only models that require fine-tuning on the labeled dataset, and decoder-only models that are prompted with zero-shot or fewshot in-context instructions. figure 1 provides an outline of evaluation methods of finlmeval. ## datasets our evaluation relies on nine datasets designed to evaluate the financial expertise of the models from diverse perspectives. finsent is a newly collected sentiment classification dataset containing 10,000 manually annotated sentences from analyst reports of s&p 500 firms. fpb sentiment classification @xcite is a classic sentiment dataset of sentences from financial news. the dataset consists of 4840 sentences divided by the agreement rate of 5-8 annotators. we use the subset of 75% agreement. fiqa sa (fiqa) is a aspect-based financial sentiment analysis dataset. following the "sentences for qa-m" method in @xcite , for each (sentence, target, aspect) pair, we transform the sentence into the form "what do you think of the {aspect} of {target}? {sentence}" for classification. esg evaluates an organization's considerations on environmental, social, and corporate governance. we collected 2,000 manually annotated sentences from firms' esg reports and annual reports. fls, the forward-looking statements, are beliefs and opinions about a firm's future events or results. fls dataset, aiming to classify whether a sentence contains forward-looking statements, contains 3,500 manually annotated sentences from the management discussion and analysis section of annual reports of russell 3000 firms. qa contains question-answering pairs extracted from earnings conference call transcripts. the goal of the dataset is to identify whether the answer is valid to the question. headlines (sinha and khandait, 2020) is a dataset for the commodity market that analyzes fomc @xcite aims to classify the stance for the fomc documents into the tightening or the easing of the monetary policy. among the datasets, finsent, esg, fls, and qa are newly collected proprietary datasets. ## experiments this section introduces the experiment setups and reports the evaluation results. ## conclusions we present finlmeval, an evaluation framework for financial language models. finlmeval comprises nine datasets from the financial domain, and we conduct the evaluations on various popular language models. our results show that fine-tuning expert encoder-only models generally perform better than the decoder-only llms on the financial nlp tasks, and adding in-context demonstrations barely improves the results. our findings suggest that there remains room for enhancement for more advanced llms in the financial nlp field. our study provides foundation evaluations for continued progress in developing more sophisticated llms within the financial sector.