E con NLI : Evaluating Large Language Models on Economics Reasoning (2024)

Abstract: Large Language Models (LLMs) are widely used for writing economic analysis reports or providing financial advice, but their ability to understand economic knowledge and reason about potential results of specific economic events lacks systematic evaluation. To address this gap, we propose a new dataset, natural language inference on economic events (EconNLI), to evaluate LLMs’ knowledge and reasoning abilities in the economic domain. We evaluate LLMs on (1) their ability to correctly classify whether a premise event will cause a hypothesis event and (2) their ability to generate reasonable events resulting from a given premise. Our experiments reveal that LLMs are not sophisticated in economic reasoning and may generate wrong or hallucinated answers. Our study raises awareness of the limitations of using LLMs for critical decision-making involving economic reasoning and analysis. The dataset and codes are available at https://github.com/Irenehere/EconNLI .

## introduction economics is a social science that studies the behavior and interactions of economic agents such as individuals, firms, and nations @xcite . it has significant social impact as it covers nearly all vital aspects of human life, including social production, distribution, and consumption @xcite @xcite . studying economics enables people to understand human society better and forecast future trends. recently, with the development of large language models (llms), many financial institutions and investment firms are using these models to assist in writing economic analysis reports or providing financial advice @xcite @xcite . for example, an llm may be used to analyze the impact of changes in interest rates on the stock market or to predict the 𝑃: the available amount of hard currency per person falls 𝐻: the purchasing power of each unit of currency increases will 𝑃 cause 𝐻? no reference: yes if 𝑃 happens, what event will be caused? the value of the currency will decrease reference: the purchasing power of each unit of currency increases potential effects of a new government policy on a particular industry. these tasks require llms to understand economic knowledge and possess the ability to reason and infer the potential results that may arise from certain economic events or actions. however, there lacks a systematic evaluation of such tasks, and how well the llms can understand the economic knowledge and reason for the economic events is unknown. to fill this research gap, we propose a new task, natural language inference on economic events (econnli), to evaluate the llms' knowledge and reasoning ability on economic events. to facilitate this task, we construct an econnli dataset that contains sentence pairs of premise and hypothesis, both of which are economic events. figure 1 provides an example chatgpt cannot correctly answer. in this example, the premise is the event that the available amount of hard currency per person falls, and the hypothesis is the purchasing power of each unit of currency increases. different from the traditional nli datasets @xcite @xcite , the premise does not entail the hypothesis in semantics; instead, it requires understanding the economic theory to conduct the inference. inference on the example in figure 1 is based on the quantity theory of money. in the case of a decrease in the available amount of hard currency per person, the money supply in the economy will decrease, leading to a decrease in the price level. as a result, the purchasing power of each unit of currency will increase. we assess the performance of llms on econnli from two aspects: classification and generation. as shown in figure 1 , in terms of classification, we provide the premise and the hypothesis to the llms and evaluate whether the llms can correctly classify the causal relationship between the premise and the hypothesis. for generation evaluation, we ask the llms to generate the possible resulting events of the given premise and compare the generated events with the hypothesis of the positive example, which serves as a reference. we conduct extensive experiments on various language models, from open-source to commercial llms, and from the general to the finance-specific domain models. for the classification task, we find that without supervised fine-tuning on the training set, the open-source llms perform closely to the random guess, and the advanced commercial llms such as chatgpt and gpt-4 also have unsatisfactory performance. as for the generation task, we find that the llms can generate the wrong or hallucinated answers. our findings suggest that llms are not sophisticated in economic reasoning, and there remains room for improving the reliability of llms in financial and economic analysis. in conclusion, this paper presents a novel dataset designed to evaluate the reasoning ability of llms in the economic domain. the results of our experiments suggest that these models are far from perfect in their ability to economic reasoning. given that llms are now widely applied in writing financial reports or providing financial advice without carefully evaluating their correctness, our study shows llms can sometimes lead to errors or unreliable results in economic analysis. therefore, it is essential to be aware of the limitations of these models when using them for decision-making. meanwhile, some works evaluate these llms' capability on the financial tasks. @xcite find the chatgpt and gpt-4 struggle in answering the mock exam questions of the chartered financial analyst (cfa) @xcite show gpt-4 performs comparably to human experts as a data analyst. @xcite shows while some llms demonstrate notable performance on some financial nlp tasks, they generally lag behind the fine-tuned expert models. besides, several datasets are developed to evaluate llms in various financial nlp tasks, such as sentiment analysis @xcite , text classification @xcite or question answering @xcite . however, none of them involve assessing the reasoning ability of llms in the financial and economic domain. to our knowledge, we are the first to assess llms from the aspect of reasoning on the relations between the economic events. natural language inference. natural language inference (nli), which aims to reason the entailment relationship between text fragments, has long been a key task in natural language processing @xcite . various natural language inference tasks are proposed to address the inference ability from different aspects, such as rte @xcite , snli @xcite , multinli @xcite , scitail @xcite , abductive nli @xcite , defeasible nli @xcite , scinli @xcite . however, none of them address the inference in economics and finance. unlike traditional nli tasks that rely on linguistic or common sense knowledge for reasoning, econnli demands an understanding of economic theories to perform event reasoning. ## natural language inference on economic events we study the problem of causal relation inference on the economic events in the natural language form. in our study, we define an event as somebody or something having some actions or states. economic events are the events that are related to the economic subject or phenomenon, for example, events describing production, distribution, and consumption @xcite . we construct a dataset, econnli, to evaluate llms' causal relation inference ability on economic events. it consists of sentence pairs of (premise, hypothesis), both of which represent economic events. in positive pairs, the occurrence of the premise event leads to the occurrence of the hypothesis event based on economic theories. for negative pairs, no such causal relationship is supported by any theory. note that in the real world, multiple premises may cause the hypothesis. for example, premise 1 and premise 2 may be the joint causes of the hypothesis. however, in our benchmarking, we only consider whether premise 1 causes the hypothesis and premise 2 causes the hypothesis separately. if premise 1 and premise 2 are joint causes of the hypothesis, we exclude this example in our dataset. ## econnli dataset construction in this section, we describe the construction of econnli. the outline of the construction framework is shown in figure 2 . we first prepare a corpus about the economics from wikipedia, and extract the events from the corpus using a supervised finetuned llama2 model. next, we generate candidates for premises and hypotheses based on a given reference sentence and assign a label. to ensure the dataset's scale and quality, we use chatgpt and gpt-4 to assign labels to the training set and annotate the testing set with manual inspection to align with human domain knowledge. ## econnli evaluation in this section, we use econnli to benchmark the llms' performance on economic event reasoning. we examine the language models' capabilities in inferring the correct relations between the event pairs and the correctness in generating the possible resulting events of a given premise. ## conclusions in conclusion, we propose a new task to evaluate (financial) llms' ability to reason about economics. to do so, we construct a new dataset, econnli, and apply it to benchmark llm's ability to perform classification and generation tasks. through experiments, we find that llms have limitations in understanding economic knowledge and reasoning about the potential results of specific economic events. this work provides a new benchmark and insights for constructing financial llms. by emphasizing the need for continued research in this area, we hope to inspire future studies that will further enhance the reasoning capabilities of llms in the financial domain.