B iz B ench: A Quantitative Reasoning Benchmark for Business and Finance (2024)

Abstract: Answering questions within business and finance requires reasoning, precision, and a wide-breadth of technical knowledge. Together, these requirements make this domain difficult for large language models (LLMs). We introduce BizBench, a benchmark for evaluating models’ ability to reason about realistic financial problems. BizBench comprises eight quantitative reasoning tasks, focusing on question-answering (QA) over financial data via program synthesis. We include three financially-themed code-generation tasks from newly collected and augmented QA data. Additionally, we isolate the reasoning capabilities required for financial QA: reading comprehension of financial text and tables for extracting intermediate values, and understanding financial concepts and formulas needed to calculate complex solutions. Collectively, these tasks evaluate a model’s financial background knowledge, ability to parse financial documents, and capacity to solve problems with code. We conduct an in-depth evaluation of open-source and commercial LLMs, comparing and contrasting the behavior of code-focused and language-focused models. We demonstrate that the current bottleneck in performance is due to LLMs’ limited business and financial understanding, highlighting the value of a challenging benchmark for quantitative reasoning within this domain.

## introduction large language models (llms) show strong performance on question-answering (qa) and code generation tasks @xcite . nonetheless, it remains particularly difficult for models to reason about quantities and numbers @xcite . this poses issues for using llms for real-world problems in business † equal contribution. ## reasoning in business and finance business and finance professionals are frequently required to reason about quantities. they often search for specific quantities within large sets of distractors (e.g., searching for the revenue of a particular division for a specific quarter, and doing so from a report that contains a comprehensive detailing of financial metrics across many divisions). they also rely on financial domain knowledge across banking, accounting, and finance to manipulate these quantities in useful ways. for example, computing an ebitda margin requires first identifying a series of relevant quantities (net income, interest expense, taxes, depreciation, amortization, and revenue) and then applying a specific formula to these quantities (taking the sum of the first five quantities and computing their percentage of revenue). since large sums of money are often at stake, financial professionals need to communicate transparent rationales for their decisions. researchers have developed tasks to measure some aspects of quantitative financial reasoning capabilities of ai systems. finqa consists of 8,281 questions written by financial professionals, paired with context and tables from earnings reports of the largest companies listed on american stock exchanges @xcite . answering finqa questions often requires multiple reasoning steps, including extracting quantities, performing mathematical operations with them, and reasoning about time periods such as fiscal quarters and years. additionally, these questions test the financial knowledge of currency and scale (e.g. percentage, millions, billions). similar to finqa, tat-qa consists of questions, answers, and relevant context of text and tables @xcite . the 16,552 questions in tat-qa cover a range of skills including span-extraction, comparison, and arithmetic. program synthesis improves transparency of model outputs, allowing for auditing of reasoning steps -which in turn increases trust and usability. since large sums of money are often at stake, financial professionals need to communicate transparent rationales for their decisions. generating programs allows qa models to avoid arithmetic calculations, which is challenging for smaller models. ## task details the main focus of bizbench is evaluating a model's financial understanding through program synthesis. we additionally provide tasks in two domains we view to be the building blocks in this regard: quantity extraction and financial domain knowledge. in order to answer a complex program synthesis question, the model must be able to understand and extract relevant values within a question or context. it must also have some knowledge of the necessary formulas and other information required to then compute the answer. additional skills and features of each task are shown in ## few-shot experiments we benchmark state-of-the-art open-source and proprietary models on the bizbench dataset using the eleutherai lm evaluation harness @xcite , establishing how performance ranges across model sizes and pretraining strategies. for large models, we provide zero-and few-shot results. we evaluate falcon @xcite , mpt (mosaicml, 2023), starcoder @xcite , llama-2 and codellama @xcite , mistral/mixtral @xcite , and openai's gpt(s) @xcite . we were unable to benchmark pre-existing financial llms due to three factors: the models we hoped to benchmark were either proprietary (e.g., bloomberggpt @xcite ), designed for languages other than english (e.g. bbt-fin-t5 @xcite and xuanyuan 2.0 @xcite ) or did not include code in their financial tuning and thus led to very poor results (e.g. finma @xcite , fingpt @xcite , and fin-llama (william todt, 2023)). bizbench requires models to be adept in both finance and coding. we discuss this limitation further in section 7. model size and alignment impact performance: generally, larger models perform better, and models using rlhf and instruction tuning, such as gpt variants, perform best of all. there are notable outliers: mistral 7b outperforms larger models-mpt, falcon, and llama 2-across a range of settings. ## supervised finetuning we finetune llama-2-7b in a number of settings to evaluate how the availability of both in-domain and multi-task training data impacts performance. specifically, we investigate how different amounts of training data examples impact performance across codefinqa, codetat-qa, and sec-num. to test this, we downsample the training data from 100 to 5,000 samples. figure 6 shows the relationship between accuracy and training data size for finetuned llama-2-7b models across these sizes compared to the few-shot results of the pretrained models. next, we evaluate llama-2-7b with multi-task training across these same datasets, codefinqa, codetat-qa, and sec-num. analysis. llama-2-7b requires fewer than 500 samples to outperform the llama-2-13b on all three datasets. to outperform llama-2-70b, llama- we study how knowledge is transferred across these three tasks. ## related work bizbench focuses on code generation and financial information extraction, complementing other financial nlp benchmark datasets which tend to focus on non-quantitative tasks @xcite , or quantitative trading tasks @xcite . financial nlp. non-quantitative tasks like sentiment detection @xcite , named entity recognition @xcite , classification @xcite , question answering @xcite , and boundary detection tasks @xcite are included in the financial benchmarks flare, introduced within pixiu @xcite , and flue @xcite . @xcite additionally include entity extraction, event extraction, and natural language to sql task for chinese. recently, stock movement prediction @xcite and pair trading @xcite were introduced as prediction tasks. bizbench complements these benchmarks by specifically focusing on quantitative reasoning via program synthesis, providing the first tasks of this kind within financial nlp. code generation. the apps and humaneval benchmarks test a model's ability to write code from arbitrary natural language specifications @xcite . @xcite introduce mbpp, a collection of simple programming questions, as well as a python version of the mathqa dataset introduced in @xcite . we take inspiration from this approach to augmenting existing data when creating codetat-qa and codefinqa. like in our work, @xcite constructs a dataset of code solutions to data science problems which involve using libraries like numpy and pandas. our coding tasks also involve reading comprehension, quantity manipulation, and financial knowledge. quantity extraction. previous work explores extracting numbers, metric scales and semantic descriptions in financial @xcite and scientific documents @xcite . many financial qa datasets require precise quantity extraction such as hybridqa @xcite , tat-qa @xcite , multi-hiertt @xcite , finqa @xcite , and convfinqa @xcite . our bizbench benchmark includes three datasets dedicated to testing numerical extraction capabilities, with our novel sec-num dataset encompassing values, scales, currency, dates, times, and periods. ## conclusion numerical reasoning is critically important for business and finance, as well as other domains, because small errors can incur large costs. to address this, we introduce bizbench, a benchmark for business and finance that measures models' abilities across three categories of relevant tasks: domain knowledge, quantity extraction, and program synthesis. the benchmark includes eight tasks, five of which provide novel data or meaningful extensions of existing datasets. we focus on program synthesis, as success on these challenging tasks requires strong capabilities in both domain knowledge and quantity extraction. additionally, we evaluate many stateof-the-art models, illustrating that models need improvement to meet the demands required for reasoning in real-world, high-stakes domains.