D avid vs. Goliath: Cost-Efficient Financial QA via Cascaded Multi-Agent Reasoning (2025)

Abstract: Large language models (LLMs) have demonstrated remarkable reasoning capabilities, including in financial question answering (FQA). However, the performance in FQA remains limited, particularly in questions that require deep financial knowledge and complex numerical reasoning. While supervised fine-tuning and closed-source LLMs have shown promise, they are often constrained by high costs or computational inefficiency. In this paper, we propose a low-cost yet effective framework, named FinMAN (Financial multi-agent framework), that enables small LLMs (e.g., 8B) to perform complex reasoning tasks without relying on expensive models or task-specific fine-tuning. FinMAN improves formula selection, extraction, and calculation to help small-scale models solve FQA tasks more accurately, with a lightweight verification mechanism to correct common errors. Experimental results show that FinMAN outperforms the best open-source model on BizBench by 10.46% and achieves competitive performance to GPT-3.5 using significantly fewer parameters. Our code and data are publicly available at https://github.com/coenliu/MultiAgentFin.

## introduction recent advances in llms @xcite @xcite @xcite have led to remarkable progress across a wide range of natural language processing (nlp) tasks @xcite @xcite @xcite . however, their performance remains limited when tackling domain-specific and knowledge-intensive problems such as financial question answering (fqa) @xcite . in real-world fqa scenarios, high performance is typically achieved with large, closed-source models (e.g., gpt-3.5 * corresponding author the formula for interest for rate: @xmath0: what will the worth of the investment be after 2 years for a shareholder? @xmath1. @xmath2: what is the total proportion of cost of revenue as a percentage of revenue in 2017 and 2018? context: "… revenue": {"2019": 43, "2018": 50, "2017": 55} … case1 case3 case2 figure 1 : illustration of error cases generated by smaller llms (e.g., llama2) and correct answers by finman. @xcite ), which are expensive and inaccessible to many users @xcite . in contrast, smaller, open-source models (e.g., llama (meta, 2024)) are more affordable but underperform, particularly when dealing with long and complex financial documents. in this work, we aim to bridge this gap by developing an effective and affordable llm-based method tailored for real-world fqa applications. small-scale llms typically face three challenges in fqa. first, they often struggle to provide accurate and reliable financial knowledge. for instance, in case 1 of fig. 1 , a small llm (e.g., llama2) applied an incorrect formula to calculate annual income, demonstrating insufficient domain knowledge for answering professional financial questions. second, they have difficulty in extracting key financial values from lengthy financial contexts. as shown in case 2, they fail to extract the correct value for statement of operations from the given long financial document. third, they underperform in multi-step reasoning and calculations, frequently making errors in intermediate steps or the final computation (case 3). given that fqa involves complex, multi-step solutions, any type of these errors can lead to incorrect answers. to address these challenges, a common approach is to apply supervised fine-tuning (sft) @xcite on small llms for fqa. however, finetuning requires annotated data and substantial computational resources, making it costly and less accessible. in response to specific challenges, some studies have explored alternative strategies, such as incorporating external financial knowledge via retrieval-augmented generation (rag) @xcite or enhancing numerical reasoning through external computational tools @xcite . while these approaches show promise, they often depend heavily on external retrieval systems or tools and fall short of offering a unified solution to the broader challenges of fqa. additionally, some efforts have been made to improve small llms' reasoning ability by knowledge distillation from large models like gpt-3.5 and deepseek @xcite . however, distilling the intricate reasoning capabilities required for complex fqa tasks remains a significant hurdle, limiting the effectiveness of such methods in guiding meaningful improvements @xcite . in this work, we explore how smaller llms can achieve strong performance in fqa without relying on closed-source llms. to this end, we propose a cost-efficient method using cascaded multiagent reasoning, named finman. specifically, we decompose fqa into modular subtasks, each assigned to a distinct agent specialized in a particular role. we design four agents: a formulator (financial expert) for reasoning and formula planning, a resolver (data analyst) for report comprehension and quantity extraction, an executor that generates and runs code to compute the answer, and a cross-cutting evaluator that verifies and corrects outputs at each stage. these subtasks become more facilitated once decomposed and clearly defined for each agent, particularly for the data analyst and executor, whose responsibilities are straightforward. moreover, the financial expert, based on small llms, is required to master deep, domainspecific knowledge. to support this agent, we employ monte carlo tree search (mcts) @xcite to simulate and explore reasoning paths, and we integrate an evaluator agent that leverages external financial knowledge to score and verify each intermediate step. consequently, finman gains an accuracy of 25.5% in the out-of-domain financemath dataset (zhao et al., 2024a), surpassing gpt-3.5. additionally, to ensure reliability and transparency, we incorporate a stepwise verification mechanism, making each output traceable and verifiable. our method draws inspiration from the human strategy of breaking down complex tasks into manageable subtasks, allowing smaller llms to become proficient by addressing these subtasks foot_0 . to validate its effectiveness, we evaluate finman on widely used fqa tasks, covering both quantity extraction and financial mathematical reasoning. by enabling cooperation among multiple smaller agents (e.g., llama3-8b), our approach improves mathematical reasoning accuracy from 32.98% to 46.63%, reaching or even surpassing the performance of gpt-3.5 (i.e., 36.1%). our contributions are summarized as follows: llama3-8b output: solution: the bond's price can be calculated using the formula: bond @xmath3.06)^3 bond @xmath4.1618 bond @xmath5.93 question: the following information pertains to a bond, what will be the bond's price? table: coupon rate 3% -- ## methodology the overall architecture of our finman framework is shown in figure 3 . the basic idea is a cascaded pipeline of four agents. the formulator agent uses mcts to explore reasoning paths and generate financial formulas and variables. next, the resolver agent retrieves relevant text and table segments to extract variable values. then, the executor agent converts these formulas and values into python code and executes it to obtain the answer. finally, the evaluator agent performs error detection to verify and correct each intermediate output. formulator agent with mcts. in fqa, selecting the appropriate formula and its variables is crucial for obtaining the correct result. given a question that includes context e, tables t , and query q, the formulator agent m f ormulator integrates mcts to explore candidate reasoning trajectories composed of intermediate steps. we define an action set @xmath6, c, d, i, a} that represents thinking, clarify, decompose, identify, and abstract, where each child node represents an intermediate question: what is the net change in cash during 2016? instruction: please choose one of the action to solve the @xmath7,<t: thinking>, <c: clarify>,<d: @xmath8: identify>, <a: @xmath9. step s i produced by m f ormulator . a candidate trajectory is designed by concatenating the question q and a sequence of