Synthetic Data Generation Using Large Language Models for Financial Question Answering (2025)

Abstract: Recent research has shown excellent performance of large language models (LLMs) for answering questions requiring multi-step financial reasoning. While the larger models have been used with zero-shot or few-shot prompting, the smaller variants need fine-tuning on training data containing questions and the corresponding answers that includes detailed reasoning demonstrations. To alleviate the significant cost of creating a data set with complex questions and corresponding answers, we explore the use of synthetic data for financial question answering using a multi-step LLM based approach to generate question as well as the answers with reasoning steps. We consider standard as well as conversational financial question answering scenarios. We experiment with synthetic data generation for three different real financial reasoning problems that already have manually collected data sets created with the help of financial experts. Using the same document sources, we use the proposed LLM based approach to generate synthetic questions and answers. To measure the effectiveness, we train multiple small language models (SLMs) on these synthetic data and compare the performance with that of the same SLMs trained on the real data. We further perform extensive experimental analysis generating important evidence on the potential of using synthetic data in financial reasoning tasks.

## introduction developing machine learning systems for answering questions in the financial domain is a challenging problem. these systems must be capable of complex multi-step reasoning using real-world financial data. in recent years, the creation of largescale financial question-answering datasets has led to significant improvements in this specialized domain @xcite . nonetheless, assembling these datasets is a complicated, labor-intensive, and costly process, requiring the expertise of skilled annotators @xcite . as llms continue to advance, researchers have explored their potential to address these financial reasoning problems. using methods that rely on an llm to encode the reasoning steps into python programs which are then executed by external python interpreters, state-of-the-art results have been obtained @xcite . while these extremely large models offer the benefit of easy use through prompting and eliminate the need for large-scale manual data set curation, their deployment at scale is expensive due to significant computational and inference costs. to alleviate the reliance on extremely large models, recent research has focused on fine-tuning slms using data containing reasoning demonstrations that are generated using a large model @xcite . promising results on various tasks including arithmetic, symbolic, common-sense reasoning @xcite and financial reasoning @xcite have been achieved. however, for tasks without any existing data, these methods still rely on the time-consuming and expensive manual collection of data. synthetic data generation via zero-shot or fewshot llm prompting provides an appealing alternative to manual data creation, as demonstrated in recent studies @xcite @xcite @xcite . while conceptually simple, achieving both high correctness and diversity in synthetic data sets is challenging @xcite , with current methods showing variable success @xcite . in the realm of question answering (qa), the generation of synthetic qa data from text has been previously investigated @xcite @xcite with promising results. these studies have focused on question generation requiring deep semantic comprehension, as opposed to questions that demand numerical analysis. currently, there is a scarcity of studies examining the use of llms to create high-quality datasets specifically tailored for financial reasoning tasks. in this work, we undertake a detailed and methodical inquiry into the effectiveness of llms driven synthetic financial reasoning data generation from financial documents. we focus on studying zero-shot prompting both with and without example questions. in addition to a standard scenario that requires the creation of a single question from a provided financial text passage, we consider the creation of a set of questions representative of a conversation over a financial document. the conversational scenario challenges the llm to create a series of inter-related questions that are coherent, require context tracking and reference resolution across the questions. we conduct thorough experiments in both scenarios to evaluate the ability of llm-based techniques for creating questions demanding multi-hop numerical reasoning and their detailed answers with reasoning steps. in the standard scenario, we design a zero-shot prompt with constraints to direct the type of question generation, sometimes adding actual examples. answers, formatted as python code encoding the required calculations, are produced separately and then screened to remove any incorrect pairs. the filtering process excludes pairs with codes that are non-executable or yield outputs in incorrect formats, without evaluating the data's domain-specific correctness. for conversational data synthesis, we include an additional instruction that directs the llm to formulate a sequence of questions conversationally. our key contributions are outlined below: our results indicate that models trained on synthetic data nearly match the performance of those trained on real data for standard financial qa. synthetic data sets yield acceptable results for conversational financial qa, though they fall short of real data's effectiveness. additionally, two key results hint at better generalization of models fine-tuned with synthetic data (1) slm fine-tuned on synthetic data outperformed the same model trained on real data when evaluated on a similar but independent test data set (2) slm trained on a dataset deliberately crafted to have low similarity to the real data performed on par with the same model trained on data with higher similarity. these findings highlight interesting characteristics of synthetic financial reasoning data that merit further investigation. ## related work llm generated synthetic data has been shown to be effective in multiple domains @xcite . @xcite study synthetic text classification data generation by zero-shot and few-shot prompting of an llm, finding the effectiveness to be task dependent. @xcite classify synthetic data generation into three types: answer augmentation, question rephrasing, and new question creation from real samples, noting that their performance varies with the problem. for mathematical reasoning tasks, data augmentation has been shown to be effective @xcite . further, on mathematical tasks models have been shown to benefit from scaling the training data using synthetic data @xcite . in @xcite an instruct tuned model is used to generate synthetic qa pairs from japanese wiki articles, news and contexts from jsquad. these prior studies do not focus on generating synthetic data for numerical multi-hop reasoning over financial reports. for financial question answering, @xcite @xcite create data sets that support the development of multi-step financial reasoning systems. @xcite ) enhance these data with reasoning demonstrations generated by an llm and fine-tune slms using these data sets, demonstrating significant improvement in slm performance. we use the same real datasets primarily as a baseline for evaluating the synthetic variants of these datasets, which we generate us-ing only llms. more recently, finllms @xcite provide a method to generate synthetic data starting with a compilation of a list of common financial formulas, while @xcite generate new contexts for questions in an existing financial dataset to augment the training data. in contrast, we use llms to directly generate question answer pairs for both standard and conversational settings, from financial reports without providing any additional financial knowledge. ## methodology we now present the procedure for generating synthetic multi-hop financial reasoning question answer pairs from financial document excerpts utilizing llms. for these problems, we choose to generate the answer in the form of python code that encodes the reasoning to solve the generated question. the python code is executed using an external python interpreter to generate the actual answer to the financial question. as shown in previous work @xcite , for numerical reasoning, a code generation and external execution strategy is more effective than methods requiring the language model to perform the computation. our approach encompasses two distinct data generation strategies tailored to different settings: first, the financial qa setting in which the llm is prompted to generate financial questions from document excerpts. second, the conversational financial qa setting in which a sequence of interconnected sub-questions are generated that collectively lead to the resolution of a complex financial query. in both cases, the python code (answers) for the synthetic question is generated separately using zero-shot program of thought (pot) approach @xcite . ## conclusion we explored synthetic data creation for financial reasoning in both standard and conversational settings through a multi-step process. to assess the generation methods, synthetic datasets were produced from the same sources used for creating three existing manually annotated financial reasoning datasets. by comparing slms trained on both synthetic and real data, we demonstrated the viability of synthetic data for both standard and conversational financial qa. our findings provide valuable insights into the strengths and limitations of large language models in generating synthetic datasets for financial reasoning tasks.