Investigating the effectiveness of length based rewards in DPO for building Conversational Financial Question Answering Systems (2025)

Abstract: In this paper, we address the numerical reasoning challenges of financial question-answering systems. We propose a two-stage approach where models first generate intermediate calculations and then produce the final answer. We perform two set of experiments to evaluate the performance of our approach. In the first, we compare single-step and multi-step approaches, demonstrating that incorporating intermediate calculations significantly improves numerical accuracy. In the second experiment, we compare traditional DPO and iterative DPO (iDPO) with length-regularized DPO. We show that while traditional DPO reduced parsing errors, it introduces verbosity; iDPO improves reasoning iteratively but faces diminishing returns. On the other hand, Length-regularized DPO reduces verbosity of intermediate calculation as well as enhances numerical accuracy across all models. These results highlight the potential of combining intermediate reasoning steps with domain-specific optimizations to build robust financial question-answering systems.

## introduction finance has emerged as a prominent area of focus for large language models (llms) @xcite @xcite @xcite since financial data @xcite presents a unique set of complexities and challenges @xcite . these challenges arise from the intricate nature of financial texts and often require precise numerical calculations and an understanding of contextual dependencies that general-purpose llms struggle with. common errors often include failure to perform precise numerical reasoning and generating inaccurate or irrelevant information due to an inadequate understanding of the context @xcite . ## length regularization as explicit reward in dpo consider r to be a set of earnings reports, and @xmath0. let @xmath1. the task of question answering on earnings reports can be expressed as maximizing p (y|q, r). in this paper, we fine-tuned our models to output a calculation c first and then arrive at the answer y based on c. our approach can be expressed as maximizing p (c|q, r). a typical challenge in applying dpo to fine-tuning llms is 'length bias' @xcite @xcite -the tendency of models to generate unnecessarily long or convoluted outputs, especially when performing complex reasoning. in the context of our task, models finetuned using dpo without explicit length control exhibited this bias, frequently generating verbose and convoluted calculations for relatively straightforward financial questions. this over-generation introduces new errors, particularly in mathematical computations where simpler expressions are preferable. previous works have explicitly highlighted the importance of length-based regularization @xcite in the context of classical rlhf pipelines and the dpo variants. inspired by this, we introduced length regularization as an explicit reward in the dpo framework to mitigate this issue. specifically, we penalize overly long calculations, encouraging the model to generate more concise outputs without sacrificing the necessary depth of reasoning. by incorporating this reward, we aim to balance the model's preference for providing comprehensive answers with the need for clarity and precision in financial contexts. ## experiments in this section, we first describe our experimental setup and implementation details. we begin with off-the-shelf llms and fine-tune them using the following approaches: (a) supervised fine-tuning, (b) traditional dpo (one-step dpo), (c) iterative dpo @xcite , and (d) dpo with length regularization as an explicit reward. finally, we provide a detailed analysis of the effectiveness and impact of our proposed approach. ## conclusion in this study we have investigated the effectiveness of employing length regularzed dpo in multi step fashion towards answering financial questions. our first experiment demonstrated that introducing intermediate calculations before generating final answers improved multi-step reasoning and accuracy. our second experiment showed that traditional dpo reduced parsing errors but introduced verbosity, while iterative dpo (idpo) iteratively improved reasoning but faced diminishing returns. length-regularized dpo emerged as the most effective approach, balancing concise outputs with reasoning depth, reducing verbosity, improving numerical accuracy, and enhancing efficiency across all tested models. these findings underscore the importance of domain-specific strategies to improve reliability and precision in financial questionanswering systems. prompt model no et 0.1% et hybrid mistral-7b 63.53 (48.89) 65.19 reasoning llama-3.2 61.22 (48.27) 64.59 prompt phi-3 62.97 (47.77) 65.02 gpt-3.5 turbo 61.34 (60.2) 62.61 gpt-4o 85.97 (84.81) 86.31