Enhancing Numerical Reasoning with the Guidance of Reliable Reasoning Processes (2024)

Abstract: Numerical reasoning is an essential ability for NLP systems to handle numeric information. Recent research indicates that fine-tuning a small-scale model to learn generating reasoning processes alongside answers can significantly enhance performance. However, current methods have the limitation that most methods generate reasoning processes with large language models (LLMs), which are “unreliable” since such processes could contain information unrelated to the answer. To address this limitation, we introduce enhancing numerical reasoning with reliable processes (Encore), which derives the reliable reasoning process by decomposing the answer formula, ensuring which fully supports the answer. Nevertheless, models could lack enough data to learn the reasoning process generation adequately, since our method generates only one single reasoning process for one formula. To overcome this difficulty, we present a series of pre-training tasks to help models learn the reasoning process generation with synthesized data. The experiments show that Encore yields improvement on all five experimental datasets with an average of 1.8%, proving the effectiveness of our method.

## introduction numerical reasoning is an essential ability for nlp systems to handle arithmetic questions in real scenarios, which refers to generating answers to numerical questions with given evidence @xcite . the evidence is to support reasoning since the question could not furnish all necessary information, as seen in passages and tables in numerical data @xcite . the answer generated encompasses various elements, including values @xcite , programs @xcite , and formulas (zhu et al., 2021a) 2 . ## background the numerical reasoning task is to generate single or multiple formulas as answers based on the given question as well as the textual and tabular evidence. about the input, textual evidence contains several paragraphs, and tabular evidence consists of one or more tables, where the first row and column are called table headers, which describe information about the cells of the corresponding rows or columns. about the output, one formula consists of operators and operands (e.g., 2 + 1 × 3). the operands refer to the values manipulated or processed in the formula (e.g., 2, 1, 3). the operators are arithmetic symbols or functions that are to be performed on the operands (e.g., +, ×). however, in practical applications, answers could not be annotated with formulas. we also try to apply our method to such data without formulas. for the questions with simple calculations (e.g., drop @xcite )), we can directly generate the formulas following the previous work foot_0 . for the more complex calculations (e.g., gsm8k @xcite )), we employ llms with in-context learning, which can generate answers with a few samples without fine-tuning, to prove the effectiveness of the reasoning process we designed. ## methodology in this section, we present our numerical reasoning method called encore. first, we introduce the pipeline of encore, which generates reliable reasoning processes fully supporting the answers ( § 3.1). then, we propose three pre-training tasks based on encore to enhance the generation performance of the reasoning process ( § 3.2). ## conclusion we propose a novel numerical reasoning method called encore to address the limitations of the reasoning process generation. compared with previous methods, encore can guarantee to generate reliable reasoning processes that fully support the answer and aim models to learn to generate the process with pre-training. according to experiments, encore brings significant performance improvement over all five experimental datasets, leading to an average improvement of 1.8% compared with baselines, which shows the effectiveness and generalization of our method. meanwhile, our method is superior by about 10% in comparison to the reasoning process generated by gpt-3.5-turbo, which proves the higher quality of the reasoning processes generated by encore.