N um D ecoders at S em E val-2024 Task 7: F lan T 5 and GPT enhanced with C o T for Numerical Reasoning (2024)

Abstract: In this paper we present a Chain-of-Thought enhanced solution for large language models, including flanT5 and GPT 3.5 Turbo, aimed at solving mathematical problems to fill in blanks from news headlines. Our approach builds on adata augmentation strategy that incorporates additional mathematical reasoning observations into the original dataset sourced from another mathematical corpus. Both automatic and manual annotations are applied to explicitly describe the reasoning steps required for models to reach the target answer. We employ an ensemble majority voting method to generate finalpredictions across our best-performing models. Our analysis reveals that while larger models trained with our enhanced dataset achieve significant gains (91% accuracy, ranking 5th on the NumEval Task 3 leaderboard), smaller models do not experience improvements and may even see a decrease in overall accuracy. We conclude that improving our automatic an-notations via crowdsourcing methods can be a worthwhile endeavor to train larger models than the ones from this study to see the most accurate results.

## introduction numeval is a task first introduced in 2024 @xcite building on previous work such as @xcite 's fine-grained sentiment analysis (semeval-2017 task 5) and @xcite 's clinical inference (semeval-2023 task 7). these prior tasks highlighted the importance of understanding numerical values in legal and medical contexts for determining outcomes. the primary objective of numeval is to perform quantitative reasoning to generate numerical values corresponding to provided contexts. in this project, we particularly focused on subtask 1 of task 3 @xcite where our * all authors have equal contributions system must execute several mathematical calculations based on information from a provided passage to yield a numerical result used to fill in a headline with a blank. for instance, to complete the cia cited concerns about snowden ____ years ago headline, the model must subtract the article's publishing date by the explicitly stated date in the article @xcite . some entries involve a series of multiple mathematical operations that the model must perform. although numerical reasoning continues to present challenges to large language models (llms), advancements in larger models like deepseekmath @xcite demonstrate promising capabilities in solving mathematical computations. deepseekmath is finetuned using different mathematical datasets and evaluated using chain-of-thought (cot) prompting to provide intermediate reasoning steps. inspired by cot systems, we have developed a system pipeline that trains an encoder-decoder flant5 @xcite and an open source gpt 3.5 version foot_0 with additional mathematical corpora. these corpora include the discrete reasoning over the content of paragraph (drop) dataset @xcite and another dataset which was manually and automatically annotated to include reasoning steps to reach the desired response. the core idea is that explicit intermediate reasoning, akin to chain-of-thought prompts, can enhance a model's quantitative reasoning capabilities @xcite . in our revised approach, not only do we use smaller models (θ ≤ 1b) foot_1 , but we also utilize multiple pipelines to determine the conditions under which our model achieves the highest accuracy. firstly, we establish a baseline by fine-tuning with the provided dataset @xcite , then we incorporate additional observations from the drop dataset into our training data. thirdly, we adopt a chain-of-thought (cot) approach, fine-tuning both a flant5 model and a generative open-source openai model (gpt 3.5 turbo) with more detailed inputs and outputs, including string normalizations and quantitative reasoning steps. finally, we employ an ensemble majority voting method to select the best results from these models, resulting in a 91% accuracy and 5th place on the leaderboard of the numeval competition 3 . ## related works through pre-training on a vast amount of text data, llms can develop a broad knowledge base encompassing numerical concepts, arithmetic operations, and mathematical relationships. @xcite propose a language model named minerva, which demonstrates strong performance on various quantitative reasoning tasks, including undergraduate-level physics or chemistry problems. numerical reasoning has been extensively studied across diverse contexts, including word embedding @xcite @xcite and math word problems @xcite . within the domain of question answering, several approaches have been proposed. @xcite present a framework called diagnosing numerical capabilities (dnc), which involves two stages: recognition of numbers in the context and question to treat them as candidate operands, followed by the correct selection of operands and operations based on understanding questions and context. @xcite proposes an attention-masked reasoning model that learns to leverage the number-related context to alleviate the over-reliance on parametric knowledge and enhance the numerical reasoning capabilities of the qa model. other studies, such as those by @xcite and @xcite , explore the infusion of external knowledge to augment the numerical reasoning skills of the models. @xcite focus on numerical reasoning over text (nrot) using t5 models, employing five training pipelines and multitasking training to progressively enhance model performance through tasks such as general reading comprehension and fine-tuning on the drop dataset @xcite . additionally, in reasoning tasks, chain-of-thought prompting has shown promise in improving the performance of large language models (ling et al., 3 github repository for our system 2024). while chain-of-thought (cot) allows models to generate more comprehensive reasoning processes, it also introduces challenges such as hallucinations and accumulated errors. to mitigate these issues, the authors propose enabling explicit and deductive rigorous reasoning within language models. they emphasize the importance of self-verification for trustworthiness, which leads to significantly improved answer correctness in reasoning tasks. drawing inspiration from these cot-based methods, we incorporate them into our approach due to their superior performance in numerical reasoning tasks. ## system description in our system, we defined three main pipelines that were compared against a baseline encoder-decoder model. specifically, we used an instruction finetune model version (flan) of the text-to-text transfer transformer (t5) @xcite . this flant5 model underwent fine-tuning in its small, base, and large versions, employing a learning rate of 5e-5 for 5 epochs and a batch size of 2. ## results based on the results presented in tables 2 and 3 , we observe that the difference in performance between the small and base flant5 models is not par-def get_ans_sent(item): @xmath0:get_copy_placeholder,"trans":get_trans_placeholder, "span":get_span_placeholder,"round":get_round_placeholder,"paraphrase": ↩→ get_paraphrase_placeholder, "subtract":get_subtract_placeholder, "sround":get_round_placeholder} for operation, function in operations.items(): if check_calculation(item, operation): return function(item) return f"so the answer is {item['ans']}" ## discussion our cot results, as observed in tables 2 and 3 align with the findings reported by @xcite , indicating that smaller models do not experience significant gains when using prompting, partly due to their fewer parameters. in their study, it is explicitly mentioned that models in the range of 100 billion parameters or more exhibit the highest gains. however, all of the flant5 models we utilized have significantly fewer parameters, failing to reach the 1 billion mark @xcite . we believe that conducting cot experiments with the xl and xxl versions of these models would likely result in much more significant improvements. ## conclusion in this paper we observed that introducing additional observations with detailed reasoning steps can enhance a model's ability to solve mathematical problems while also highlighting areas where its reasoning may fall short. nevertheless, our results suggest that larger models may derive the most benefit from a cot + finetuning approach. because of this, we argue that leveraging larger llms could lead to even greater gains in quantitative reasoning tasks. furthermore, while we have provided access to our open dataset foot_3 , we recognize the importance of improving automatic annotations through crowdsourcing to achieve more accurate results. given that our automatic annotations sometimes exhibit issues in reasoning steps compared to manual annotations, and certain entries in the original dataset are ambiguous or erroneous, we emphasize the necessity of data cleanup to enhance mathematical reasoning in language models.