ZXQ at S em E val-2024 Task 7: Fine-tuning GPT -3.5-Turbo for Numerical Reasoning (2024)

Abstract: In this paper, we present our system for the SemEval-2024 Task 7, i.e., NumEval subtask 3: Numericial Reasoning. Given a news article and its headline, the numerical reasoning task involves creating a system to compute the intentionally excluded number within the news headline. We propose a fine-tuned GPT-3.5-turbo model, specifically engineered to deduce missing numerals directly from the content of news article. The model is trained with a human-engineered prompt that itegrates the news content and the masked headline, tailoring its accuracy for the designated task. It achieves an accuracy of 0.94 on the test data and secures the second position in the official leaderboard. An examination on the system’s inference results reveals its commendable accuracy in identifying correct numerals when they can be directly “copied” from the articles. However, the error rates increase when it comes to some ambiguous operations such as rounding.

## related work this work draws inspiration from two key research areas: instruction tuning and leveraging intermediate reasoning steps for solving math word problems. instruction tuning shows that incorporating prompts or instructions into training datasets, as proposed by @xcite , can improve the language models' performance on unseen data. @xcite also suggest that using diverse prompts to augment training datasets can help language models to achieve better generalization. regarding the use of intermediate reasoning steps, some researchers express these steps in symbolic format, such as chiang and chen (2019) who train language models to generate equations leading to final answers, and @xcite who map math word problems to predefined operations. others opt for natural language descriptions, like @xcite , who propose a system utilizing a language model to generate reasoning steps and final answers in natural language, along with a verifier model to assess the generated answers. ## problem definition the numhg dataset comprises 21,157 pieces of news articles, each ranging between 200 to 300 words in the content. the headlines of these news articles include one or more numerals, with one numeral intentionally omitted from each headline. the novelty of the numhg dataset lies in its provision of meticulous annotations detailing the computational processes used to arrive at the missing numerals in the headlines. this dataset articulates 12 fundamental operations for numeral computation, encompassing actions such as copying, addition, and multiplication. the pre-defined operations are listed in the sub-task of numerical reasoning requires us to develop a system capable of accurately calculating the omitted numerals based on the news contents and potentially utilizing the provided annotations. ## system overview in this section, we provide a detailed introduction to our system. figure 1 illustrates a general outline of our system. ## experiment setup the dataset allocated for the sub-task of numerical reasoning comprises 21,157 instances, which is further split into 80% for training data and 20% for test data. fine-tuning of the gpt-3.5-turbo-0613 model is performed via the openai api, adhering to the guidelines outlined in the openai api documentation. the hyper-parameters for model training, including learning rate, batch size, and epochs, are configured to auto. throughout the training phase, evaluation metrics such as training loss, training token accuracy, validation loss, and validation token accuracy are provided by openai. following the completion of the training process, the fine-tuned model is employed to generate inferences on the test data with temperature set to default. figure 3: one example for the prompts used for instructing the model to also generate the operations 6 results the fine-tuned gpt-3.5-turbo-0613 model from the experiment 1, which calculates the numerals directly from news content, demonstrates a commendable accuracy of 0.94, securing the second position on the leaderboard. the model from the experiment 2, which also output the intermediate operations required to arrive at the numerals, only achieves an accuracy of 0.90. an analysis of model errors has been conducted, with detailed statistics presented in the test data comprises 4,921 instances, featuring a total of 5,237 operators in the annotations. the training phase. the unanswerable questions refer to instances identified by annotators when the numerals cannot be inferred from the news content. figure 4 shows one example of the unanswerable questions in the test data. another anomaly arises as the model trained for generating intermediate operations shows lower accuracy, contradicting prior works' conclusion that incorporating intermediate reasoning steps in symbolic formats should enhance language model performance by @xcite and @xcite . to improve the model's ability to utilize annotations and boost overall accuracy, future research should explore alternative methods such as experimenting with different language models, adjusting model architecture, and employing chain-of-thought prompting @xcite . ## conclusion in this paper, we propose to finetune the gpt-3.5-turbo specifically tailored for handling numerical reasoning in the headline generation con-text. through the carefully engineered prompt that aggregates the content of news articles and the masked headlines during the fine-tuning process, we achieved an accuracy of 0.94 and ranked second in the official leaderboard. however, our model exhibits relatively high error rates particularly in operations such as rounding numbers. additionally, further development is needed to better utilize the annotations to improve the model's accuracy.