YNU - HPCC at S em E val-2024 Task 7: Instruction Fine-tuning Models for Numerical Understanding and Generation (2024)

Abstract: This paper presents our systems for Task 7, Numeral-Aware Language Understanding and Generation of SemEval 2024. As participants of Task 7, we engage in all subtasks and implement corresponding systems for each subtask. All subtasks cover three aspects: Quantitative understanding (English), Reading Comprehension of the Numbers in the text (Chinese), and Numeral-Aware Headline Generation (English). Our approach explores employing instruction-tuned models (Flan-T5) or text-to-text models (T5) to accomplish the respective subtasks. We implement the instruction fine-tuning with or without demonstrations and employ similarity-based retrieval or manual methods to construct demonstrations for each example in instruction fine-tuning. Moreover, we reformulate the model’s output into a chain-of-thought format with calculation expressions to enhance its reasoning performance for reasoning subtasks. The competitive results in all subtasks demonstrate the effectiveness of our systems.

## introduction in numerous domains, precise numerical information within text is decisive in decision-making and planning. understanding and generating text-numbers would be beneficial for improving the model's performance on specific tasks. however, it poses challenges for existing models. also, previous research indicates that current models struggle to properly represent textual numbers @xcite , often leading to inaccuracies. therefore, task 7 of semeval @xcite ) 2024 focuses on numerically-aware language comprehension and generation, which includes quantitative understanding @xcite , reading comprehension of the numerals in text @xcite , and numeral-aware headline generation @xcite . we explored all the subtasks of task 7 and designed corresponding systems for each subtask. our work and contributions can be summarized as follows: for subtask 1, we adopt the paradigm of instruction tuning @xcite to complete all subtasks and explore manually crafting instances. our results demonstrate that the instruction tuning model (flan-t5) @xcite performs comparably to the bert model @xcite on the quantitative understanding task. for subtask 2, we utilized the mt5 model @xcite pre-trained on multilingual corpus and the randeng-t5 @xcite pre-trained on chinese corpus to implement the respective systems, as this task involves chinese. consistent with task 1, we designed an instruction template for inputs and employed instruction fine-tuning. for subtask 3, similar instances are retrieved and organized into the input-output format to further enhance model's performance in in-context learning. specifically, we structured the model's output into the format of chain-of-thought (cot) @xcite and inserted calculation expressions to improve model's reasoning performance. our system achieved the highest scores of rouge, bertscore, and moverscore in headline generation while ranking 3 th in numerical reasoning task. the remainder of this paper is organized as follows. in section 2, we describe the related work of our system. the system overview is presented in section 3. the details of the experiments, main results, and a conclusion are drawn in sections 4, 5, and 6, respectively. ## related work in-context learning. as a novel paradigm, incontext learning @xcite has proven to enable large language models to adapt to unseen tasks with instruction and a few lm predict the magnitude for [num] in the headline: b-sens streak to 4-[num] win over norfolk magnitude: choose a correct answer to the following questions. question: a bmx bike can ride 1 m on a quarter pipe and 90 cm on a half pipe. this means the bmx bike will get hotter on the? option 1: quarter pipe option 2: half pipe generate a headline for the news, following this example: news: police are still hunting for an armed and dangerous man... headline: 3rd victim dead in quarry shooting; manhunt still on news: turns out you won't even have to leave your house... headline: 1 option 2 you can watch the interview at 1pm demonstrations, and it doesn't conduct any parameter updates. furthermore, selecting semantically similar instances can further enhance the model's performance. @xcite . recent work has also applied in-context learning to fine-tuning small models @xcite . ## main result and analysis comparison results on quantitative 101 as results shown in comparison results on nquad as shown in comparison results on numhg tables 5 and 6 show that larger models perform better on both headline generation and numerical reasoning. ## conclusion during task 7 of semeval2024, we participated in all the subtasks and implemented the corresponding systems by instruction fine-tuning. we utilized instruction fine-tuning with demonstrations to expand its format. we also reformulated the output in the form of a chain of thought to improve the model's reasoning abilities. our approach proved to be highly effective by outstanding performance across all the subtasks. in future work, we plan to further explore the impact of varying instance quantities, instruction templates, and model sizes on the results.