JN 666 at S em E val-2024 Task 7: N um E val: Numeral-Aware Language Understanding and Generation (2024)

Abstract: This paper is submitted for SemEval-2027 task 7: Enhancing the Model’s Understanding and Generation of Numerical Values. The dataset for this task is NQuAD, which requires us to select the most suitable option number from four numerical options to fill in the blank in a news article based on the context. Based on the BertForMultipleChoice model, we proposed two new models, MC BERT and SSC BERT, and improved the model’s numerical understanding ability by pre-training the model on numerical comparison tasks. Ultimately, our best-performing model achieved an accuracy rate of 79.40%, which is 9.45% higher than the accuracy rate of NEMo.

## introduction in the field of natural language processing (nlp), the understanding and analysis of textual data have always been the main focus. however, the numerical information in these textual data is often overlooked. numerals play a significant role in our daily life and work, providing rich information such as dates, times, quantities, proportions, and money. although numerals may not occupy a large proportion in the text, their existence is crucial for understanding the meaning of the text. to better evaluate the numerical understanding ability of models, @xcite proposed the numeral-related question answering dataset (nquad) @xcite , which is specifically designed to evaluate and enhance the model's ability in reading comprehension of the numerals in text. in our work, our goal is to improve the performance of the bert model on the nquad @xcite . for this purpose, we propose a new method that can effectively handle numerical information. our experimental results show that our method has achieved significant performance improvement on the nquad @xcite . this proves the effectiveness of our method in handling numerical information and also shows the potential of our method in enhancing the numerical understanding ability of the model. the rest of this paper is organized as follows. section 2 introduces the related work. section 3 introduces the dataset and tasks. section 4 describes our method in detail. section 5 reports our experimental results and analysis. finally, we conclude our work. ## related work in recent years, pre-training tasks have become increasingly prevalent in the field of natural language processing (nlp). the design goal of pretraining tasks is to enhance the model's understanding of natural language through learning from a large amount of unlabeled data, thereby improving the model's performance on specific tasks in the subsequent fine-tuning stage. in this trend, @xcite shows that by pre-training on numerical comparison tasks, the model's understanding of numerals can be significantly improved. the model, after pre-training, also shows a significant performance improvement on other numeral-related tasks. multiple choice @xcite format represents a category within machine reading comprehension tasks. in this context, @xcite proposed a multi-stage multi-task learning framework (mmm) aimed at enhancing the performance of multiple choice tasks, @xcite introduced the dual co-matching network (dcmn+), a model that emulates human problem-solving strategies, and @xcite presents a two-step strategy for enhancing the performance of large language models (llms) on multiple choice tasks. recent research has increasingly recognized the significance of numerical data within text. the nquad @xcite dataset has been instrumental in examining the relationship between numerical values in news headlines and the corresponding figures within the articles. similarly, the fnxl @xcite dataset has brought attention to the numerical data contained within the periodic financial reports submitted by publicly listed companies. the bertformultiplechoice model is a variant of the bert @xcite , specifically designed for multiplechoice tasks. figure 1 is the architecture of bert-formultiplechoice, it adds a multiple-choice classification head on top of the bert model, mainly for handling multiple-choice tasks. the input to the bertformultiplechoice model includes a question and multiple potential answers (options), and the goal of the model is to select the most reasonable answer. this model has shown excellent performance in tasks that require choosing one answer from multiple options, such as reading comprehension and sentiment analysis. nquad @xcite collects news articles from the data vendor, moneydj1, and get the news articles within the period from june 22, 2013 to june 20, 2018. a total of 75,448 chinese news articles are collected. a total of 43,787 news articles are selected, and 46.97% of the headlines contain more than one numeral. the average number of numerals in the headline and in the content are 1.65 and 29.48, respectively. each numeral in each headline is used to form a question, thus nquad @xcite dataset finally obtain 71,998 questions and separate 80% of the instances as the training set and the rest of the instances form a test set. ## result as shown in ## conclusion in this work, we select the bertformultiple model as the baseline model and propose two new models based on it: mc bert and ssc bert. the accuracy of these models all exceeded the results of nemo @xcite , which confirmed the applicability of the bertformultiple model framework for our task, the results also indicate that the method of inserting options into the blanks of the question and individually comparing them with key sentences is effective. we also refer to the method in reference @xcite , pre-trained the bert-base-chinese on numerical comparison task, and use the pre-trained model for subsequent training. the experimental results show that this method can improve the model's numerical understanding ability, thereby making the model perform better on numerical related tasks. finally, we found that the pre-ssc bert model had the highest accuracy, and its accuracy was 9.45% higher than the nemo model @xcite , which further proved the effectiveness of our method.in summary, our research proposes a new model training and optimization strategy, which has been proven to be effective and superior in experiments. however, in analyzing the cases where our model made incorrect judgments, we identify some shortcomings. we notice that humans can easily understand the equivalence between different descriptions of the same thing, but the model cannot. for example, our model cannot understand that '%' and 'cheng'('into')are equivalent, 'eps' and 'earning per share' are equivalent, and it cannot understand that 'january' is 'q1'. to address this issue, we tried to add some specific examples to the dataset of numerical comparison tasks, such as '10% is equal to 1 cheng' and 'eps 100 is equal to earning per share 100', hoping that the model could understand the equivalence between two different descriptions of things through this method. however, the final result did not meet our expectations, indicating that our model still needs improvement in handling these types of problems. we will continue to explore this issue in future research, with the aim of improving the model's understanding ability and accuracy.