CYUT at S em E val-2024 Task 7: A Numerals Augmentation and Feature Enhancement Approach to Numeral Reading Comprehension (2024)

Abstract: This study explores Task 2 in NumEval-2024, which is SemEval-2024(Semantic Evaluation)Task 7 , focusing on the Reading Comprehension of Numerals in Text (Chinese). The datasetutilized in this study is the Numeral-related Question Answering Dataset (NQuAD), and the model employed is BERT. The data undergoes preprocessing, incorporating Numerals Augmentation and Feature Enhancement to numerical entities before model training. Additionally, fine-tuning will also be applied. The result was an accuracy rate of 77.09%, representing a 7.14% improvement compared to the initial NQuAD processing model, referred to as the Numeracy-Enhanced Model (NEMo).

## introduction numeric information holds a crucial significance within narratives across various domains, including medicine, engineering, and finance. @xcite numerals presented in tables @xcite and the content @xcite of a document have garnered considerable attention from researchers. machine-based numeral comprehension stands out as an emerging research area, still in its nascent stages. numeval-2024 task 2(semeval-2024 task 7) focus on reading comprehension of the numerals in text, models are required to identify the correct numerical value from four given options, based on a provided news article. the dataset utilized for this task is nquad @xcite , which is in chinese. the initial model devised for addressing this task is referred to as the numeracy-enhanced model (nemo), which achieves an accuracy of 69.95%. @xcite in this study, a pretrained bert model will be employed to undertake the task, with the objective of enhancing accuracy through data preprocessing, numeral augmentation, feature enhancement, and fine-tuning. 1 1 †contact author the remaining sections of this study are structured as follows: in the second section, we delve into a discussion of the data and its preprocessing. the data undergoes denoising through the removal of special symbols. moving on to the third section, we thoroughly examine the methodology employed, the results obtained, and the subsequent discussion. the data undergoes further denoising through stop word removal. additionally, numeral augmentation and feature enhancement are introduced. numeral augmentation reduces dependence on specific numerical values, while feature enhancement aims to compel the model to pay attention to numerals. as a result, our model achieves an accuracy of 77.09% on the nquad dataset. the concluding section presents a summary of the insights gained, along with considerations for future research endeavors. although the model is performing well, there are opportunities for further enhancement. we will delve into these matters in the error analysis and discussion section. ## method in this study, the pretrained bert model served as the baseline. to enhance model performance, two steps were implemented. first, bert with stop word removal (bert-swr) was introduced. this step contributes to improved model performance through dimensionality reduction, noise reduction, and enhanced generalization. building upon bert-swr, additional measures were taken, including numerals augmentation and feature enhancement(bert-nafe). numerals in the text were replaced with a special symbol and the corresponding answer index. this substitution aimed to eliminate the model's consideration of the meaningless numerical values in the article. the use of the special symbol, represented as cashtag, compelled the model to prioritize attention to the numerals within the text. this additional step further increased the model's accuracy. please be aware that prior to evaluation, fine-tuning was applied to each model. ## result ## discussion and error analysis 6.1 discussion the baseline model, bert, achieved an accuracy of 67.95% after fine-tuning, closely approaching the initial model nemo. upon the removal of stop words from the text, there was a notable improvement of approximately 3% in accuracy. considering dimensionality reduction, stop words, characterized as high-frequency terms, pervade most texts. their exclusion significantly diminishes the dimensionality of the feature space, thereby reducing computational complexity. consequently, this eases the requirements on both model training and inference processes. the elimination of stop words contributes to noise reduction. these words often lack essential information, and retaining them may introduce interference, hindering the model's ability to concentrate on learning crucial information. their removal enables the model to focus its attention and learning capacity on genuinely meaningful vocabulary. additionally, excluding stop words during model training enhances the comprehension and generalization of the text. stop words frequently serve as grammatical connectors without conveying specific semantic information. their removal allows the model to more effectively concentrate on learning the actual relationships between words, unburdened by the intricacies of entire sentence structures. in addition to bert-swr, numerals augmentation and feature enhancement were incorporated to further elevate the accuracy to 77.09%. two reasons account for this enhancement. to address the model's inclination to overly rely on particular numerical values encountered during training, numeral augmentation aims to counteract this fixation. this is achieved by introducing variations in the representation of numbers, guiding the model to develop a less rigid fixation on specific instances and fostering a more generalized understanding of numerical information. another central focus of the augmentation process is the mitigation of numerical ambiguity. numerals within a given context may have multiple interpretations or ambiguous meanings. to alleviate this ambiguity, the augmentation process exposes the model to diverse representations of numbers. this exposure aids the model in adapting to different contexts and facilitates the disambiguation of numeral meanings based on the surrounding textual information. after computing the current precision and recall from figure 6 , it was observed that the system tends to produce class 2 outputs at a rate of 35.9%, surpassing the training set's rate of 33.1%. conversely, the system's output proportion for class 3 is notably low, standing at only 15.8%. perhaps adjusting the training set's distribution to a balanced 1:1:1:1 ratio could enhance the system's performance. it is worth noting that the numerical labels 0, 1, 2, and 3 used in the numerals augmentation step lack intrinsic significance and merely denote positions. for future work, adjusting the distribution ratio should be a straightforward task. various symbols and triple symbols for feature enhancement were also experimented with. in the case of different symbols, comparable performance was observed. however, when employing triple symbols, there was a slight decrease in accuracy, with approximately 74%, representing a 3% decline. ## conclusions this study has achieved significant performance improvements through fine-tuning the bert model and optimizing text processing methods. firstly, during the fine-tuning process, we observed a notable increase in model accuracy by removing stop words from the text, particularly high-frequency vocabulary. the exclusion of stop words not only reduced the dimensionality of the feature space, lowering computational complexity, but also contributed to noise reduction, enabling the model to better focus on meaningful vocabulary. secondly, the introduction of numerals augmentation and feature enhancement further elevated the model's accuracy. numerals augmentation, by introducing variations in the representation of numbers, assisted the model in overcoming overreliance on specific numerical values, fostering a more generalized understanding of numeric information. additionally, feature enhancement strategies helped alleviate numerical ambiguity, allowing the model to adapt to different contexts and accurately interpret numeric meanings. in summary, our research findings suggest that, building upon the bert model, fine-tuning and text processing optimizations can effectively enhance performance in numeral reading comprehension.