CTYUN - AI at S em E val-2024 Task 7: Boosting Numerical Understanding with Limited Data Through Effective Data Alignment (2024)

Abstract: Large language models (LLMs) have demonstrated remarkable capabilities in pushing the boundaries of natural language understanding. Nevertheless, the majority of existing open-source LLMs still fall short of meeting satisfactory standards when it comes to addressing numerical problems, especially as the enhancement of their numerical capabilities heavily relies on extensive data.To bridge the gap, we aim to improve the numerical understanding of LLMs by means of efficient data alignment, utilizing only a limited amount of necessary data.Specifically, we first use a data discovery strategy to obtain the most effective portion of numerical data from large datasets. Then, self-augmentation is performed to maximize the potential of the training data. Thirdly, answers of all traning samples are aligned based on some simple rules. Finally, our method achieves the first place in the competition, offering new insights and methodologies for numerical understanding research in LLMs.

## introduction in recent years, the field of natural language processing (nlp) has witnessed remarkable advancements, particularly with the advent of generative large language models @xcite @xcite . these models have predominantly focused on textual data, demonstrating impressive capabilities in understanding and generating human-like text. however, an often-overlooked aspect of these developments is the nuanced role that numerical data plays in fully grasping the semantics of language. this oversight becomes particularly glaring in specialized fields such as stock market analysis, medical diagnostics, and legal decisions @xcite @xcite . in these domains, subtle numerical differences can have far-reaching implications, significantly affecting outcomes and decisions. thus, the ability to understand and work with numbers, in these contexts underscores a critical gap in the semantic understanding capabilities of current language models. acknowledging this deficiency, there has been a growing interest within the nlp community towards enhancing the textual numeracy and computational abilities @xcite of language models. this burgeoning interest has culminated in the introduction of semeval2024's shared task 7. this innovative task is strategically designed to elevate the standards in the field by promoting the development of models that excel not only in literacy but also in computational skills. such models promise to significantly boost usefulness and efficiency across a wide array of applications, ranging from automated financial analysis to predictive healthcare diagnostics and beyond. however, the enhancement of numerical capabilities of llms heavily relies on the inclusion of a large amount of data, posing two significant challenges. on one hand, obtaining high-quality numerical annotated data is costly, as it requires significant economic costs and manual effort from professional annotators. on the other hand, the extensive use of as much data as possible to train the model can diminish the utility of high-value data and lead to increased computational consumption. as illustrated in figure 1 , this paper focuses on how to efficiently use a limited amount of data to improve the numerical capabilities of llms. our goal is to enhance the model's performance in numeral while minimizing costs, such as data annotation expenses and computational requirements for model training. our method propose effective data alignment by employing strategies of data discovery, self-augment and answer alignment. the contributions are summarized as follows: ## system overview in this section, we will introduce our proposed method from several aspects. we start with data analysis of semeval-2024 task 7. then, we present the proposed data discovery, self-augment and alignment strategies. ## experimental setup we utilized the qwen-72b-chat model as our base model, on which we performed full parameter finetuning. the experiments were conducted on nvidia a800 gpu with 80gb of memory. during training, we set the maximum token length to 2048, batch size to 8, and performed gradient accumulation every step. an initial learning rate of 5e-6 was set, employing a cosine decay strategy, for a total of 3 epochs of training. all samples processed by our strategies were used as training data, while the validation set of numhg were utilized for error analysis. for inference, we employed the default inference parameters. ## conclusion in this work, we have demonstrated an approach to enhance numerical understanding in large language models (llms) using limited data through effective data alignment. our method integrated data discovery, self-augment, and answer alignment strategies, and significantly improved the model's performance on numerical reasoning tasks. our success in semeval-2024 task 7 highlights the potential of our method in advancing natural language processing, particularly for enhancing the various basic capabilities of large language models.