MAMET at S em E val-2024 Task 7: Supervised Enhanced Reasoning Agent Model (2024)

Abstract: In the intersection of language understanding and numerical reasoning, a formidable challenge arises in natural language processing (NLP). Our study delves into the realm of NumEval, focusing on numeral-aware language understanding and generation using the QP, QQA and QNLI datasets. We harness the potential of the Orca2 model, Fine-tuning it in both normal and Chain-of-Thought modes with prompt tuning to enhance accuracy. Despite initial conjectures, our findings reveal intriguing disparities in model performance. While standard training methodologies yield commendable accuracy rates. The core contribution of this work lies in its elucidation of the intricate interplay between dataset sequencing and model performance. We expected to achieve a general model with the Fine Tuning model on the QP and QNLI datasets respectively, which has good accuracy in all three datasets. However, this goal was not achieved, and in order to achieve this goal, we introduce our structure.

## introduction in the realm of natural language understanding (nlu), the quest for models capable of comprehending and reasoning with textual data has been a longstanding pursuit. the numeval task, focusing on numeral-aware language understanding and generation, stands at the frontier of this endeavor, challenging researchers to develop models adept at grasping numerical information embedded within linguistic contexts. in this study, we delve into the intricacies of fine-tuning methodologies and their impact on the performance of language models, particularly focusing on the qp, qqa and qnli datasets. @xcite the primary challenge in nlu lies in imbuing models with the ability to interpret and reason with textual information akin to human cognition. traditional approaches often face hurdles in capturing the nuances of language, especially when numerical data intertwines with linguistic expressions. one possible cause of this problem is that numerals can have various notations, some of which are difficult to understand from their subwords. while models like orca2 @xcite , an instance of large language models (llms), exhibit remarkable capabilities, their performance nuances in understanding numeral-aware contexts warrant deeper exploration. the qp, qqa and qnli datasets @xcite , @xcite , @xcite , @xcite serve as test for evaluating the efficacy of language models in understanding questions, questionanswering and natural language inference, respectively. these datasets present a diverse array of linguistic challenges, including numeral-aware reasoning, prompting the need for sophisticated training strategies. our study uses the orca2 model, an advanced llm known for its language comprehension skills. through meticulous fine-tuning and evaluation on the qp, qqa and qnli datasets. we aim to catalyze discourse and innovation in the field of nlu, steering towards more robust and nuanced language models capable of navigating the complexities of numeral-aware language understanding and generation in real-world scenarios. however, the road to achieving robust language understanding is fraught with challenges, chief among them being the inherent ambiguity and variability present in natural language. numerical information adds an additional layer of complexity, requiring models to not only parse linguistic constructs but also interpret and reason with numerical data embedded within textual contexts. conventional training methodologies, while effective to a certain extent, often fall short in encapsulating the intricate interplay between linguistic semantics and numerical reasoning. the advent of large-scale language models has undoubtedly propelled the field forward, but their performance on numeral-aware tasks remains an area ripe for exploration and refinement. after conducting several experiments, we have determined that fine-tuning a model for a specific subtask yields significantly higher accuracy compared to fine-tuning a model across all subtasks. our attempt to fine-tune a generalized model across all subtasks while maintaining accuracy proved unsuccessful. upon reviewing the results, we recognized the effectiveness of the orca 2 model utilizing the lora method for each subtask. consequently, we trained the model using qlora, resulting in improved accuracy. to establish a robust framework for addressing a range of reasoning 1 1058 subtasks, we propose a structured approach that employs an agent as a supervisor capable of categorizing subtasks. this agent determines which of our fine-tuned models should address each task. ## related work the intersection of numerical evaluation (numeval) and natural language processing (nlp) has witnessed a surge in research endeavors aimed at refining models' numerical comprehension and processing capabilities. within this burgeoning field, a multitude of studies have delved into diverse methodologies and frameworks to deepen our understanding of numeracy in linguistic contexts. @xcite a seminal study by @xcite introduced the utilization of digit-based encoders to represent numerals, laying the groundwork for subsequent investigations into numerical representation methods. expanding upon this, @xcite pioneered the exploration of scientific notation for numerical representation, shedding light on the efficacy of alternative numerical formats in quantitative skill tasks. these methodologies not only deepen our understanding of numerals' linguistic representation but also pave the way for novel approaches in numerical evaluation tasks. pretraining tasks have played a pivotal role in enhancing language models' capabilities in comprehending numerical language. @xcite revolutionized the field by introducing masked language model (mlm) and next sentence prediction (nsp) tasks, ushering in a new era of transformer-based nlp research. building upon this foundation, @xcite proposed the document relation prediction (drp) task, specifically designed to bolster models' performance in multi-hop reasoning and multi-document understanding tasks. these pretraining paradigms have significantly enriched models' numerical understanding capabilities, underscoring the pivotal role of pretraining in enhancing nlp models' numerical acumen. in parallel, pre-finetuning strategies have emerged as a promising avenue for enhancing models' numerical comprehension. the comparative numbers dataset (cnd), introduced as a pre-finetuning resource, has garnered attention for its efficacy in enhancing models' numerical reasoning abilities. experimental investigations leveraging bert, roberta, linkbert and finbert @xcite have demonstrated notable improvements in models' performance across various numerical evaluation tasks, underscoring the potential of pre-finetuning methodologies in augmenting models' numerical understanding. @xcite generally, the landscape of numeval research is characterized by a dynamic interplay of numerical representation methods, pretraining tasks and pre-finetuning strategies, each contributing to the advancement of language models' proficiency in numerical understanding and processing. @xcite the results of our tests are very promising in this field and on the tested datasets, the accuracy is higher than the accuracy of the reference article. ## approach through our exploration with the fine tuning model, we discovered commendable accuracy in each sub-task individually. upon scrutinizing the test outcomes, a noteworthy observation emerged: encoding numerical values within the text as statement-char significantly boosts accuracy in the qnli task. however, a pertinent challenge persists: determining the appropriate agent for input assignment. to address this challenge, we devised a framework. initially, the input undergoes classification by an agent, ensuring its allocation to the most suitable model. in the case of the qnli task, we preprocess words into statement-char format before directing them to the designated agent. furthermore, leveraging the orca model's remarkable 100% accuracy in the qqa task, we opted to employ the base model as our agent. this strategic decision underscores our commitment to optimizing task performance and model efficacy. ## analysis our analysis has illuminated the intricate nature of numeral-aware language tasks, emphasizing the imperative for ongoing scrutiny and enhancement of model architectures and training methods. the insights derived from our investigation notably elucidate the performance and adaptability of the orca2 model within the numeval task domain. upon reflection, we determined that optimizing the generality of the structure entails employing expert agents for individual sub-tasks, facilitated by a supervisory agent to assign tasks effectively. this strategic adjustment yielded heightened accuracy across all sub-tasks, marking a significant advancement in our approach. ## conclusion in conclusion, our analysis highlights the intricate interplay between dataset characteristics, model architectures, and training methodologies in the numeval task. while achieving significant advancements in numeralaware language understanding and generation, our study underscores the importance of comprehensive evaluations encompassing both quantitative metrics and qualitative assessments to unravel the complexities of numerical reasoning in natural language understanding tasks. moving forward, further research into adaptive learning strategies and nuanced dataset annotations promises to enrich our understanding and advancement in numeralaware language processing tasks.