HIJLI _ JU at S em E val-2024 Task 7: Enhancing Quantitative Question Answering Using Fine-tuned BERT Models (2024)

Abstract: In data and numerical analysis, Quantitative Question Answering (QQA) becomes a crucial instrument that provides deep insights for analyzing large datasets and helps make well-informed decisions in industries such as finance, healthcare, and business. This paper explores the “HIJLI_JU” team’s involvement in NumEval Task 1 within SemEval 2024, with a particular emphasis on quantitative comprehension. Specifically, our method addresses numerical complexities by fine-tuning a BERT model for sophisticated multiple-choice question answering, leveraging the Hugging Face ecosystem. The effectiveness of our QQA model is assessed using a variety of metrics, with an emphasis on the f1_score() of the scikit-learn library. Thorough analysis of the macro-F1, micro-F1, weighted-F1, average, and binary-F1 scores yields detailed insights into the model’s performance in a range of question formats.

## introduction quantitative question answering (qqa) is a crucial tool in the large field of data and numerical analysis because it uses sophisticated computer methods to extract and interpret significant information from large datasets. imagine it as a strong force that has particular sway over important industries like finance, healthcare, and business, where it plays a crucial role in forecasting trends and assisting in the making of well-informed decisions. qqa acts as a driving force behind wise decisions by skillfully converting apparently complicated data into useful knowledge and clearing a way through the complexities of numerical data. our team, "hijli_ju," participated in task 1 (quantitative understanding) of numeval within semeval 2024 foot_0 , thereby actively engaging in the competitive landscape. an annual series of international natural language processing (nlp) com-petitions called semeval (semantic evaluation) evaluates the state-of-the-art in a range of tasks pertaining to semantic analysis and understanding. these challenges serve as a forum for practitioners and scholars from both academia and industry to investigate and expand the field of computational linguistics. semeval is well known for its broad range of tasks, which address a variety of difficulties in natural language processing. the presented qqa task in the context of the se-meval 2024 numeval competition provides a platform for researchers and developers to showcase advancements in quantitative understanding (num). semeval, an annual challenge, encompasses diverse language tasks, including sentiment analysis and word meanings, contributing to the ongoing progress in systems designed for language understanding and processing. this paper is organized as follows: in section 2, a survey of related literature is presented, and in section 3, a detailed description of the dataset is provided. section 4 explores the details of our suggested model while section 5 explains the experimental setup. experiments using our model are shown in section 6, and observations are discussed in section 7. bringing everything together, we wrap up the paper in section 8 and offer some suggestions for future directions for study. sandip presented a novel approach to enhance science-based multiple choice question answering (mcqa) systems by leveraging distributed semantic similarity and a classification approach. three models (model 1, model 2, and model 3) were developed to address differences in dataset formats, specifically focusing on ijcnlp task 5 and sciq datasets @xcite . zuccon looks into using fancy word techniques from computer language models for finding information better. they use special methods to understand words and put them into a translation model. the results show that this approach improves how well information is found, and it's flexible -it works well even if the word understanding is done differently or comes from a different set of information @xcite . researchers in the field of quantitative question answering (qqa) have been investigating ways to enhance computer systems' capacity to respond to numerical questions. they've tried a range of tactics, such as deep learning and sophisticated machine learning. to improve response accuracy, certain studies might incorporate external data. for qqa, standardized tests (datasets) covering a range of numerical questions in disciplines like science and finance are being developed. they're also developing equitable methods to evaluate the efficacy of these q&a platforms. the brittleness of existing ai systems, including large-scale language models, in arithmetic reasoning within natural language understanding is addressed by the proposed multi-task benchmark numglue @xcite . in order to prepare ai systems for increasingly difficult mathematical tasks, the benchmark attempts to promote the development of systems that are able to reason robustly about arithmetic in language. equate is a framework assessing quantitative reasoning in textual entailment for natural language understanding systems @xcite . state-of-the-art models don't consistently outperform a basic baseline, highlighting a potential gap in implicit quantity reasoning. this framework aims to spur the development of models focusing on quantitative reasoning in language understanding. chen and his colleague investigates whether neural network models can acquire numeracy skills, focusing on predicting numeral magnitudes in text @xcite . introducing the numeracy-600k benchmark dataset, the study explores various models. additionally, they highlights a practical application scenario by demonstrating the task's utility in detecting exaggerated information. chen also addresses innumeracy issues in pretrained language models, focusing on the fundamental task of teaching language models to understand numerals in text @xcite . it suggests a method that combines a comparing-number task with number notation exploration, modification, and pre-finetuning. their research shows enhanced performance in three benchmark datasets for tasks related to quantitative analysis, especially for roberta. the fish glided with a speed of 4 mph through the water and 1 mph through the jello because the _____ is smoother.? jello water option 2 type_3 ## dataset semeval-2024 utilizing current benchmark datasets for three different task types-quantitative prediction (qp), quantitative natural language inference (qnli), and quantitative question answering (qqa)-is the one of the task of numeval task 1: quantitative understanding @xcite . managing numbers, forecasting numerical values, deciphering logical connections in numerical sentences, and responding to inquiries requiring numerical data are all part of these tasks. the goal is to assess and improve the performance of models in handling these quantitative tasks. the provided data format appears to represent a set of questions along with options, correct answers, and additional attributes 2 , 3 . in summary, this data format is designed to offer various question formats, including options and the right response, as well as various ways to represent numerical values (scientific notation, characters, masked placeholders, etc.). it appears to be intended as a test of numerical information interpretation and comprehension in various formats. ## system description we explored the realms of natural language processing, immersing ourselves in hugging face's dynamic ecosystem known for its transformative libraries such as transformers and datasets. at the heart of our machine learning endeavor was the fine-tuning of a bert model for nuanced multiplechoice question answering, including numerical complexities. hugging face's berttokenizer and tfbertformultiplechoice powered our training, and the fine-tuned model effortlessly transitioned into competent inference on the test dataset foot_2 . we started our data preparation process by adding "context" and "label" to json files. next, we transformed the data into dataset objects that were kept in a datasetdict. we managed a pre-process_function, used berttokenizer, and used the datasets map method with '@xmath0. performance was improved by enabling dynamic sentence padding during collation using the data collator for multiple choice modification. figure 1 shows the system description of hijli_ju for the participation in semeval-2024 task 7. bert, or transformers' bidirectional encoder representations, which had been pre-trained using masked language modeling and next sentence predictions on a substantial amount of unlabeled text data. our approach was based on its bidirectional capabilities, which enabled it to simultaneously capture semantic subtleties from both sides. ## experimental setup we set up the parameters for training with a batch size of 16 over ten epochs, a starting learning rate of 0.00001, and no warm-up phases. our datasets' dimensions, comprising 564 examples for training, 81 for development, and 162 for testing, demonstrated accuracy. the purpose of this rigorous training setup was to guarantee our qqa model's generalizability and robustness. using jupyter notebooks, google colab is a cloud-based platform that offers an interactive and collaborative environment for python coding. wellknown for providing free access to gpu and tpu resources, colab has grown to be a well-liked option in the data science and machine learning domains. its smooth integration with google drive makes sharing and collaborative editing simple, which improves the effectiveness of team projects. because of its intuitive interface and free availability of robust computational resources, google colab is an indispensable resource for individuals and groups working on a variety of computational tasks. ## results in our quest to evaluate the efficacy of our quantitative question answering (qqa) model, we employed a comprehensive set of metrics and examined its performance across various question formats. the scikit-learn library's f1_score() function served as our tool for this evaluation, offering insights into the model's proficiency in different contexts. here, tp represents the number of true positives, fp represents the number of false positives, and fn represents the number of false negatives. the f1 score is a fundamental metric in machine learning, providing a balanced evaluation of classification models by combining precision and recall. this versatile metric has several variants, each suited to different scenarios. in this essay, we delve into macro-f1, micro-f1, weighted-f1, average, and binary-f1, exploring their applications and significance. ## observations the observed results highlight the nuanced performance of the quantitative question answering (qqa) model across different question formats. notably, questions presented in the character format consistently outperform other representations, demonstrating its robustness in handling diverse classes independently, particularly in imbalanced datasets. the macro-f1, micro-f1, and weighted-f1 scores consistently identify the question_char format as the most effective in achieving a balanced evaluation. this format excels not only in independently handling varied classes but also in proportionally contributing to overall performance based on class instances. the average f1 scores further affirm the versatility of the question_char format, emphasizing its capacity for a well-rounded ## conclusion and future work in conclusion, we found that quantitative question answering (qqa) is like a helpful tool for understanding numbers better. it's useful in important areas like business, healthcare, and finance, helping with predicting trends and making smart decisions. qqa is like a guide that empowers organizations and people to understand and use tricky data. the research we did shows how important qqa is for understanding numbers better. looking ahead, there are exciting possibilities for more research on qqa. we could explore new tasks and find ways to use qqa in specific areas like healthcare. working with experts in different fields could help make qqa more useful in different situations. also, we can improve how we measure the success of qqa and make it better by using the latest technology and techniques in language understanding. this ongoing exploration will keep pushing qqa to new places and make it even more important in understanding both language and numbers.