S em E val-2024 Task 7: Numeral-Aware Language Understanding and Generation (2024)

Abstract: Numbers are frequently utilized in both our daily narratives and professional documents, such as clinical notes, scientific papers, financial documents, and legal court orders. The ability to understand and generate numbers is thus one of the essential aspects of evaluating large language models. In this vein, we propose a collection of datasets in SemEval-2024 Task 7 - NumEval. This collection encompasses several tasks focused on numeral-aware instances, including number prediction, natural language inference, question answering, reading comprehension, reasoning, and headline generation. This paper offers an overview of the dataset and presents the results of all subtasks in NumEval. Additionally, we contribute by summarizing participants’ methods and conducting an error analysis. To the best of our knowledge, NumEval represents one of the early tasks that perform peer evaluation in SemEval’s history. We will further share observations from this aspect and provide suggestions for future SemEval tasks.

## introduction in the past, semeval has predominantly focused on discussions surrounding words in text, with limited exploration of numbers in text. recognizing the significance of understanding numbers can enhance performance in certain tasks. for instance, there is a notable difference in the sentiment degree between "expecting the stock price to increase by 30%" and "expecting the stock price to increase by 3%" in fine-grained sentiment analysis, as the former suggests a higher sentiment degree than the latter (semeval-2017 task 5 @xcite ). similarly, "stealing @xmath0100,000" could result in differing court judgments (semeval-2023 task 6 @xcite ), and contrasting systolic blood pressure readings of 119 versus 121 offer different clinical inferences @xcite ). these examples underscore the importance of numerical understanding in text, suggesting it as a potential research direction for enhancing the performance of downstream tasks. recent interest has surged in the numeracy of textual data and models within the nlp community, marking an opportune moment to evaluate current models' performance in numeral-aware language understanding and generation. to this end, we propose a collection of five published datasets encompassing three tasks: quantitative understanding, reading comprehension of numerals in text, and numeral-aware headline generation. for quantitative understanding tasks, we utilize the quantitative 101 dataset @xcite . the nquad dataset @xcite serves to explore reading comprehension with numerically rich documents, and num-hg @xcite , annotated for numerical reasoning, facilitates the investigation of numeral-aware headline generation. in summary, while these are foundational nlp tasks, our focus is on discussing instances that require numeracy and the capacity to understand numbers for resolution. in this paper, we first provide an overview of the dataset and subsequently summarize the methods and performances of participants. the comparison of models and error analysis will be included. additionally, we employ peer evaluation to annotate and evaluate the generated outputs of participants' systems. our analysis and observations, based on the annotations from participants, will be shared. we hope this pilot trial can offer insights and share experiences for future studies planning to conduct human evaluations among different teams. to go one step further, num-hg extends the rc task in nquad. it provides numerical reasoning annotations to 27,746 news, and offers two subtasks, numerical reasoning and headline generation. the major goal of this task is to generate a headline that contains key numerical information in the news article. news: at least 30 gunmen burst into a drug rehabilitation center in a mexican border state capital and opened fire, killing 19 men and wounding four people, police said. gunmen also killed 16 people in another drug-plagued northern city. the killings in chihuahua city and in ciudad madero marked one of the bloodiest weeks ever in mexico and came just weeks after authorities discovered 55 bodies in an abandoned silver mine, presumably victims of the country's drug violence. more than 60 people have died in mass shootings at rehab clinics in a little less than two years. police have said two of mexico's six major drug cartels are exploiting the centers to recruit hit men and drug smugglers, ... headline (question): mexico gunmen kill ____ answer: 35 annotation: add(19,16) to calculate the correct number of the blank part in the news headline. in the headline generation subtask, models must generate a headline based on the given news. because each headline in the proposed num-hg contains one number, models are expected to generate the same number as journalists. our rationale is that the number selected by the journalists should be the most informative for summarizing the news article. therefore, we will evaluate whether the generated number is correct or not. additionally, we will further evaluate the generated headline by automatical metrics, such as rouge and bertscore, and manual evaluation. specifically, participants manually evaluate the system outputs from other teams. ## participants and automatic evaluation there are 124 teams registered for numeval, with 20 teams submitting their system description papers. this section provides an overview of the major methods employed in each paper, with detailed explorations available in the respective papers. as participants can select specific subtasks, results are reported in a fine-grained manner to encompass partial outcomes. sengupta et al. (2024) emphasize the significance of number representation in character format. kalantari et al. (2024) employ orca2 (mitra et al., 2023) with fine-tuning and chain-of-thought (cot) prompting (wei et al., 2022), achieving high performance across most tasks. veerendranath et al. (2024) introduce the pre-calc approach, which incorporates operation classification tasks during roberta training and utilizes this knowledge to decide on calculator usage for results, highlighting the value of tool utilization. saha (2024) experiment with bert, while liang et al. (2024) leverage the abe-7b model enhanced by human feedback during training, surpassing several large language models (llms). in summary, findings from quantitative 101 suggest that learning calculator usage and character-format number representation can aid in quantitative tasks. additionally, employing tailored language models like orca2 or integrating human feedback can further enhance performance. ## conclusion in this paper, we explored the complexities of numerical understanding and generation in text, an area that has garnered increasing interest within the nlp community. by introducing and evaluating a set of tasks across diverse datasets, our work highlighted significant progress towards enhancing models' numerical comprehension and their application in practical scenarios, including quantitative analysis and numeral-aware headline generation. our comprehensive evaluation, encompassing both automatic and human assessments, demonstrated the capabilities and limitations of current methodologies, emphasizing the sophisticated understanding necessary to effectively manipulate and interpret numerical information in textual formats. as we approach the mastery of simple numerical questions with the appropriate selection of language models, our research indicates a shift towards more intricate and nuanced challenges in numerical nlp. the advancements facilitated by numeval set the stage for future investigations into the deeper integration of numeracy and language, aiming not only for models that comprehend numbers but also for those capable of reasoning, inferring, and generating text that accurately reflects the quantitative dimensions of the world.