hinoki at S em E val-2024 Task 7: Numeral-Aware Headline Generation ( E nglish) (2024)

Abstract: Numerical reasoning is challenging even for large pre-trained language models. We show that while T5 models are capable of generating relevant headlines with proper numerical values, they can also make mistakes in reading comprehension and miscalculate numerical values. To overcome these issues, we propose a two-step training process: first train models to read text and generate formal representations of calculations, then train models to read calculations and generate numerical values. On the SemEval 2024 Task 7 headline fill-in-the-blank task, our two-stage Flan-T5-based approach achieved 88% accuracy. On the headline generation task, our T5-based approach achieved RougeL of 0.390, BERT F1 Score of 0.453, and MoverScore of 0.587.

## introduction comprehension of numerical values can significantly enhance performance in certain tasks as numbers provide important information in words. numerical values are particularly important in accounting and finance fields as the majority of data is in monetary terms. while words can be ambiguous, numbers provide clear and precise information. they not only represent exact numerical values, but can also indicate a magnitude of the subject matter, which can be critical to fully understand a text. despite the significance of numerical values, much natural language processing work has treated numerical words in the same manner as all other words, without any direct understanding of the values they represent. as a result, numerical reasoning is still challenging for natural language processing models, even the pre-trained language models that have been so successful on other natural language processing tasks. numeval @xcite provides shared tasks that encourage research systems to generate headlines with accurate numeral information. we fine-tuned pre-trained models for two sub-tasks. in the first, models are required to compute the correct number to fill the blank in a news headline given the corresponding news article. in the second, models are required to construct an entire headline (including its numerical information) based on the provided news article. ## related work a math word problem (mwp) consists of a short natural language narrative describing a state of the world and poses a question about some unknown quantities @xcite . the mwp task is a type of semantic parsing task where given an mwp the goal is to generate an equation, which can then be evaluated to get the answer. the task is challenging because a machine needs to extract relevant information from natural language text as well as perform mathematical reasoning to solve it. @xcite proved in their paper that the existing models can rely on superficial patterns present in the narrative of the mwp and achieve high accuracy without even looking at the question. @xcite proposed a numerical machine reading comprehension model named num-net, which utilizes a numerically-aware graph neural network to make numerical comparison and performs numerical reasoning over numbers in the question and passage. their numnet model achieved some numerical reasoning ability with exact match (em) of 64.56 and numerically-focused f1 score of 67.97 on the test data. however, num-net is not applicable when an intermediate number has to be derived in the reasoning process such as from arithmetic operation. geva et al. (2020) proposed a general method for injecting additional skills into language models, assuming automatic data generation is possible. they applied their approach to the task of numerical reasoning over text, using a general-purpose model called genbert, and a simple framework for generating large amounts of synthetic examples. their experiments demonstrated the effectiveness of their method, showing that genbert successfully learns the numerical skills, and performs on par with similarly sized state-of-the-art numerical reasoning over text models. @xcite proposed arithmetic-based pre-training that combines contrastive learning to improve the number representation, and a novel inferable number pre-training objective to improve numeracy. their experiments showed performance improvements due to better numeracy in three different state-of-the-art pre-trained language models, bart, t5, and flan-t5, across various tasks and domains, including reading comprehension, inference-on-tables, and table-to-text generation. @xcite proposed a novel pre-trained model, namely mathbert, which is the first pretrained model for mathematical formula understanding. mathbert was jointly trained with mathematical formulas and their corresponding contexts to evaluate three downstreamtasks, including mathematical information retrieval, formula topic classification and formula headline generation. formula headline generation is a summarization task aiming to generate a concise math headline from a detailed math question which contains math formulas and descriptions. in addition, in order to further capture the semantic-level structural features of formulas, a new pre-training task is designed to predict the masked formula sub-structures extracted from the operator tree (opt), which is the semantic structural representation of formulas. ## conclusion t5 language models seem capable of generating meaningful headlines including appropriate numerical values. although the models can reasonably compute the correct numbers from the provided news to fill the blank in headlines, they sometimes failed reading comprehension and arithmetic operations. in hope of overcoming those limitations, we trained them to generate the calculation methods first and then trained again with those calculations as inputs to predict the numerical values to fill the blank in the news headlines, but it did not significantly improve the results. in the future, we plan to try larger pre-trained models, which might improve performance. also, the training datasets that we used are relatively small. if we increase the data size by data augmentation, we may be able to obtain better results.