Team NP _ PROBLEM at S em E val-2024 Task 7: Numerical Reasoning in Headline Generation with Preference Optimization (2024)

Abstract: While large language models (LLMs) exhibit impressive linguistic abilities, their numerical reasoning skills within real-world contexts re- main under-explored. This paper describes our participation in a headline-generation challenge by Numeval at Semeval 2024, which focused on numerical reasoning. Our system achieved an overall top numerical accuracy of 73.49% on the task. We explore the system’s design choices contributing to this result and analyze common error patterns. Our findings highlight the potential and ongoing challenges of integrat- ing numerical reasoning within large language model-based headline generation.

## introduction the capacity to understand and manipulate numerical information within natural language text is essential for various nlp applications. tasks such as news summarization, report generation, and the creation of data-driven narratives increasingly rely on the accurate interpretation and generation of numerical expressions. semeval 2024 task 7 @xcite addresses these challenges through two intriguing subtasks: numerical headline generation and numerical headline number fill-in-the-blanks. generating numerical headlines necessitates models capable of synthesizing a succinct and attention-grabbing title that accurately reflects a news article's core numerical quantities and trends. conversely, the fill-in-the-blanks subtask tests the model's ability to comprehend numerical relationships and infer the missing value to complete a provided headline. these tasks present a complex intersection of numerical reasoning and natural language generation/understanding. existing text generation and numerical understanding work often leverage sequence-to-sequence * equal contribution architectures and specialized pre-trained language models. however, semeval 2024 task 7's emphasis on numerical reasoning within headlines creates a distinct demand for techniques capable of accurately grounding representations of numbers and quantities within the linguistic context. this paper describes our approach to semeval 2024 task 7. we worked on both tasks separately and created two separate models. we used techniques such as parameter-efficient fine-tuning of large language models and then doing direct preference optimization on top to align models better. the remainder of this paper proceeds as follows. section 3 reviews related work in numerical reasoning and headline generation. section 4 details our models and methodology. section 4 presents our experimental evaluation of the semeval 2024 task 7 dataset and includes a thorough analysis of our results. finally, section 5 summarizes our findings and outlines potential future research directions. ## background and related work headline generation within nlp has a rich history, evolving from early extractive techniques towards modern abstractive generation methods. initial extractive approaches primarily focused on selecting the most salient sentences from the source document to compose the headline @xcite @xcite . these methods offered interpretability but lacked the fluency and novelty often desired in generated headlines. the advent of deep learning and sequence-to-sequence models enabled abstractive headline generation, empowering models to synthesize new phrases and expressions @xcite . attention mechanisms @xcite proved pivotal in aligning source text and headline generation. recent advancements in generative ai have led to significant improvements in this field with state-of-the-art results @xcite . gsum @xcite , for example, initially performs extractive summarization and then incorporates the extractive summaries into the input for abstractive summarization. season @xcite adopts a dual approach, learning to predict the informativeness of each sentence and using this predicted information to guide abstractive summarization notably, most of these works focus on the selection of words and the structure of sentences. ## numeval numeval is part of semeval 2024; the task we focused on and worked on requires models to generate concise and informative headlines that accurately reflect the core numerical information in news articles. systems must demonstrate an understanding of how numbers convey meaning and should prioritize the most relevant numerical aspects for inclusion in the headline. subtask 1: numerical reasoning -models are required to compute the correct number to fill the blank in a news headline. subtask 2: abstractive headline generationmodels must construct a headline based on the provided news; this headline should incorporate the numerical reasoning within. the organizers released it provides over 27,000 news articles with detailed annotations designed to facilitate the development of models that accurately understand and summarize numerical information. for subtask 1, each data point has an answer operator added, which signifies how the numerical answer is obtained, which includes copy (direct retrieval), trans, span, round, paraphrase, add, subtract, multiply, and divide. meanwhile, subtask 2 requires the model to generate complete headlines from given news content. ## results for subtask 1, we evaluate our model on two fill-inthe-blank types. one accuracy on copy, where the answer is directly copied from the article. second is reasoning, which includes different reasoning techniques such as addition, subtraction, multiplication, and paraphrasing. copy reasoning 0.922 0.784 for comparative semeval two-stage evaluations, we scored 0.89 in the open and 0.86 in the hidden stage, respectively. for subtask 2 we performed best in numerical accuracy overall and reasoning scores, we out-shined in reasoning accuracy as difference between 1st and 2nd rank was around 7 points. we open-source our final models on huggingface foot_0 . ## limitations since our method is based on 7b llm, our performance is capped by this model's ability to draw rationale for the numerical reasoning. this limitation is in line with the hardware resource as well; we used an rtx 4090 24gb gpu-based machine for our work, which can load and fine-tune the models with upto 7-10 b parameters as well.