C luster C ore at S em E val-2024 Task 7: Few Shot Prompting With Large Language Models for Numeral-Aware Headline Generation (2024)

Abstract: The generation of headlines, a crucial aspect of abstractive summarization, aims to compress an entire article into a concise, single line of text despite the effectiveness of modern encoder-decoder models for text generation and summarization tasks. The encoder-decoder model commonly faces challenges in accurately generating numerical content within headlines. This study empirically explored LLMs for numeral-aware headline generation and proposed few-shot prompting with LLMs for numeral-aware headline generations. Experiments conducted on the NumHG dataset and NumEval-2024 test set suggest that fine-tuning LLMs on NumHG dataset enhances the performance of LLMs for numeral aware headline generation. Furthermore, few-shot prompting with LLMs surpassed the performance of fine-tuned LLMs for numeral-aware headline generation.

## introduction news articles are one of the most important sources of information in everyday life. news headlines are vital in selecting which news seems relevant to read. as delineated in studies @xcite , headlines play a significant role in making news viral on social media and influencing readers' opinions @xcite . inaccurate, incongruent or misinformation headlines also lead to the spread of misinformation and disinformation over digital platforms @xcite @xcite . consequently, generating an accurate headline for a news body is essential. therefore, ensuring the accuracy of headlines is essential for maintaining the credibility and usefulness of news publications. the task of headline generation, which is a form of text summarization, aims to condense a lengthy source text into a concise summary. this summary, typically presented as a headline, encapsulates the main points of the original text, providing readers with a quick overview of the content @xcite . in earlier studies on headline generation, various sequence-to-sequence and encoder-decoder methods have been employed to extract relevant headlines from news articles @xcite @xcite . however, encoder-decoder methods faced challenges in processing large sequences of text. to address these limitations, recent studies @xcite @xcite @xcite have proposed transformer-based models for headline generation by summarizing news articles. while transformer-based models have indeed showcased enhanced capabilities in handling longer text sequences and have exhibited promising outcomes in headline generation tasks; it is noteworthy that their performance in numeral-aware headline generation tasks need to be consistently superior. despite their overall advancements, transformer-based models may face challenges in accurately incorporating and representing numeric information within generated headlines. motivated by such observations, the study @xcite proposed numeral aware headline generation datasets. this paper introduces our proposed approach and provides a comprehensive analysis of the task of numeral-aware headline generation (task 3 (2)). our proposed methodology leverages few-shot prompting with llms, which involves applying few-shot learning techniques to large language models (llms) for numeral-aware headline generation tasks. we conduct our experiments using the numhg dataset @xcite and the test set provided by the organizer of numeval task-3(2). our experimental results suggest that fewshot prompting-based methods with llms are efficient for numeral-aware headline generation. ## proposed method as mentioned above, the paper aims to study the effect of two important aspects of numeral aware headline generations. first, we study the effectiveness of large language models (llms) for numeralaware headline generations. second, we propose a few prompting-based methods for numeral-aware headline generations. ## error analysis this study also conducts an error analysis to examine the strengths and weaknesses of large language models (llms) across different setups for numeralaware headline generation. through this analysis, we aim to identify patterns of errors and limitations inherent in the models, providing valuable insights into areas for improvement and optimization in future model development and training methodologies. we selected fifty news headline-body pairs, where numeral figures in the headline are crucial for accurately representing the news content. our examination of the generated headlines by the models revealed the following insights: (i) robert a the model generates a headline, which is representative of the news body, however in some instances is failed to consider the numeric value for headline generation. consequently, robert a is deemed unsuitable for numeral-aware headline generation. however, fine-tuning the robert a model enhances generated headline quality, which is also evident by the performance comparison between its pretrained and fine-tuned setups. (ii) the bart models, whether in the pretrained or fine-tuned setups, demonstrate proficiency in generating efficient headlines that include valid numeric values. however, it is noteworthy that the inclusion of valid numeric values in headlines is more prevalent in the fine-tuned models compared to those without finetuning. (iii) the t 5 models, in both the pretrained and fine-tuned setups, consistently produced headlines with more efficient and valid numerical values compared to robert a, bart , and gp t . this indicates that t 5 models are particularly more effective in numeral aware headline generations. (iv) the llam a model stands out for its ability to generate accurate and efficient headlines containing valid numerical values when compared to robert a, bart , gp t , and t 5. this suggests that the llam a model excels in incorporating precise numeric information into its generated headlines, surpassing other models in this aspect. figure 2 presents an example' comparison between headline generated by fine-tuned t 5 model and headline generated by t 5 with three shot prompt. from figure 2 , it is apparent that the headline generated by t 5 with three three-shot prompts better represents the news body compared to the headline generated by the fine-tuned t5 model. this further validates our claim that a few-shot prompt helps the llms generate headlines. ## conclusion and future work this study conducted an empirical research on llms for numeral-aware headline generation and proposed a few shots prompting with llms for numeral-aware headline generation. we conduct our experiments on numhg and test set data provided by the organizer of numeval-2024. our experimental results suggest that finetuning llms over numhg dataset improves the performance of numeral-aware headline generation. further, few shot prompting with llms outperform finetuned llms for numeral-aware headline generation. this study identifies prompt tuning using llms for numeral-aware headline generation.