Challenges at S em E val 2024 Task 7: Contrastive Learning Approach on Numeral-Aware Language Generation (2024)

Abstract: Although Large Language Model (LLM) excels on generating headline on ROUGE evaluation, it still fails to reason number and generate news article headline with accurate number. Attending SemEval-2024 Task 7 subtask 3, our team aims on using contrastive loss to increase the understanding of the number from their different expression, and knows to identify between different number and its respective expression. This system description paper uses T5 and BART as the baseline model, comparing its result with and without the constrative loss. The result shows that BART with contrastive loss have excelled all the models, and its performance on the number accuracy has the highest performance among all.

## introduction this paper is a description of the methods we have applied for our implementation on this year's se-meval task 7, numeval: numeral-aware language understanding and generation. semeval is an annual workshop which is consisted of various natural language processing shared tasks. teams that join the tasks is required to design systems that could enhance the understanding or improve results on various kinds of semantic evaluation challenge. the task we decide to join was task 7, nu-meval: numeral-aware language understanding and generation @xcite . specifically, we focus on the second subtask from the third task in numeval, which centers on generating proper news headline based on the provided news articles. different from article summary, a headline must condense the essence from the full length article. although the encoder-decoder language models nowadays has excelled on generation of the text based on the rouge metric, it still fails on providing precise numeral generation in headlines owing to the fact that the representation of the number may differ in various kinds of forms. therefore, the goal of this task is to enhance the accuracy of the model in the generation of the number from the headline of a news article. while the numeral expression in the article consists of text and numbers, our system aims to use the technique of contrastive learning. with this technique, it is possible to help the model enclose the similarity between the number and its text expression, and enlarge the difference between different numbers (and its respective expression). the evaluation of the performance is divided into two parts, one is to evaluate the accuracy of the predicted number, the other is to evaluate the accuracy of the word prediction, which uses rouge, bertscore, and moverscore to evaluate the results. two datasets are included in the task. one is the dry-run dataset that is provided on the official site. it contains a total of 100 instances; the other is the official training set that is provided after the registration of the task, which includes a total of 21157 instances. all of them have the same data structure. each instance contains both the news article that includes the date of release, and its respective headline. each team is expected to generate a precise news title from its respective news article. numbers are one of the most important element among medical, business, and legal article, and it could be dangerous if the large language model has misunderstood the content of the article. the finding of this paper could further discover a better way for large language model to detect and reason the numbers from the respective article, and thus generate the accurate headline with correct number. our system description is divided into three main section. first, we present the main approach that we used as the final submission result in detail, including the preparation of the dataset, its augmentation, and the structure of the model in contrastive learning that we designed in order to solve the task. hence, we talk about the adjustment of the model, including parameter optimization on the model. finally, we present our experimental result. in the end, we discuss about the possible future work, and conclude with a brief summary of our system. ## related work large language model (llm) like chatgpt has been long commented with its brittleness on the ability of numerical reasoning. when the questions presented in the varying textual form (comprising words and numbers), llm would result in inconsistent performance @xcite . in @xcite 's work, it shows that although large language model excelled based on rouge metrics, it still fails to generate precise numeral in headline. researchers have applied contrastive learning in natural language processing. to generate headline with different author style, @xcite has applied contrastive learning to integrate the stylistic feature of the author into the model. this research hence inspires us to use contrastive learning on integrating the numeral features in different text form to the model, in order to let model identify the correct number from the news article. ## methodology in this section, we propose the methodology of our system. it mainly consists of three parts. first, we will talk about the augmentation of the data. secondly, the model training and fine-tuning, and finally, parameter optimization. ## experimental results