Infrrd.ai at S em E val-2024 Task 7: RAG -based end-to-end training to generate headlines and numbers (2024)

Abstract: We propose a training algorithm based on retrieval-augmented generation (RAG) to obtain the most similar training samples. The training samples obtained are used as a reference to perform contextual learning-based fine-tuning of large language models (LLMs). We use the proposed method to generate headlines and extract numerical values from unstructured text. Models are made aware of the presence of numbers in the unstructured text with extended markup language (XML) tags specifically designed to capture the numbers. The headlines of unstructured text are preprocessed to wrap the number and then presented to the model. A number of mathematical operations are also passed as references to cover the chain-of-thought (COT) approach. Therefore, the model can calculate the final value passed to a mathematical operation. We perform the validation of numbers as a post-processing step to verify whether the numerical value calculated by the model is correct or not. The automatic validation of numbers in the generated headline helped the model achieve the best results in human evaluation among the methods involved.

## introduction in our busy lives, we barely have time to read the newspapers or an online article. even a short period will not be enough to read all the latest news articles from different sources. the headline attached to the article attracts the reader only if it is interesting or provokes interest in the reader. a reader may have different interests and may not cover all areas. some may be interested in movies, politics, science and technology, economy, environment, governance, sports, celebrities, weather, etc. the information fed to a reader is large and must be condensed and remembered. a unique headline condenses the unstructured text into a few words with some numbers. the numbers are presented to attract the reader's attention from the part of the unstructured text. these numbers can be based on positive or negative sentiments. a negative sentiment has a greater influence than a positive sentiment. a positive sentiment aims to create new information in the reader's mind. however, a negative sentiment harms the reader's mind by correcting and updating the information. named entities, such as name, location, and number, are easier to remember for a long time than the rest of the text. the named entities are used more often by many people in several contexts with high-frequency usage. the narration in the text is constructed with the entities with a relationship. the occurrence of numbers in the relationship is more frequent than in other entities, especially in news articles. we perform text summarization on unstructured text to obtain specific and highlighted information. it is helpful in many areas and reduces the time spent on unnecessary or irrelevant texts. several events would occur in the process from the beginning to the end. all events may not be relevant or may appear as information overload. to reduce the list of events, we are using a text summary. medical report summary, annual report summary, election results summary, movie reviews, product reviews, and sports reviews will highlight the main results of the research or the results of the conducted activities. a person must read the content to prepare a summary of the text. the likelihood is that many relevant points of the content should be included as part of the summary. each person can prepare different lists of points using their previous knowledge and preferences. the main points of the different lists will be a central part of any summary. the core part can form a headline to attract readers to read the contents of the unstructured text. summary generation is a time-consuming process in which many people must contribute to preparing the highlights of the text content. in natural language processing (nlp), a model has to process text content sequentially using a seq2seq model and generate these highlights. it reduces the time required to generate highlights, increases knowledge aggregation, and filters interesting content. a vast literature of text summarization tasks in nlp shows the required attributes of a machine. a summary is presented in plain text in most cases and may not always contain a number. in the generation of headlines, we need a few numbers to highlight the content. the numbers in a headline play an important role in attracting readers' attention. here, the model must process the unstructured text sequentially and locate the numbers. all numbers in the text cannot be part of the headline. only a few numbers can cover the complete information from the unstructured text. a model has to identify the numbers that cover the news content in order to generate the headline @xcite @xcite . the main contributions of the proposed method are as follows: ## related work large language models (llms) have started to demonstrate reasoning, calculation, knowledge acquisition, planning, and many more (llama2; mistral; openai). at this point, we need to explore the potential capabilities of llms by proposing a wide range of problems that deal with a kind of artificial intelligence embedded in the model. in this paper, we study the numerical ability of a model. equate benchmark was prepared to make quantitative reasoning on different measures in the natural language inference (nli) @xcite . the data set is prepared to understand whether a model can reason on the text. the results show that the models have the ability to reason and obtain an inference for the statements. the scale of the predicted number was done to understand whether a model can find the magnitude of the predicted numbers through an nlp @xcite . a language model does not explicitly distinguish numbers from words @xcite . the notation of a number cannot be clearly understood by the model. this can be due to a missing number in the training data. we cannot provide all numbers to the model by any means, so we need to use different symbols to express the numbers in the text so that the model can use the numbers to perform the reasoning tasks. however, there was scope to add additional challenges to the data set. the numglue was proposed to identify the performance of llms through natural language understanding (nlu) @xcite . there are eight different tasks based on common sense reasoning, arithmetic calculation, quantitative prediction, fill-in-the-blanks, and arithmetic word problems. natural language optimization (nlopt) is a competition to solve arithmetic word problems for linear programming problems @xcite . we proposed an ensemble approach to detect the named entities (nes) in an unstructured text @xcite . the solution was generic and detected most of the entities in the text. of all these tasks, the headline generation or summarization focused on numbers was missing. the numhg data set was prepared to cover the headline generation task by focusing on numbers @xcite @xcite . the numeval competition is held to evaluate different models that can understand the numbers and generate the headline according to the ground truth specified (numeval). the model can choose any random number from the text, which may not be relevant in many cases. the model is pushed to perform the calculation that provides the fill-in-theblank task, where the model has to calculate the missing number from the headline or summary. a model must use mathematical operations to get the answer. the computational ability of the model is explored in this approach and is known as the task of 'numerical reasoning.' several mathematical operations can be performed using a model. however, the news data set mainly covers the reproduction of the number, the conversion of a word into a number, and the rounding of the number. the distribution of mathematical operations is very narrow. we do not have sufficient samples for other types of mathematical operations, and the model may not be well suited for these types of operations even after fine-tuning. the data set is designed in such a way ## proposed method a given llm may not have full knowledge of the generation of headlines using a text. we need to provide some support to the model to excel in the task of headline generation. we propose a retrievalaugmented generation (rag) model that is trained from end to end. the system consists of three modules: knowledge base, dense retrieval, and generation modules. figure 1 shows the block diagram of the proposed approach. the knowledge base (kb) was built using the training data provided. a pre-trained bert base model is used to encode training news samples into vectors @xcite . then, these vec-tors are indexed using faiss to enable the task of searching for vector-based similarity @xcite . the same bert-based model is used during training and inference time. a given input text is encoded in a vector. encoded vectors are used to search for dense vector similarity. top-k similar news articles are retrieved by a dense retrieval module. each result obtained will be added to the original input in a specific template to generate the headline. the purpose is to provide similar examples to generative models to help generate a better result. the prompt used in the experiment is shown in figure 1 . we use the llama2-7b model to generate the output headline @xcite . we use the rag token model (rag) on the selected top-k retrieved examples as shown in figure 1 . each retrieved news is sent along with the input news to obtain two separate token predictions from the llama2-7b model. these token predictions are marginalized, and the process is repeated until all tokens are generated. ## experiments llms have shown the ability to perform well in unknown tasks even without training. therefore, it is useful to check whether the model is well suited for the numeval dataset. ## conclusion we proposed a rag-based fine-tuning of llms to generate headlines and numerical values through reasoning. the model was trained from end to end to optimize the output of results. the model was trained for 3 epochs for headline generation and 1 epoch for numerical reasoning. we would like to train the model for a longer number of epochs in the future to confirm whether the model can improve performance. the verification step used to validate the generated numbers by the model is very useful to improve the confidence of the generated headline. there may be several numbers in the news, but the extraction and verification of numbers that can contribute to headline generation is a more concentrated approach. the additional verification stage helped the human evaluator select our proposed methodology as the most efficient among the competitors. we would like to explore further the rationale steps followed by the model in cot and improve the model performance by taking advantage of mathematical operations. we would like to divide operations into a subset of operations. a combination of results from multiple models is attempted without fully exploring the ensemble approach. we would like to explore the possibility of combining models at different levels such as input, architecture, and so on. elective surgeries. donations are at their lowest in 15 years. in june, the organization came up 50,000 units short. july isn't looking good, eithera particular problem given the increased number of accidents typical in the summer, when people travel, a rep tells nbc news. storms in the eastern and midwestern us both increased demand and cut supply, as the red cross was forced to cancel drives. with students, who account for 20% of donations, donating far less in the summer, the problem is compounded. we normally try to keep a three-day supply on hand locally, and we are down to a one-day supply, warns an ohio red cross worker. and the need never, ever goes away, notes another representative. pred_seq: red cross faces worst blood shortage in 15 <@xmath0: red cross faces worst blood shortage in 15 years the method searches for any xml tag in the generated sequence. here, the verification has found a tag. the method searches for the number '15' in the text because it is copied directly from the news section. the number '15' is found by the verification stage and removes the xml tag present in the generated headline. a.2 examples with errors in the verification stage a.2.1 example 1 news: (dec 20, 2016 5:40 pm) forty-three days after the election, all the votes have finally been tallied and certified. history will show hillary clinton beating donald trump by a final count of nearly 3 million votes, the hill reports. according to a tweet tuesday from the nonpartisan cook report, clinton received 65,844,610 votes (48.2%) to trump's 62,979,636 (46.1%). however with the electoral college officially making trump the 45th us president on monday, history will also show clinton as the second democrat in the past five elections to win the popular vote but lose the presidency. meanwhile, the huffington post reports trump had the third worst popular-vote performance by a winning candidate on record. pred_seq: 43 <@xmath1,1) </@xmath2, all votes have been counted issues: wrong subtracted value ('43', '<@xmath3,1) </@xmath4, 42), value 43 not found in snippet selected_headline: history will show clinton won popular vote by 3m votes the method searches for any xml tag in the generated sequence. here, the verification has found a tag. when we search for the number '43', it is not found in the news section. we discard the generated headline and select the headline generated by the brio model as the generated headline.