XL - H ead T ags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tags (2024)

Abstract: Millions of news articles published online daily can overwhelm readers. Headlines and entity (topic) tags are essential for guiding readers to decide if the content is worth their time. While headline generation has been extensively studied, tag generation remains largely unexplored, yet it offers readers better access to topics of interest. The need for conciseness in capturing readers’ attention necessitates improved content selection strategies for identifying salient and relevant segments within lengthy articles, thereby guiding language models effectively. To address this, we propose to leverage auxiliary information such as images and captions embedded in the articles to retrieve relevant sentences and utilize instruction tuning with variations to generate both headlines and tags for news articles in a multilingual context. To make use of the auxiliary information, we have compiled a dataset named XL-HeadTags, which includes 20 languages across 6 diverse language families. Through extensive evaluation, we demonstrate the effectiveness of our plug-and-play multimodal-multilingual retrievers for both tasks. Additionally, we have developed a suite of tools for processing and evaluating multilingual texts, significantly contributing to the research community by enabling more accurate and efficient analysis across languages.

## introduction the headline serves as a concise and attentiongrabbing summary of a news article. articles with compelling headlines are more likely to attract increased views or shares @xcite . unlike summaries, which provide a broad overview @xcite , headlines aim to produce brief and engaging statements that draw figure 1 : our content selection approach. auxiliary information, including images and captions, is used as queries to extract salient and relevant sentences from documents via two modules: imgret for images (visual modality) and capret for image captions (textual modality). our modules are designed as plug-and-play components that can be integrated with language models of any size and type. another important feature of a news article are topic tags or "semantic markers", which serve as dynamic connectors and navigational aids, enhancing the coherence and accessibility of information across articles. the task of tag generation is related to keyphrase generation @xcite . while keyphrases summarize the main themes succinctly of an article, tags provide a broader overview, guiding readers to related articles and facilitating navigation through connected themes. despite their significant role, the generation of news article tags has remained unexplored in existing literature. in this work, we introduce a unified framework for generating headlines and tags for news articles in a multimodal-multilingual context, covering 20 languages across 6 diverse language families. despite the emergence of open-source large language models (llms) like llama @xcite and @xcite , multilingual models such as mt5 @xcite and mt0 @xcite , fine-tuned with task-specific data, remain the preferred solution for multilingual tasks, especially for low-resource languages @xcite @xcite . news articles often include extensive content, incorporating additional details, author quotes, historical context, and advertisements, among others. this poses a challenge in identifying article segments that are both salient and relevant for the creation of highly concise outputs like headlines and tags. this complication leads to what is known as the "lost-in-the-context" problem @xcite , wherein vital information embedded within lengthy documents is frequently overlooked by the models @xcite . fortunately, it is now very common for both online and printed news media to use multimedia content to enhance visibility, support, and context for articles @xcite . digital assets, like images, often serve as thumbnails across social media, blogs, and various platforms. equally important are the captions accompanying these images, which not only clarify and enrich the image but also optimize articles for search engines and make news more accessible to those with vision impairments @xcite . this has motivated us to utilize auxiliary information (images and captions as illustrated in figure 1 ) to distill salient and localized information from news articles in a multimodal-multilingual context using the clip-vit-b32 model @xcite . this model aligns text and images within a unified dense vector space. our method is inspired by the cognitive load theory @xcite , reflecting how humans employ visual cues and summaries to understand the essence of lengthy texts without being overwhelmed by information. building on this, with the contents selected using our multimodal retrievers, we utilize instruction tuning to generate both headlines and tags for news articles in a multilingual context. our contributions are summarized as follows: ## multiragen retrieval-augmented generation (rag) @xcite involves two key phases: first, retrieving contextually relevant information, and sec-ond, using this information to guide the generation process @xcite . rag has been applied to various tasks such as machine translation @xcite , dialogue generation @xcite , and abstractive summarization @xcite . inspired by these applications, we leverage multimodal information like images and captions to select salient content from news articles, introducing multiragen (multimodal retrieval augmented generation). multiragen comprises two main components: (1) multimodal retrievers ( §3.1) and (2) instruction tuning ( §3.2). ## multilingual tools we also develop tools by accumulating open-source resources for processing and evaluating multilingual texts, making it an easy-to-use one-stop destination. these tools include (1) multilingual rouge scorer, (2) multilingual sentence tokenizer, and (3) multilingual stemmer. these resources are invaluable to the research community focusing on multilingual nlp, providing essential support for accurate processing and evaluation 5 (also detailed in appendix c). multilingual sentence tokenizer sentence tokenization aims to divide a given document into individual sentences. to achieve this, we integrated various open-source resources for multiple languages into a unified library. this tool can perform tokenization in 41 different languages (details of the sources in appendix ( ## evaluating tags generation the terms "tag words" and "keyphrases," while sharing a common objective, differ in their application. in the keyphrase generation process, a model predicts a set of distinct keyphrases ŷ = (ŷ 1 , . . . , ŷm ) from a given source text, with these predictions ŷi being ordered based on their relevance @xcite . the ground truth keyphrases for the source text are denoted as y. it's important to note that in the context of tag words, the order of predictions does not necessarily reflect the quality of each prediction. to measure predictive performance, three standard evaluation metrics-namely macro-averaged precision, recall, and f-measure (f 1 )-are commonly used @xcite . formally, the metrics of precision, recall, and f 1 score are defined as follows: ## experiments 6.1 data & evaluation metrics data to ensure a balanced distribution, our dataset division for training, validation, and test sets across all languages consists of 95% (394,353 samples), 1% (5,187 samples), and 4% (15,577 samples) respectively from our xl-headtags dataset (detailed in appendix ( evaluation metrics for the evaluation of generated headlines, we utilize f1 score of our multilingual rouge scorer ( §4) and the bleu score @xcite , f1 bert score @xcite , meteor score @xcite , and length ratio (lr). for assessing tag words, we apply the metrics we have proposed in section 5.1 and normalize the tag words using our multilingual stemmer ( §4). ## results and discussion the evaluation results for headline generation are detailed in ## conclusion in this paper, we compile the xl-headtags dataset for headline and tag generation tasks, including 20 languages across 6 diverse language families. we introduce a novel content selection approach that leverages auxiliary information from both textual and visual modalities to pinpoint the most salient content within news articles. we employ instruction tuning to generate both headlines and tags in controlled and unrestricted manners. furthermore, we have developed a suite of tools by accumulating open-source resources for processing and evaluating multilingual texts.