Shironaam: B engali News Headline Generation using Auxiliary Information (2023)

Abstract: Automatic headline generation systems have the potential to assist editors in finding interesting headlines to attract visitors or readers. However, the performance of headline generation systems remains challenging due to the unavailability of sufficient parallel data for low-resource languages like Bengali and the lack of ideal approaches to develop a system for headline generation using pre-trained language models, especially for long news articles. To address these challenges, we present Shironaam, a large-scale dataset in Bengali containing over 240K news article-headline pairings with auxiliary data such as image captions, topic words, and category information. Unlike other headline generation models, this paper uses this auxiliary information to better model this task. Furthermore, we utilize the contextualized language models to design encoder-decoder model for Bengali news headline generation and follow a simple yet cost-effective coarse-to-fine approach using topic-words to retrieve important sentences considering the fixed length requirement of the pre-trained language models. Finally, we conduct extensive experiments on our dataset containing news articles of 13 different categories to demonstrate the effectiveness of incorporating auxiliary information and evaluate our system on a wide range of metrics. The experimental results demonstrate that our methods bring significant improvements (i.e., 3 to 10 percentage points across all evaluation metrics) over the baselines. Also to illustrate the utility and robustness, we report experimental results in few-shot and non-few-shot settings.

## introduction news headlines can significantly affect the number of visitors and play a crucial part in the life-cycle of a news article @xcite . therefore, representative and interesting headlines are arguably essential to any news document to grab the attention of potential readers @xcite . nowadays, online and printed news releases significantly increase the article's visibility, support, and context by using multimedia content. as a picture is worth a thousand words, digital assets such as images and videos are the goto candidates for the thumbnails used in different social media, blogs, and many other platforms. the captions that go with the images or videos are equally significant as the actual content. captions describing the images can clarify and enhance the image, optimize news articles for search engines, and improve the accessibility of the news for people with vision impairments foot_0 . headline generation, given a news article, is a special case of abstractive summarization @xcite , which involves sentence compression, syntactic reorganization, sentence fusion, and lexical paraphrasing @xcite . unlike text summaries, which often feature many or single long sentences to summarize a document's important concepts @xcite , news headlines frequently have a single short catchy statement to grab the readers attention and entice them to read the story. even though bengali is the seventh most spoken language with approximately 337 million speakers worldwide foot_1 @xcite , generating quality headlines for a low-resource language such as bengali is more challenging due to the unavailability of large-scale human-annotated dataset @xcite . contextualized language models such as bert @xcite , roberta @xcite , t5 @xcite help improving several downstream tasks in nlp, such as summarization, question answering, and text classification. unfortunately, these models suffer from a limitation as they can handle input sequences up to a certain limit @xcite . as a result, this limitation burdens some nlp tasks, especially where the input is necessarily long @xcite , such as transcript analysis of the phone calls, document topic prediction, news headline generation, etc. the most natural way to address this problem is to trim the input sequences to a maximum length. however, trimming the long input document is tricky, especially for headline generation. the news articles usually maintain coherence and relevant parts may be located at the bottom of the document, which may prevent models from generalizing well to positions beyond the cutoff point 4 . in this paper, we utilize topic words to retrieve important sentences as a context for the bert model by following a simple yet cost-effective coarse-tofine approach. we present shironaam, a large-scale abstractive bengali news article dataset that includes over 240k professionally annotated headline-article pairings as well as auxiliary information such as image captions, topic words, and category information. each sample can be represented as a tuple of (article, image caption, topic-words, category, headline). to the best of our knowledge, shironaam is the first bengali news article dataset incorporating auxiliary information and a benchmark for the news headline generation task. this corpus has the potential to authorize and encourage research on such a low-resource language, bringing technological advancements to a previously underserved community. rather than the one-to-one mapping (i.e., input is an article, and output is a headline) used in the earlier works @xcite , we treat the headline generation task as a three-to-one mapping with the inputs being an image caption, a list of topic words, and an article where the output is a headline. based on the transformer architecture, we utilize pre-trained language models for generating headlines and present a new concept of fusing image caption parallelly @xcite with the input article to support the three-to-one mapping and to encode long documents. we design and compare numerous input mechanism alternatives as part of the suggested strategy. extensive experiments on our proposed dataset reveal that the suggested method is capable of generating high-quality news headlines (see section f in the appendix) and brings significant improvements over the state-of-the-art baselines across all evaluation metrics ( our main contributions can be summarized as follows: we train an encoder-decoder model for this task almost from scratch, which utilizes pretrained language model (figure 1 ). ## the shironaam corpus in this section, we present the first-of-its-kind corpus (we name it shironaam) for news headline generation in bengali like low-resource language. this includes auxiliary information in addition to the usual headline-article pairs. we explain the curation process involving raw data crawling, preprocessing, and cleaning. ## experiments and benchmarks in this section, we set a new benchmark for bengali news headline generation using shironaam corpus and compare it with the other state-of-theart baselines. after a clean comparison, we perform two ablation experiments on the superior base model. then, we analyze the performance gap between the baselines and the ablations and further evaluate the best model on news domains with a few samples (few-shot). finally, after proper analysis, we seek to find out the answers to the following research questions: ## related works headline generation is an under-explored subtask of abstractive summarization, particularly in languages with limited resources. for the english language, an attention-based neural network has been proposed by rush et al. (2015) for abstractive sentence summarization. the authors propose a model that utilizes a recurrent neural network (rnn) and an attention mechanism to summarize input sentences into a compact summary. takase et al. (2016) build an amr encoder for headline creation based on an encoder-decoder architecture. using a dual-attention seq2seq model, zhang et al. (2018) proposes a way for question headline development. in limited resource settings, tilk and alumäe (2017) pretrain a neural encoder and de-coder model to enhance headline generation outputs. a sentence encoder, a gate network for sentence selection, and a headline decoder are the three stages of zhou et al. (2017)'s headline generation approach. tan et al. (2017) proposes a coarse-to-fine strategy that extracts the most important sentences before generating the headlines based on the context. for headline generation, kumar et al. (2022) have released the indicnlg, a collection of multilingual datasets. however, they do not provide any additional attributes besides the headline-article pairs. in summary, the majority of the past works for generating headlines primarily used the article content to generate headlines. ## conclusion and future work in this paper, we contributed a large-scale dataset (a.k.a., shironaam) with auxiliary information such as image captions, topic words, and category for bengali news headline generation. we employ contextualized language models to incorporate such auxiliary information and proposed a simple yet effective solution to encode long articles using topic words. experimental results demonstrate the superiority of our approach across different domains and settings. we anticipate that our efforts will motivate the community to expand the scope of headline generation tasks beyond english, particularly for a low-resource language like bengali. our future work will look into incorporating auxiliary information to support more languages and build a language-agnostic model.