PMI ndia S um: Multilingual and Cross-lingual Headline Summarization for Languages in I ndia (2023)

Abstract: This paper introduces PMIndiaSum, a multilingual and massively parallel summarization corpus focused on languages in India. Our corpus provides a training and testing ground for four language families, 14 languages, and the largest to date with 196 language pairs. We detail our construction workflow including data acquisition, processing, and quality assurance. Furthermore, we publish benchmarks for monolingual, cross-lingual, and multilingual summarization by fine-tuning, prompting, as well as translate-and-summarize. Experimental results confirm the crucial role of our data in aiding summarization between Indian languages. Our dataset is publicly available and can be freely modified and re-distributed.

## introduction the era of deep learning has witnessed great advancements in various natural language processing (nlp) tasks. yet prevalent solutions usually rely on large datasets, which are limited to high-resource languages. this is particularly pronounced in india, where languages spoken by a large population have been historically overlooked in research and under-resourced @xcite . in text summarization, where a system generates a brief description of a longer text, the availability of datasets for indian languages is restricted in terms of both language coverage and size @xcite . 2 moreover, some existing datasets are not easily accessible or have been criticized for their quality @xcite . given the multilingual nature of india, having 122 major languages and 22 official ones, the development of cross-lingual summarization is desirable for information access, however, it is challenging due to the absence of reliable resources. to address these challenges, we present pmindi-asum, a massively multilingual and cross-lingual summarization dataset sourced from the prime minister of india website. 3 this website publishes political news, usually available in various languages covering the same content. in accordance with established methods @xcite , we use a news article as the document and its headline as the summary. beyond monolingual data, as figure 1 depicts, the website's multilingualism enables cross-lingual document-summary alignment with a firm confidence in quality. our efforts lead to an extensive coverage of 196 language directions from 14 languages across four families, making the corpus the current widest collection of indian language pairs for summarization. there are 76,680 monolingual document-headline pairs and 620,336 cross-lingual pairs in total. we display language family and code information in though there exist summarization datasets supporting languages in india, some concerns need to be addressed with regard to developing datasets for indian language summarization. explicitly, data construction should (1) respect copyright permissions and ensure transparency in data processing; ## analysis and discussions 4.1 error analysis to assess and interpret errors made by different models in different language settings, we conducted an error analysis by comparing system outputs with references. we establish error categories and invite human annotators to label the errors in outputs, with setup and explanations of error categories detailed in appendix h. we randomly sample 50 test outputs from all models that deal with monolingual en, hi, te, as well as en↔hi cross-lingual summarization. annotation outcome is presented in in monolingual cases, the prevalent error is omission for both indicbart and mbart models. conversely, we have different discoveries in crosslingual models. dedicated cross-lingual models for a single language pair usually make factuality and omission mistakes. regarding multilingual systems, indicbart suffers from language mismatch badly, whereas mbart is associated with factuality, omission and redundancy. approximately 53% monolingual summaries and 30% cross-lingual summaries are considered to be correct. although decent automatic scores were seen previously, manual inspection suggests that there is ample room for model improvement on our dataset. ## conclusion and future work we have presented pmindiasum, a headline summarization corpus focused on 14 languages in india, supporting monolingual, cross-lingual, and multilingual summarization in 196 directions. we outlined the processing steps, explained data statistics, discussed quality considerations, and made the dataset publicly accessible. we also published benchmarks for extractive baselines, summarizeand-translate, and fine-tuning, involving two pretrained language models. experiment results emphasize the value of our dataset for summarization between languages in india. a natural extension is to continuously integrate new articles from the source website to update the dataset. however, to ensure that experimental comparisons remain fair, one needs to maintain consistent data sizes in model training. another direction is to ingest more websites to expand the size, domain coverage, language availability, and modality.