I ndic NLG Benchmark: Multilingual Datasets for Diverse NLG Tasks in I ndic Languages (2022)

Abstract: Natural Language Generation (NLG) for non-English languages is hampered by the scarcity of datasets in these languages. We present the IndicNLG Benchmark, a collection of datasets for benchmarking NLG for 11 Indic languages. We focus on five diverse tasks, namely, biography generation using Wikipedia infoboxes, news headline generation, sentence summarization, paraphrase generation and, question generation. We describe the created datasets and use them to benchmark the performance of several monolingual and multilingual baselines that leverage pre-trained sequence-to-sequence models. Our results exhibit the strong performance of multilingual language-specific pre-trained models, and the utility of models trained on our dataset for other related NLG tasks. Our dataset creation methods can be easily applied to modest-resource languages as they involve simple steps such as scraping news articles and Wikipedia infoboxes, light cleaning, and pivoting through machine translation data. To the best of our knowledge, the IndicNLG Benchmark is the first NLG benchmark for Indic languages and the most diverse multilingual NLG dataset, with approximately 8M examples across 5 tasks and 11 languages. The datasets and models will be publicly available.

## introduction nlg is the process of generating textual output @xcite . initial work on nlg focused on tabular input @xcite but in a general setting the input can also belong to one or more modalities such as text, images, videos, audio, etc. nlg progress had been hindered by data scarcity for different tasks and languages but, recently, the increasing availability of large scale datasets @xcite @xcite , along with the advancements in neural networks pretrained on large amounts of text @xcite have led to substantial progress in nlg. most of the aforementioned progress is for european languages and especially for english, mainly because it is the lingua franca, making it easy to obtain data for it @xcite . however, english is not the native language for a vast majority of the world's population, which tend to use their country's or region's native languages on a daily basis. india, with its population of 1.4 billion people foot_0 (18% of the world population) is a quintessential example where only 10% of the population speaks english whereas a significant portion of the remaining 90% speaks one or more of the 22 'scheduled' indian languages listed in the constitution of india foot_1 . it is not surprising that most people in india tend to consume literature and media in indian languages rather than english. therefore, we believe that it is important to focus on indic nlg, which lacks datasets for diverse nlg tasks. given that there is no existing or widely used datasets for diverse indic nlg tasks foot_2 , this paper aims to fill this gap, via the indicnlg benchmark task languages communicative intent input type size biography generation l-{gu, mr} one-sentence biography key-value pairs 57k headline generation l news article headlines news article 1.31m sentence summarization l synonymous compact sentence sentence 431k paraphrase generation l synonymous sentence sentence 5.57m question generation ## related work this paper focuses on data creation, modeling and benchmarking for nlg using pre-trained models and multilingualism. ## indicnlg benchmark the indicnlg benchmark is a collection of datasets which we use to benchmark nlg performance for 5 nlg tasks spanning 11 indic languages. in this section, we describe the datasets and their sizes. ## dataset quality we study the quality of our automatically created datasets by conducting a human evaluation exercise. we choose two languages from each of indo-aryan and dravidian language families as representative languages viz we choose hi and bn from indo-aryan and ml and kn from dravidian language families. we conduct this study for only three tasks out of four (wikibio, headline generation and paraphrasing) due to limited annotation budget. we annotate 250 examples in wikibio and paraphrasing, and 100 examples in headline generation (the examples are longer here, so we annotate fewer examples to save on annotation cost). for each language, we hire two native-language annotators and pay them higher wages than the minimum hourly wages. for annotation of wikibio and headline generation we follow the human evaluation setup of @xcite . specifically, we ask raters to annotate for three properties. property a is yes if the output and input pair are aligned. property b is yes if the output contains information inconsistent with the input. property c is yes if the output contains extra in-formation that cannot be inferred from the input. the numbers of property a, b and c mean the ratios of 'yes' given by the annotators. it is desirable that values of property a should be high, and values for property b and c be low. we find that property a is greater than 90% indicating high match between the input and output example pair. property c values are greater than 10% in wik-ibio, indicating the extra information present in output, an observation also made in @xcite . we find that the amount of extra information present in the output of indic wikibio dataset is similar in nature to abstractive summarization dataset such as xsum (71.7% from for the annotation of paraphrasing data, we follow the setup of @xcite , and adopt finegrained labels on the similarity between the input and output pairs, with a value of 5 indicating perfect match, down to 0 indicating unrelated pairs (see we release the annotated data along with the datasets and the trained models. ## experimental settings we establish strong baseline models using pretrained models and multilingualism. we describe the experimental settings for generating benchmark scores for all the tasks. ## results and analysis in this section, we present the results obtained using models trained for a variety of tasks and analyze them from various perspectives. we compare between models fine-tuned on indicbart (ib), separate script indicbart (ssib) and mt5 in monolingual and multilingual settings. ## conclusion and future work we present the indicnlg benchmark, a collection of datasets for 5 diverse nlg tasks in 11 indic languages and 8m examples, with the aim of creating much-required standard benchmarks to drive indic nlg research. to the best of our knowledge, this is the most diverse multilingual nlg dataset. our methods are simple enough to create similar datasets for modest resourced languages. we trained a variety of monolingual and multilingual models and showed the impact of the combination of multilingualism and pre-trained models. in general, multilingual models outperform their monolingual counterparts. language-family specific pre-trained models like indicbart are valuable as they are competitive with large multilingual models like mt5 despite needing only 14% the compute (147b vs 1t training tokens), while having smaller vocabularies (64k vs 250k) and fewer params (244m vs 300m). given these ob-servations, we recommend future baselines to consider multilingual fine-tuning of language familyspecific models as the starting point. future work will also involve extending the datasets for additional indic languages and new tasks.