I ndic BART : A Pre-trained Model for Indic Natural Language Generation (2022)

Abstract: In this paper, we study pre-trained sequence-to-sequence models for a group of related languages, with a focus on Indic languages. We present IndicBART, a multilingual, sequence-to-sequence pre-trained model focusing on 11 Indic languages and English. IndicBART utilizes the orthographic similarity between Indic scripts to improve transfer learning between similar Indic languages. We evaluate IndicBART on two NLG tasks: Neural Machine Translation (NMT) and extreme summarization. Our experiments on NMT and extreme summarization show that a model specific to related languages like IndicBART is competitive with large pre-trained models like mBART50 despite being significantly smaller. It also performs well on very low-resource translation scenarios where languages are not included in pre-training or fine-tuning. Script sharing, multilingual training, and better utilization of limited model capacity contribute to the good performance of the compact IndicBART model.

## introduction recently, there has been significant progress in deep learning based natural language generation (nlg) for machine translation, abstractive summarization, data-to-text generation, etc. due to the adoption of attention-based sequence-to-sequence (s2s) models (conditional language models) @xcite @xcite . pre-trained s2s models have been shown to be useful to improve performance on various nlg tasks @xcite @xcite . specifically, multilingual pre-trained s2s models jointly trained on monolingual corpora from multiple languages such as mbart25 @xcite , mbart50 @xcite and mt5 @xcite have seen increased adoption and low-resource languages have benefitted from cross-lingual transfer. however, these massively multilingual massive (m3) models have major limitations. they serve only a few of the world's languages (<100 languages), the pre-training corpora are dominated by highresource languages, the vocabulary representation for low-resource languages is inadequate, and the models are large, making them expensive and slow to train, fine-tune and decode. an alternative approach is to build pre-trained s2s models for a group of related languages. previous work has shown the benefits of pre-trained language models as well as nmt models that cater to a set of related languages @xcite @xcite . owing to their public availability, these models have seen heavy adoption 1 . however, such a study on multilingual pre-trained s2s models for indic languages is missing in the literature. in this work, we address this gap in the literature by studying multilingual pre-trained s2s models for indic languages. the result of this study is indicbart, a multilingual pre-trained sequence to sequence model specifically trained for indic languages, which are spoken by more than a billion users 2 . it supports english and 11 indian languages including 7 indo-aryan (assamese, bengali, gujarati, hindi, marathi, oriya, punjabi) and 4 dravidian (kannada, malayalam, tamil, telugu) languages. of these, mbart25, mbart50 and mt5 support only 2, 7 and 9 languages respectively. there are linguistic similarities between the two language families on account of contact relatedness resulting from geographical colocation. within, the two language families there are genetic relations between languages due to them being derived from common ancestor languages 34 . due to this, the indian subcontinent is considered to be a linguistic area or sprachbund @xcite . there is evidence that such contact-relatedness can result in positive cross-lingual transfer for nlp applications like nmt @xcite . hence, we train a single model for all indic languages. it is a compact model with just 244m parameters, which is much smaller than the m3 models such as mbart50 and mt5(-base) which contain 611m and 580m parameters respectively. we also propose a variant of indicbart, i.e. indicalbart, that is highly compact with just 97m parameters. we compare indicbart with m3 models on two downstream generation tasks: machine translation and extreme summarization @xcite . the results indicate that indicbart is competitive or better by up to 2 bleu/rouge compared to m3 models like mbart50. indicbart also performs well in the following zero-shot scenarios: (a) on languages not included in pre-training, and (b) languages for which there is no fine-tuning data. the following aspects of the indicbart model contribute to its strong performance and increased language coverage within the indic group vis-à-vis m3 models, while being highly compact: 1. it is trained on a smaller set of related languages, which reduces model capacity requirements. moreover, available model capacity is effectively utilized, since transfer learning works when languages share linguistic features and data represents shared topical themes. 2. it is trained on the largest publicly available indic language corpora, indiccorp @xcite , which includes large, high-quality news crawls for indian languages as well as english content from indian websites -thus being representative of indian english and topics. 3. we utilize the orthographic similarity between indic scripts @xcite to map all the indic language data to a single script, effectively reducing the number of scripts from 9 to 1 (each script having approximately 50 characters). this increases the shared subwords in the vocabulary, and we observe that single script models enable better cross-lingual transfer while fine-tuning. since subwords embeddings consume a significant fraction of the parameter space, single script models also better utilize available vocabulary budget foot_4 . 4. extremely compressed pre-trained s2s models (indicalbart) suitable for deployment can be trained by sharing parameters across layers of the transformer layers. for related languages, we show compressed pre-trained models are competitive with full models on downstream tasks when fine-tuned on distilled data. the indicbart model and its variants, along with details on how to fine-tune them, can be accessed at https://github.com/ ai4bharat/indic-bart/ . we also release the models on the huggingface model hub at https://huggingface.co/ai4bharat/ indicbart and https://huggingface. co/ai4bharat/indicbartss . models are available under an mit license to spur further innovation in nlg for indic languages and study of pre-trained s2s models for related languages. our results indicate that these results hold when fine-tuning on pre-trained s2s models as well. ## indicbart the indicbart model is conceptually based on the mbart25/50 model family, which are transformer models @xcite trained on monolingual corpora with masked span reconstruction objective. we refer the readers to the mbart literature @xcite for architectural details and highlight specific details and differences from the mbart25/50 setup. ## experiments: nmt machine translation is a standard, popular, crosslingual generation task for which various pretrained models are evaluated. we compare in-dicbart and its variants with mbart50, which should be the most directly comparable model. we study their performance in: (a) low-resource, (b) multilingual and (c) zero-shot training settings. ## experiments: extreme summarization we compare the performance of fine-tuning in-dicbart, its variants and mbart50 on the challenging extreme summarization task @xcite for indic languages. the small datasets, enable a good study of the utility of pre-training. ## conclusion and future work we presented indicbart, a multilingual, pretrained sequence-to-sequence model to support development of nlg applications for indian languages. indicbart supports 11 indian languages and english, and utilizes the orthographic similarity of indic scripts to enable better cross-lingual transfer. indicbart presents a case-study for language group-specific pre-trained s2s models. our experiments on fine-tuning indicbart for nmt and summarization showed that the model is competitive with large models such as mbart50. we further compressed indicbart while maintaining downstream task performance via parameter sharing (indicalbart) combined with multilingual distillation. we showed that script unification has a strong positive impact on translation and summarization. we also showed that indicbart, thanks to its script independent nature, can be readily used for enabling translation for languages such as sinhala and nepali which indicbart has not been explicitly pre-trained for. furthermore, we showed that fine-tuning indicbart on one set of languages enables translation for another unseen set of languages, which shows that pre-trained models enable translation without parallel corpora. in the future, we plan to support more indic languages in indicbart; starting with all the 22 17 languages listed in the 8 th schedule of the indian constitution. increased language coverage and models with lower compute demands can democratize access to nlp technologies. we also plan to focus on training models on longer text chunks (documents) and larger text corpora, incorporating advances in multilingual pre-training, cross-lingual transfer and cross-lingual tasks for indian languages.