B angla NLG and B angla T 5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in B angla (2023)

Abstract: This work presents ‘BanglaNLG,’ a comprehensive benchmark for evaluating natural language generation (NLG) models in Bangla, a widely spoken yet low-resource language. We aggregate six challenging conditional text generation tasks under the BanglaNLG benchmark, introducing a new dataset on dialogue generation in the process. Furthermore, using a clean corpus of 27.5 GB of Bangla data, we pretrain ‘BanglaT5’, a sequence-to-sequence Transformer language model for Bangla. BanglaT5 achieves state-of-the-art performance in all of these tasks, outperforming several multilingual models by up to 9% absolute gain and 32% relative gain. We are making the new dialogue dataset and the BanglaT5 model publicly available at https://github.com/csebuetnlp/BanglaNLG in the hope of advancing future research on Bangla NLG.

## introduction the emergence of pretrained language models @xcite @xcite has brought about a revolutionary change in natural language processing (nlp). with little taskspecific fine-tuning, these models have achieved state-of-the-art results on many nlp tasks @xcite @xcite . however, the focus of these models has predominantly been on natural language understanding (nlu). even models pretrained with generative objectives @xcite concern themselves with nlu tasks more than natural language generation (nlg) tasks. although there have been recent efforts to uplift nlg @xcite , they are primarily geared towards high-and mid-resource languages. for example, despite being the sixth most spoken language in the world with over 230 million native speakers comprising 3% of the world's total population, 1 bangla has remained an underrepresented 1 https://w.wiki/psq language in the nlp literature @xcite . there have been only a handful of benchmark studies on bangla nlg @xcite , and that too without bangla being the main focus. this can be attributed to the lack of diverse nlg tasks under a single benchmark and strong pretrained bangla nlg models. to this end, we present 'banglanlg,' a comprehensive benchmark for bangla language generation comprising six representative tasks on machine translation, text summarization, question answering, dialogue generation, headline generation, and cross-lingual summarization. to our knowledge, banglanlg is the first nlg benchmark exclusively for a low-resource language. to establish a strong baseline for this benchmark, we pretrain banglat5 -a sequence-to-sequence transformer model @xcite pretrained on a 27.5 gb clean bangla text corpus covering a broad range of domains. in summary: banglat5 outperforms similar-sized multilingual models, achieving new state-of-the-art results on three tasks with a 4% gain on average. we are releasing the banglat5 model and a live leaderboard to promote future research on bangla nlg. ## banglat5 we introduce banglat5, a sequence-to-sequence transformer model @xcite , to establish a strong baseline for banglanlg benchmark. in this section, we describe the pretraining data, objectives, and model architecture of banglat5. ## experiments & results we compared banglat5 it with four multilingual models: mt5 (base) @xcite , mbart-50 @xcite , xlm-prophetnet @xcite , and indicbart (both unified and separate script variants) @xcite . 4 all pretrained models were fine-tuned for 3-15 epochs with batch size 32 (128 for mt). we used linear warmup with a ratio of 0.1, label smoothing of 0.1 @xcite , and weight decay of 1e-6 with the adam optimizer @xcite . the learning rate was tuned from the set {5e-5, 1e-4, 5e-4}. the best model was evaluated based on the validation performance after each epoch. during inference, we used beam-search @xcite with beam size 5 (on all tasks except qa), removed duplicated trigrams during beam search @xcite , and used a length penalty @xcite of 0.6. for qa, we used greedy decoding, i.e., picking the most probable token during each decoding step. the evaluation results are presented in we find the mt results particularly interesting, where banglat5 outperforms larger multilingual models in both directions. this suggests that despite having very little english data in the pretraining corpus, banglat5 can generalize well to a new translation language, given high-quality fine-tuning data. we explore this more in the appendix. conspicuously, all the models achieve relatively poor scores on the xls task. this can be attributed to the smaller amount of training data. banglat5 proves its superiority in compute and memory efficiency along with its performance due to its smaller size (less than half the parameters of all multilingual models except indicbart). in practice, we observe 2-2.5x faster training and inference times with banglat5 than these larger multilingual models. ## related works pretrained models nlp has witnessed a sea of change with the advent of pretrained language models like ulmfit @xcite , elmo @xcite , and most notably bert @xcite , achieving state-of-the-art results in many nlu benchmarks. besides these nlu models, more and more pretrained models designed for nlg tasks have been proposed. @xcite adopted pretrained nlu model checkpoints for generative tasks. gpt-2 @xcite , and later gpt-3 @xcite showed that pretrained generative language models can perform remarkably well in zero-shot transfer tasks. more recently, @xcite proposed prophet-net, which introduces the future n-gram prediction mechanism for language generation. @xcite introduced indicbart, which is pretrained on 11 indic languages, including bangla. nlg benchmarks recently, many multi-task benchmarks have been proposed to drive the progress of nlg models. @xcite proposed the beng benchmark for nlg and knowledge extraction. glge @xcite is a similar benchmark with a different set of tasks and difficulty levels. however, these benchmarks are limited to english only. @xcite introduced the gem benchmark for various tasks such as summarization @xcite , datato-text generation @xcite across different languages. @xcite introduced different tasks and baselines for 3 indonesian languages. more recently, @xcite introduced indicnlg, a benchmark with five tasks in 11 indic languages, including bangla. ## conclusion & future works nlp research in low-resource languages is lagging behind due to the lack of reliable benchmarks and datasets. to facilitate the development, evaluation, and comparison of new nlg models, we introduced a multi-task evaluation benchmark for bangla nlg, a widely spoken yet low-resource language. we presented banglat5, a pretrained nlg model in bangla, setting new state-of-the-art results with banglat5. we strongly believe that our contributions in this work will help the bangla nlp community benchmark nlg tasks more easily under a unified setup. in future work, we plan to introduce new tasks to banglanlg, such as personalized dialogue generation @xcite , conversational questionanswering @xcite . we will also add more recent multilingual models to our comparison to banglat5, e.g., deltalm @xcite .