I ndo NLG : Benchmark and Resources for Evaluating I ndonesian Natural Language Generation (2021)

Abstract: Natural language generation (NLG) benchmarks provide an important avenue to measure progress and develop better NLG systems. Unfortunately, the lack of publicly available NLG benchmarks for low-resource languages poses a challenging barrier for building NLG systems that work well for languages with limited amounts of data. Here we introduce IndoNLG, the first benchmark to measure natural language generation (NLG) progress in three low-resource—yet widely spoken—languages of Indonesia: Indonesian, Javanese, and Sundanese. Altogether, these languages are spoken by more than 100 million native speakers, and hence constitute an important use case of NLG systems today. Concretely, IndoNLG covers six tasks: summarization, question answering, chit-chat, and three different pairs of machine translation (MT) tasks. We collate a clean pretraining corpus of Indonesian, Sundanese, and Javanese datasets, Indo4B-Plus, which is used to pretrain our models: IndoBART and IndoGPT. We show that IndoBART and IndoGPT achieve competitive performance on all tasks—despite using only one-fifth the parameters of a larger multilingual model, mBART-large (Liu et al., 2020). This finding emphasizes the importance of pretraining on closely related, localized languages to achieve more efficient learning and faster inference at very low-resource languages like Javanese and Sundanese.

## introduction resources such as datasets, pretrained models, and benchmarks are crucial for the advancement of natural language processing (nlp) research. nevertheless, most pretrained models and datasets are developed for high-resource languages such as english, french, and chinese @xcite @xcite . although the number of datasets, models, and benchmarks has been increasing for low-resource languages such as indonesian @xcite , bangla @xcite , and filipino (cruz and cheng, 2020), these datasets primarily focus on natural language understanding (nlu) tasks, which only cover a subset of practical nlp systems today. in contrast, much fewer natural language generation (nlg) benchmarks have been developed for low-resource languages; most multilingual nlg resources thus far have primarily focused on machine translation, highlighting the need to generalize these low-resource nlg benchmarks to other commonly used nlg tasks such as summarization and question answering. while recent work has developed more comprehensive multilingual nlg benchmarks, such as xglue @xcite and gem @xcite , these efforts still primarily evaluate the nlg models on fairly high-resource languages. in this paper, we take a step towards building nlg models for some low-resource languages by introducing indonlg-a benchmark of multilingual resources and standardized evaluation data for three widely spoken languages of indonesia: indonesian, javanese, and sundanese. cumulatively, these languages are spoken by more than 100 million native speakers, and thus comprise an important use case of nlg systems today. despite the prevalence of these languages, there has been relatively few prior work on developing accurate nlg systems for these languages-a limitation we attribute to a lack of publicly available resources and evaluation benchmarks. to help address this problem, indonlg encompasses clean pretraining data, pretrained models, and downstream nlg tasks for these three languages. for the downstream tasks, we collect pre-existing datasets for ## experimental settings in this section, we describe the models and outline how we train and evaluate our models. ## evaluation procedure for evaluation, we use beam search with a beam width of 5, a length penalty α of 1.0, and limit the maximum sequence length to 512 for all models and all tasks. we conduct both automatic and human evaluations to assess the models. we use a different evaluation metric for each task following the standard evaluation metric on the corresponding task. for machine translation, we report the sacrebleu @xcite score. for summarization, we report the rouge @xcite score. for qa, the f1 and exact match scores are reported following the original squad v2 @xcite evaluation metrics. for chit-chat, we report both the bleu and sacrebleu scores @xcite . we further conduct human evaluation on eight tasks, i.e., en ↔ id (news), su ↔ id (bible), jv ↔ id (bible), liputan6 xtreme, and xpersona. we randomly select 100 input samples from the test set of each task and evaluate six different generation models for each input sample, i.e., ground-truth label, scratch, mbart large , mt5 small , indo-bart, and indogpt. for machine translation, we measure two metrics, i.e., fluency and adequacy. for summarization, we measure four metrics, i.e., coherence, consistency, fluency, and relevance. for chit-chat, we measure three metrics, i.e., consistency, engagingness, and fluency. beyond those metrics, we gather the rank of the generated texts for each sample to measure the relative quality of the models. the complete human annotation guideline is shown in appendix d. as shown in human evaluation. as shown in figure 1 , the overall quality of models with respect to human evaluation can be ranked in the following order: mbart large , indobart, mt5 small , indogpt, and the scratch models. this finding is supported by the individual task metrics shown in ## conclusion we introduced the first indonesian benchmark for natural language generation, indonlg. our benchmark consists of six tasks: summarization, question answering, open chit-chat, and three different language pairs of machine translation tasks. we provide a large and clean pretraining corpus of indonesian, sundanese, and javanese datasets called indo4b-plus, which is used to pretrain our nlg models, indobart and indogpt. we evaluate the effectiveness and efficiency of our models by conducting extensive automatic and human evaluations on the indonlg tasks. based on the evaluation, our indobart and indogpt models achieve a competitive (albeit slightly lower) performance to help with the reproducibility of the benchmark, we release the pretrained models, including the collected data and code. in order to accelerate community engagement and benchmark transparency, we have set up a leaderboard website for the nlp community. we publish all of our resources including indobart, indogpt, and indonlg tasks at https://github.com/ indobenchmark/indonlg .