I ndo LEM and I ndo BERT : A Benchmark Dataset and Pre-trained Language Model for I ndonesian NLP (2020)

Abstract: Although the Indonesian language is spoken by almost 200 million people and the 10th most spoken language in the world, it is under-represented in NLP research. Previous work on Indonesian has been hampered by a lack of annotated datasets, a sparsity of language resources, and a lack of resource standardization. In this work, we release the IndoLEM dataset comprising seven tasks for the Indonesian language, spanning morpho-syntax, semantics, and discourse. We additionally release IndoBERT, a new pre-trained language model for Indonesian, and evaluate it over IndoLEM, in addition to benchmarking it against existing resources. Our experiments show that IndoBERT achieves state-of-the-art performance over most of the tasks in IndoLEM.

## introduction despite there being over 200m first-language speakers of the indonesian language, the language is underrepresented in nlp. we argue that there are three root causes: a lack of annotated datasets, a sparsity of language resources, and a lack of resource standardization. in english, on the other hand, there are ever-increasing numbers of datasets for different tasks @xcite @xcite , (pre-)trained models for language modelling and language understanding tasks @xcite @xcite , and standardized tasks to benchmark research progress @xcite @xcite , all of which have contributed to rapid progress in the field in recent years. we attempt to redress this situation for indonesian, as follows. first, we introduce indolem ("indonesian language evaluation montage" 2 ), a comprehensive dataset encompassing seven nlp tasks and eight sub-datasets, five of which are based on previous work and three are novel to this work. as part of this, we standardize data splits and evaluation metrics, to enhance reproducibility and robust benchmarking. these tasks are intended to span a broad range of morpho-syntactic, semantic, and discourse analysis competencies for indonesian, to be able to benchmark progress in indonesian nlp. first, for morpho-syntax, we examine part-of-speech (pos) tagging @xcite , dependency parsing with two universal dependency (ud) datasets, and two named entity recognition (ner) tasks using public data. for semantics, we examine sentiment analysis and single-document summarization. for discourse, we create two twitter-based document coherence tasks: twitter response prediction (as a multiple-choice task), and twitter document thread ordering. second, we develop and release indobert, a monolingual pre-trained bert language model for indonesian @xcite . this is one of the first monolingual bert models for the indonesian language, trained following the best practice in the field. 3 our contributions in this paper are: (1) we release indolem, which is by far the most comprehensive nlp dataset for indonesian, and intended to provide a benchmark to catalyze further nlp research on the language; (2) as part of indolem, we develop two novel discourse tasks and datasets; and (3) we follow best practice in developing and releasing for general use indobert, a bert model for indonesian, which we show to be superior to existing pre-trained models based on indolem. the indolem dataset, indobert model, and all code associated with this paper can be accessed at: https://indolem.github.io . ## related work to comprehensively evaluate natural language understanding (nlu) methods for english, collections of tools and corpora such as glue @xcite and superglue @xcite have been proposed. generally, such collections aim to benchmark models across various nlp tasks covering a variety of corpus sizes, domains, and task formulations. glue comprises nine language understanding tasks built on existing public datasets, while superglue is a set of eight tasks that is not only diverse in task format but also includes low-resource settings. superglue is a more challenging framework, and bert models trail human performance by 20 points at the time of writing. in the cross-lingual setting, xglue @xcite was introduced as a benchmark dataset that covers nearly 20 languages. unlike glue, xglue includes language generation tasks such as question and headline generation. one of the largest cross-lingual corpora is dependency parsing provided by universal dependencies. 4 it has consistent annotation of 150 treebanks across 90 languages, constructed through an open collaboration involving many contributors. recently, other cross-lingual benchmarks have been introduced, such as @xcite and @xcite . while these three cross-lingual benchmarks contain some resources/datasets for indonesian, the coverage is low and data is limited. beyond the english and cross-lingual settings, chineseglue foot_1 is a comprehensive nlu collection for mandarin chinese, covering eight different tasks. for the vietnamese language, nguyen and nguyen (2020) gathered a dataset covering four tasks (ner, pos tagging, dependency parsing, and language inference), and empirically evaluated them against a monolingual bert. elsewhere, there are individual efforts to maintain a systematic catalogue of tasks and datasets, and state-of-the-art methods for each across multiple languages, 6 including one specifically for indonesian. 7 however, there is no comprehensive dataset for evaluating nlu systems in the indonesian language, a void which we seek to fill with indolem. ## indobert transformers @xcite have driven substantial progress in nlp research based on pretrained models in the last few years. although attention-based models are data-and gpu-hungry, the full attention mechanisms and parallelism offered by the transformer are highly compatible with the high levels of parallelism that gpu computation offers, and have been shown to be highly effective at capturing the syntax @xcite and sentence semantics of text @xcite . in particular, transformer-based language models @xcite @xcite pre-trained on large volumes of text based on simple tasks such as masked word prediction and sentence ordering prediction, have quickly become ubiquitous in nlp and driven substantial empirical gains across tasks including ner @xcite , pos tagging @xcite , single document summarization @xcite , syntactic parsing @xcite , and discourse analysis @xcite . however, this effect has been largely observed for high-resource languages such as english. indobert is a transformer-based model in the style of bert @xcite , but trained purely as a masked language model trained using the huggingface foot_4 framework, following the default configura- in total, we train indobert over 220m words, aggregated from three main sources: (1) indonesian wikipedia (74m words); (2) news articles from kompas, foot_6 tempo foot_7 @xcite , and liputan6 foot_8 (55m words in total); and (3) an indonesian web corpus @xcite (90m words). after preprocessing the corpus into 512-token document blocks, we obtain 1,067,581 train instances and 13,985 development instances (without reduplication). in training, we use 4 nvidia v100 gpus (16gb each) with a batch size of 128, learning rate of 1e-4, the adam optimizer, and a linear scheduler. we trained the model for 2.4m steps (180 epochs) for a total of 2 calendar months, 13 with the final perplexity over the development set being 3.97 (similar to english bert-base). ## indolem: tasks in this section, we present an overview of indolem, in terms of the nlp tasks and sub-datasets it includes. we group the tasks into three categories: morpho-syntax/sequence labelling, semantics, and discourse coherence. we summarize the sub-datasets include in indolem in ## evaluation methodology we provide details of the evaluation methodology in this section. morpho-syntax/sequence labelling. for pos tagging, we evaluate by 5-fold cross validation using the partitions provided by @xcite . unlike @xcite who use macroaveraged f1, we use the standard pos tag accuracy for evaluation. for ner, both corpora (ner ui and ner ugm) are from the news domain. we convert them into iob2 format, and reserve 10% of the original training set as a validation set. we evaluate using entity-level f1 over the provided test set. 24 in addition, we conducted our own in-house evaluation of the annotation quality of both datasets by randomly picking 100 sentences and counting the number of annotation errors. we found that ner ui has better quality than ner ugm with 1% vs. 30% errors, respectively. annotation errors in ner ugm are largely due to low recall, i.e. annotating named entities with the tag o. for dependency parsing we do not apply 5-fold cross-validation for ud-indo-gsd, as it was released with a pre-defined test set, which allows us to directly benchmark against previous work. ud-indo-pud, on the other hand, only includes 1,000 sentences with no fixed test set, so we evaluate via 5-fold cross-validation with fixed splits. 25 note that the text in ud-indo-pud was manually translated from documents in other languages, while ud-indo-gsd was sourced from texts authored in indonesian. additionally, the translation quality of ud-indo-pud is low in parts, which impacts on evaluation, as we return to discuss in section 7. we evaluate both dependency parsing datasets based on the unlabelled attachment score (uas) and labelled attachment score (las). ## results in for the semantic tasks, indobert outperforms all other methods for both sentiment analysis and extractive summarization ( lastly, in overall, with the possible exception of pos tagging and ntp, there is substantial room for improve- ## conclusion in this paper, we introduced indolem, a comprehensive dataset encompassing seven tasks, spanning morpho-syntax, semantics, and discourse coherence. we also detailed indobert, a new bert-style monolingual pre-trained language model for indonesian. we used indolem to benchmark indobert (including comparative evaluation against a broad range of baselines and competitor bert models), and showed it to achieve state-of-the-art performance over the dataset.