I ndo NLU : Benchmark and Resources for Evaluating I ndonesian Natural Language Understanding (2020)

Abstract: Although Indonesian is known to be the fourth most frequently used language over the internet, the research progress on this language in natural language processing (NLP) is slow-moving due to a lack of available resources. In response, we introduce the first-ever vast resource for training, evaluation, and benchmarking on Indonesian natural language understanding (IndoNLU) tasks. IndoNLU includes twelve tasks, ranging from single sentence classification to pair-sentences sequence labeling with different levels of complexity. The datasets for the tasks lie in different domains and styles to ensure task diversity. We also provide a set of Indonesian pre-trained models (IndoBERT) trained from a large and clean Indonesian dataset (Indo4B) collected from publicly available sources such as social media texts, blogs, news, and websites. We release baseline models for all twelve tasks, as well as the framework for benchmark evaluation, thus enabling everyone to benchmark their system performances.

## introduction following the notable success of contextual pretrained language methods @xcite , several benchmarks to gauge the progress of general-purpose nlp research, such as glue @xcite , superglue @xcite , and clue @xcite , have been proposed. these benchmarks cover a large range of tasks to measure how well pre-trained models achieve compared to humans. however, these metrics are limited to high-resource languages, such as english and chinese, that already have existing datasets available and are accessible to the research community. most languages, by contrast, suffer from limited data collection and low awareness of published data for research. one of the languages which suffer from this resource scarcity problem is indonesian. indonesian is the fourth largest language used over the internet, with around 171 million users across the globe. 1 despite a large amount of indonesian data available over the internet, the advancement of nlp research in indonesian is slowmoving. this problem occurs because available datasets are scattered, with a lack of documentation and minimal community engagement. moreover, many existing studies in indonesian nlp do not provide codes and test splits, making it impossible to reproduce results. to address the data scarcity problem, we propose the first-ever indonesian natural language understanding benchmark, indonlu, a collection of twelve diverse tasks. the tasks are mainly categorized based on the input, such as single-sentences and sentence-pairs, and objectives, such as sentence classification tasks and sequence labeling tasks. the benchmark is designed to cater to a range of styles in both formal and colloquial indonesian, which are highly diverse. we collect a range of datasets from existing works: an emotion classification dataset @xcite , qa factoid dataset @xcite , sentiment analysis dataset @xcite , aspect-based sentiment analysis dataset @xcite , part-ofspeech (pos) tag dataset @xcite , named entity recognition (ner) dataset @xcite , span extraction dataset @xcite @xcite , and textual entailment dataset @xcite . it is difficult to compare model performance since there is no official split of information for existing datasets. therefore we standardize the benchmark by resplitting the datasets on each task for reproducibility purposes. to expedite the modeling and evaluation processes for this benchmark, we present samples of the model pre-training code and a framework to evaluate models in all downstream tasks. we will publish the score of our benchmark on a publicly accessible leaderboard to provide better community engagement and benchmark transparency. to further advance indonesian nlp research, we collect around four billion words from indonesian preprocessed text data (≈ 23 gb), as a new standard dataset, called indo4b, for self-supervised learning. the dataset comes from sources like online news, social media, wikipedia, online articles, subtitles from video recordings, and parallel datasets. we then introduce an indonesian bertbased model, indobert, which is trained on our indo4b dataset. we also introduce another in-dobert variant based on the albert model @xcite , called indobert-lite. the two variants of indobert are used as baseline models in the indonlu benchmark. in this work, we also extensively compare our indobert models to different pre-trained word embeddings and existing multilingual pre-trained models, such as multilingual bert @xcite and xlm-r @xcite , to measure their effectiveness. results show that our pre-trained models outperform most of the existing pre-trained models. ## indonlu benchmark in this section, we describe our benchmark as four components. firstly, we introduce the 12 tasks in indonlu for indonesian natural language understanding. secondly, we introduce a large-scale indonesian dataset for self-supervised pre-training models. thirdly, we explain the various kinds of baseline models used in our indonlu benchmark. lastly, we describe the evaluation metric used to standardize the scoring over different models in our indonlu benchmark. ## indobert in this section, we describe the details of our indonesian contextualized models, indobert and indobert-lite, which are trained using our indo4b dataset. we elucidate the extensive details of the models' development, first the dataset preprocessing, followed by the pre-training setup. ## results and analysis in this section, we show the results of the indonlu benchmark and analyze the performance of our models in terms of downstream tasks score and performance-space trade-off. in addition, we show an analysis of the effectiveness of using our collected data compared to existing baselines. ## conclusion we introduce the first indonesian benchmark for natural language understanding, indonlu, which consists of 12 tasks, with different levels of difficulty, domains, and styles. to establish a strong baseline, we collect large clean indonesian datasets into a dataset called indo4b, which we use for training monolingual contextual pre-trained language models, called indobert and indobertlite. we demonstrate the effectiveness of our dataset and our pre-trained models in capturing sentence-level semantics, and apply them to the classification and sequence labeling tasks. to help with the reproducibility of the benchmark, we release the pre-trained models, including the collected data and code. in order to accelerate the community engagement and benchmark transparency, we have set up a leaderboard website for the nlp community. we publish our leaderboard website at https://indobenchmark.com/ .