Towards a Standardized Dataset on I ndonesian Named Entity Recognition (2020)

Abstract: In recent years, named entity recognition (NER) tasks in the Indonesian language have undergone extensive development. There are only a few corpora for Indonesian NER; hence, recent Indonesian NER studies have used diverse datasets. Although an open dataset is available, it includes only approximately 2,000 sentences and contains inconsistent annotations, thereby preventing accurate training of NER models without reliance on pre-trained models. Therefore, we re-annotated the dataset and compared the two annotations’ performance using the Bidirectional Long Short-Term Memory and Conditional Random Field (BiLSTM-CRF) approach. Fixing the annotation yielded a more consistent result for the organization tag and improved the prediction score by a large margin. Moreover, to take full advantage of pre-trained models, we compared different feature embeddings to determine their impact on the NER task for the Indonesian language.

## introduction named entity recognition (ner) is an essential subtask in natural language processing (nlp). however, ner still suffers from data sparseness for the majority of languages, including indonesian. various indonesian ner approaches have been proposed, ranging from rule-based methods @xcite to machine learning-based techniques @xcite . the dbpedia and wikipedia datasets are mainly used for supervised approaches @xcite @xcite . other datasets include twitter @xcite and conversational datasets from chatbots @xcite , but the sizes of these datasets are limited. unfortunately, almost all of the previous studies on indonesian ner did not release their datasets, which provide essential information for machine learning-based nlp. one indonesian ner dataset with human annotation that is openly available is a news dataset from @xcite ). however, this dataset exhibits inconsistency problems. in this study, we re-annotated this dataset, thereby developing a more standardized indonesian ner resource to improve the nlp foundation for the indonesian language. the most problematic entity is organization, followed by location and person. certain tokens had been tagged as entities that they were not; for example, the term "dpp" (which means "party's representative council") is not an organization name but had been tagged as such. the most recent indonesian ner work that used bilstm-crf was conducted by @xcite using fasttext as the word representation. it has been claimed that fasttext offers advantages in handling misspelled words and out-of-vocabulary (oov) problems @xcite . therefore, we used bilstm-crf with fasttext as our baseline model. we also experimented with bidirectional encoder representations from transformers (bert), a transformer-based language model known to work best in various tasks in nlp as well as ner by acquiring contextual word meanings based on their usage in a sentence @xcite . for the experiment, we compared three models: the multilingual transformer-based models; mbert @xcite and xlm-r @xcite and a monolingual bert for the indonesian language (indobert) @xcite . regarding the limited vocabulary in our low-resourced data, we hypothesized that these embeddings could solve the oov problem because of the vocabu-lary coverage in the large-scale data used for pretraining those embeddings. our contributions can be summarized as follows: ## related works several recent ner methods employed the bidirectional neural network and a conditional random field (crf) as the encoder-decoder layer @xcite @xcite . using contextual information as a representation input to the encoder model improved the score substantially as it helped the model learn the context of the entities. @xcite introduced flair embedding, a contextual string embedding approach, and showed that stacking word and character embeddings increased the model's ability to understand contextual and word-level semantic representations. the use of a transformer (such as in bert) also demonstrated significant results for numerous nlp downstream tasks, for example, by bert, which was proven to work on certain tasks, as well as for ner @xcite . the current state-of-the-art ner model to date was created by fine-tuning a clozedriven pre-trained bidirectional transformer model @xcite . nevertheless, we focus on low-resourced language ner. the majority of previous studies on low-resourced ner also implemented bilstm-crf as the sequence labeling method and experimented with input representation @xcite @xcite . bert has also been employed in several low-resource languages, including bulgarian (marinova, 2019), arabic (antoun et al., 2020), and basque @xcite . deep learning has recently been used in indonesian ner research. the most widely used method is the bilstm algorithm @xcite . various input representation methods have been applied, such as convolutional neural networks (cnns) for word n-gram representation @xcite and pre-trained word embeddings with part-of-speech (pos) tags @xcite . in exploring the oov problem in conversational text, @xcite also employed bilstm-crf without including any pre-trained word representation. in recent work by @xcite , the same neural sequence labeling model was implemented, and pre-trained fasttext indonesian word embedding was applied as the input. in work similar to ours, @xcite investigated the impact of language model pre-training on the ner task. however, their conversational texts data is not publicly available, and therefore their study is not replicable. the latest indonesian ner work is from @xcite , who built and fine-tuned an indonesian bert (indobert) pre-trained model for twelve nlp tasks, including ner. previous work using the same dataset was conducted by @xcite for a quotation identification task. the dataset was constructed from three indonesian online news sites, namely kompas foot_1 , tempo foot_2 , and tribunnews foot_3 . the topics covered by this dataset mainly concern politics, society, and economics. in this task, the data were labeled for quotation identification. however, the ner data were manually tagged as well, as they were used in preprocessing for the quotation identification task. in this study, we focused on the ner task and re-annotated the data because of the inconsistency. ## results annotation performance. the phenomenon whereby many organization tokens were not tagged accounted for a sharp decrease in the f 1 score of the org tag in the s&n (2016) dataset (org: 54.84, loc: 82.18 and per: 86.74). using our test data, we obtained a very high overall score with more consistent performance of the organization tag, as indicated by all three tags sharing relatively similar scores. multilingual vs. monolingual pre-trained models. the monolingual model performed better than the multilingual model when used as the feature representation for bilstm-crf; the multilingual models are better when fine-tuned. wilie et al. (2020) stated that the xlm-r model might achieve a better result on the ner task as entity names usually come from english or other languages. however, most of our dataset's entity names are in indonesian, and hence, the indobert pre-trained model supports this condition better. additionally, mbert and xlm-r use wordpiece @xcite and sentencepiece @xcite tokenization respectively, whereby longer tokens are split into more common tokens. multilingual models contain many languages in their vocabularies. when the pre-trained model is frozen for feature representation use, it is possible that some indonesian sub-words are biased because they share representations with other language's sub-words. the indobert was trained on the indo4b dataset, which also contains indonesian news corpora @xcite . we hypothesize that the bilstm-crf architecture fits better with our sequence classification task, supported by the rich indonesian vocabularies covered by indobert and the domain similarity between the indo4b and our dataset. this is reflected by the very high organization scores obtained from indobert models in both approaches. ## discussion ## conclusions and future work we re-annotated the human-annotated indonesian ner dataset to produce a more consistent annotation in the indonesian ner task. this re-annotation obtained an f 1 score of 90.85 when using the baseline bilstm-crf model was used with fasttext. our implementation also demonstrated that the use of pre-trained transformer-based language models, both the multilingual and the monolingual models, yielded better prediction results. although the performance of indonesian ner using either bilstm-crf or fine-tuning depends on the pretrained language model, we found that indobert works best when using bilstm-crf architecture, compared to the fine-tuning approach. in the future, we plan to address word ambiguity in indonesian by creating a gazetteer to add more supervision and perform distant supervised learning to aid the model in differentiating a word to be classified as each entity as in @xcite . also, we would like to work on other techniques such as transferring knowledge using a teacher-student learning from a high resource language such as english to a low-resource, such as indonesian @xcite .