ICON : Building a Large-Scale Benchmark Constituency Treebank for the I ndonesian Language (2023)

Abstract: Constituency parsing is an important task of informing how words are combined to form sentences. While constituency parsing in English has seen significant progress in the last few years, tools for constituency parsing in Indonesian remain few and far between. In this work, we publish ICON (Indonesian CONstituency treebank), the hitherto largest publicly-available manually-annotated benchmark constituency treebank for the Indonesian language with a size of 10,000 sentences and approximately 124,000 constituents and 182,000 tokens, which can support the training of state-of-the-art transformer-based models. We establish strong baselines on the ICON dataset using the Berkeley Neural Parser with transformer-based pre-trained embeddings, with the best performance of 88.85% F1 score coming from our own version of SpanBERT (IndoSpanBERT). We further analyze the predictions made by our best-performing model to reveal certain idiosyncrasies in the Indonesian language that pose challenges for constituency parsing.

## introduction constituency parsing is an important task of informing how words are combined to form sentences. it uses context-free grammars (cfg) to assign a structure, usually in the form of a hierarchical syntactic parse tree, to a sentence. parse trees can be used directly in applications such as grammar checking @xcite while linguistic features engineered through parsing can be used to boost the performance of downstream models for higherlevel tasks such as semantic role labeling (fei et * equal contribution @xcite , machine translation @xcite , natural language inference @xcite , opinion mining @xcite ), text summarization @xcite and relation extraction @xcite . there is another important family of grammar formalism called dependency grammar. while dependency parsing has become increasingly prevalent, this does not obviate the need for constituency parsing since the two can be used for different purposes. for span-labeling tasks such as coreference resolution, it has been argued that the explicit encoding of the boundaries of non-terminal phrases in constituency trees makes them more beneficial to the task than dependency trees @xcite . the indonesian language is the national and primary language of indonesia, the world's fourth largest country by population at the time of writing with almost 275 million people @xcite . there has been mounting interest in the development of indonesian natural language processing (nlp) although tools for constituency parsing remain few and far between. the progress in constituency parsing for the indonesian language has been hampered by the absence of a large-scale benchmark dataset that can support the training of the current state-of-the-art (sota) transformer-based models, which have been pushing the envelope of english constituency parsing. in light of this, we introduce icon (indonesian constituency treebank), a 10,000tree benchmark constituency parsing dataset for the indonesian language. it is the hitherto largest publicly-available dataset for indonesian constituency parsing. we also establish strong baselines on this treebank using the berkeley neural parser @xcite and a suite of pre-trained embeddings. the rest of the paper is organized as follows: section 2 reviews related work. section 3 looks at the icon treebank in more detail. section 4 explains the experiments we ran on the treebank and section 5 puts forward findings from our analyses and sheds light on the challenges in indonesian constituency parsing. lastly, in section 6, we present our conclusions and lay out suggestions for future works. ## training models with icon to establish a baseline on the icon treebank, which could be used as a benchmark for future works on indonesian constituency parsing, we trained the berkeley neural parser @xcite on the treebank with a suite of indonesian and multilingual pre-trained language embeddings. ## analysis a breakdown of the performance of the best model (indospanbert-base) by constituent labeling and pos tagging can be found in appendices d and e. an in-depth error analysis of the predictions made by our trained parser revealed certain idiosyncrasies in the indonesian language that pose challenges for constituency parsing. word order is relatively flexible in indonesian @xcite despite the lack of morphological case markings. furthermore, the fact that predicates in indonesian are not only verbal, like in english, but can also be nominal, adjectival and prepositional @xcite , means that the cfg production rules are going to be much more diverse and difficult to predict for parsers. in addition, the presence of mechanisms such as topicalization as well as object voice @xcite @xcite allows verb-initial and verb-final word orders, even if the neutral word order of indonesian is svo @xcite @xcite . other than these issues, we explore three additional problems in detail in the following sections-the ambiguity in pos in indonesian, structural ambiguity in nps with demonstratives as well as difficulties in parsing coordinated structures. ## conclusion in conclusion, we have published icon, the largest publicly-available manually-annotated benchmark constituency treebank for the indonesian language with a size of 10,000 sentences and approximately 124,000 constituents and 182,000 tokens. as part of the process of building the treebank, we also reevaluated and revamped the constituent tagset and pos tagset in use in existing treebanks to ensure that the labels are relevant and suitable for the grammatical features of the indonesian language. in addition, we have established strong baselines on the icon dataset using the berkeley neural parser with transformer-based pre-trained embeddings, with our own indospanbert and the existing indolem giving f1 scores of 88.85% and 88.81% respectively. moving forward, there are still certain parts of the treebank that can be improved or are worth a second look. some possible aspects to be worked on are as follows: beyond improvements to the dataset, there are other research questions that could be explored as well: