Fast End-to-end Coreference Resolution for K orean (2020)

Abstract: Recently, end-to-end neural network-based approaches have shown significant improvements over traditional pipeline-based models in English coreference resolution. However, such advancements came at a cost of computational complexity and recent works have not focused on tackling this problem. Hence, in this paper, to cope with this issue, we propose BERT-SRU-based Pointer Networks that leverages the linguistic property of head-final languages. Applying this model to the Korean coreference resolution, we significantly reduce the coreference linking search space. Combining this with Ensemble Knowledge Distillation, we maintain state-of-the-art performance 66.9% of CoNLL F1 on ETRI test set while achieving 2x speedup (30 doc/sec) in document processing time.

## introduction coreference resolution is one of the fundamental sub-tasks for machine reading comprehension and dialogue systems that groups mentions of a same entity in a given sentence or document @xcite @xcite . recently, for english coreference resolution, span-based end-toend trained models such as e2e-coref @xcite , c2f-coref @xcite , and bert-coref @xcite have shown to outperform previous rule-based or mention-pairing approaches. however, such approaches suffer from the computational complexity effectively-being o(n 4 ), where n is the length of the input document. furthermore, as coreference resolution is a very important and complicated task, most of the research efforts have been focused on how to solve the problem through better modeling, such as higher-order coreference resolution @xcite . inevitably, [훈민정음을 창제한 세종]은 [조선의 4대 왕]이다. one of the underlying reasons for such high computational complexity was the creation of o(n 2 ) spans caused by the mixed head directionality of english, as shown in figure 1 . this makes it hard to locate the heads in the mentions because the head location is not deterministic. on the other hand, having deterministic head locations is a very desirable linguistic trait for solving the aforementioned computational complexity issue. this effectively reduces the search space for coreference linking as we can use only the heads of the mentions. korean is not only a new domain for end-toend coreference resolution but also considered a strongly head-final language @xcite , which motivates us to focus on korean. in this paper, we present the first end-to-end model in korean coreference resolution. our model leverages such head-final properties using pointer net-works @xcite and achieves comparable performance to that of state-of-the-art models with a 2x speedup. our contributions can be summarized as the following: coreference resolution is basically about linking mention pairs (which are often noun phrases). essentially, this is finding heads of noun phrases that refer to the same entity, but the head locations within mentions are unknown. while previous rule-based approaches @xcite relied on several hand-engineered features including head-related ones, recent end-to-end methods attempt to directly model the mention distribution using span-based neural networks. ## bert-sru pointer networks we propose a novel model, bert-sru pointer network, that is suitable for head-final coreference resolution. this model combines bidirectional encoder representation from transformer (bert) @xcite with bidirectional simple recurrent units (srus) @xcite and pointer networks @xcite , as shown in figure 2 , to perform the head-final coreference resolution. initially, the encoder part (which is bert) receives morphologically analyzed texts along with their pos-tags as inputs. then the decoder extracts the hidden state corresponding to the head candidates (which are all nouns) and uses them as the inputs. after that, the gated self-attention layer in decoder models head information, and the decoder outputs position corresponding to the input using the pointer networks. we use deep biaffine @xcite as the attention score of the pointer networks, and this model performs both the mention detection and the coreference resolution. ## experiments dataset and measures we use the korean coreference resolution data @xcite from the etri quiz domain of aiopen 1 . pre-training korean bert bert consists of a bidirectional transformer encoder with several layers. for pre-training bert, we reuse the hyperparameters from @xcite . we used wikipedia and news data (total 23.5 gb) collected from the web. after performing morpheme analysis on all input words, tokenization was done on the subwords using bpe. the dictionary consists of 30,349 bpe tokens. we used the etri language 1 http://aiopen.etri.re.kr/ analyzer for morpheme analysis which is also available in aiopen as a tool for korean nlp. implementation hyper-parameters of the bert-sru-based pointer networks model are as follows. we fine-tune all models on the etri korean data for 70 epochs with a batch size of six for each gpu. the model trained on 2 geforce gtx 1080 ti gpu cards. the number of hidden layer dimensions and feature dimensions of the sru was optimized to 800 and 1,600, respectively. we have optimized the stack of the sru hidden layer to two. we set the dropout as 0.1. the training algorithm we used is adam (kingma and ba, 2014), and adam weight decay was set to 1 × 10 -2 . the learning rate was set to 5 × 10 -5 , and the linear method was used in the learning rate schedule. the maximum length of the input sequence was limited to 430 because the most extended input sequence length in the test set was 428. we used the etri language analyzer to obtain pos-tagging, ner, and dependency parsing features. head candidates in general, pointer networks' target outputs align with those of the decoder inputs. for our head-final coreference resolution, we set the inputs of the decoder as the list of head candidates. these head candidates are all nouns of the source document and they is extracted using the pos-tags. by doing so, we can effectively reduce the computational complexity to o(n 2 ), as the search for coreference links is only done between the head candidates. attention masking in coreference resolution, the antecedent at position i comes before the head at j, where @xmath0. similarly, in mention detection, the beginning boundary of a mention always appears before the head. accordingly, when calculating the attention score, we perform attention masking to prevent attention from being calculated for the element that is later than the j-th position. ## results in this section, we show our experimental results for korean coreference resolution. we denote the bert-sru-based ptr-net as our model for headbased coreference resolution. the performance of the models is measured and compared using the conll average f1-score. ## conclusion we propose head-based coreference resolution that reflects the head-final characteristics of korean and present a suitable bert-sru-based pointer networks model that leverages this linguistic trait. the proposed method, as the first end-to-end korean coreference resolution model, not only achieves state-of-the-art performance in the korean coreference resolution model through ensembling but also dramatically speeds up the document processing time compared to the conventional span-based coreference resolution. our method achieves this result by reducing the problem of coreference resolution from a search over span candidates to a search over head candidates using the fact that we can easily extract the mention heads for a head-final language. moreover, our proposed method of using headdirectionality to speed up coreference resolution while maintaining the best performance is valid for not only other strongly head-final languages like japanese, but also for strongly head-initial languages as the same method of head extraction can be applied. we believe that our paper also provides an interesting and important research direction. combining linguistic theories like headdirectionality and branching with deep learning has a strong potential of more efficiently and effectively model fundamental tasks like coreference resolution.