Analysis of the P enn K orean U niversal D ependency Treebank ( PKT - UD ): Manual Revision to Build Robust Parsing Model in K orean (2020)

Abstract: In this paper, we first open on important issues regarding the Penn Korean Universal Treebank (PKT-UD) and address these issues by revising the entire corpus manually with the aim of producing cleaner UD annotations that are more faithful to Korean grammar. For compatibility to the rest of UD corpora, we follow the UDv2 guidelines, and extensively revise the part-of-speech tags and the dependency relations to reflect morphological features and flexible word- order aspects in Korean. The original and the revised versions of PKT-UD are experimented with transformer-based parsing models using biaffine attention. The parsing model trained on the revised corpus shows a significant improvement of 3.0% in labeled attachment score over the model trained on the previous corpus. Our error analysis demonstrates that this revision allows the parsing model to learn relations more robustly, reducing several critical errors that used to be made by the previous model.

## introduction in 2018, @xcite published on three dependency treebanks in korean that followed the latest guidelines from the universal dependencies (ud) project, that was udv2. these treebanks were automatically derived from the existing treebanks, the penn korean treebank (pkt; @xcite ), the google ud treebank @xcite , and the kaist treebank @xcite , using head-finding rules and heuristics. this paper first addresses the known issues in the original penn korean ud treebank, henceforth pkt-ud v2018, through a sampling-based analysis (section 3), and then describes the revised guidelines for both part-of-speech tags and dependency relations to handle those issues (section 4). then, a transformer-based dependency parsing approach using biaffine attention is introduced (section 5) to experiment on both pkt-ud v2018 and the revised version, henceforth pkt-ud v2020 (section 6). our analysis shows a significantly reduced number of mispredicted labels by the parsing model developed on pkt-ud v2020 compared to the one developed on pkt-ud v2018, confirming the benefit of this revision in parsing performance. the contributions of this work are as follows: ## observations in pkt-ud v2018 the penn korean treebank (pkt) was originally published as a phrase-structure based treebank by @xcite . pkt consists of 5,010 sentences from korean newswire including 132,041 tokens. 3 following the udv2 guidelines, @xcite systematically converted pkt to pkt-ud v2018. while this effort achieved a measure of success at providing phrase-structure-to-dependency conversion in a manner consistent across three different treebanks with distinct grammatical frameworks, it stopped short of addressing more nuanced issues that arise from aligning grammatical features of korean, that is a heavily agglutinative language, to the universal standards put forth by udv2. in building pkt-ud v2018, the pos tags were largely mapped in a categorical manner from the penn korean pos tagset. the dependency relations on the other hand were established via head-finding rules that relied on penn korean treebank's existing function tags, phrasal tags, and morphemes. @xcite did make a few targeted attempts at teasing apart more fine-grained nuances of grammatical functions. for example, the pkt pos tag (xpos) dan was subdivided into the ud pos tag (upos) det for demonstrative prenominals (e.g., 이 (this), 그 (the), and the upos adj for attributive adjectives (e.g., 새 (new), 헌 (old)) in the recognition that the xpos dan, focusing primarily on grammatical distribution, conflated two semantically distinct elements. however, such efforts were limited in scope, and the project did not examine the full breadth of language-specific issues. moreover, the converted annotation was found 3 while most korean resources have what is known as eojeol representing a token and white space is used as delimiter, ptk tokenizes apart symbols, punctuation and even occasional morphemes where strictly required by syntactic structure. to contain a share of mechanical errors. a case in point, what should have been 5,010 sentences were found to contain 5,036 roots, suggesting low-level parsing errors. additionally, a manual examination of the first five sentences in the corpus uncovered a variety of syntactic errors that raised an alarm. the worst of the five examined sentences is shown in figure 1 (and continued in figure 2 ) with errors in both the upos and the dependency relation labels (deprel). while we will not delve into particulars of each error seen in this example, the example provides a general sense for the extent of errors existent that merited our attention. these observed issues inspired us to revise pkt-ud v2018, with the aim of producing cleaner syntactic annotations that would be more faithful to the korean grammar. the following section provides specifics of the revision content. ## parsing approach our dependency parsing model is based on the biaffine parser using contextualized embeddings such as bert @xcite that has shown the state-of-the-art results on both syntactic and semantic dependency parsing tasks in multiple languages @xcite . this model is simplified from the original biaffine parser introduced by @xcite such that trainable token embeddings are removed and lemmas are used instead of word forms. this section proposes an even more simplified model that no longer uses embeddings from pos tags, so it can be easily adapted to languages that do not have dedicated pos taggers, and drops the bidirectional lstm encoder while integrating the transformer layers directly into the biaffine decoder so that it minimizes the redundancy of having multiple encoders for the generation of contextualized embeddings. given an input sentence, every token w i is first segmented into one or more sub-tokens by the sen-tencepiece tokenizer @xcite and fed into a transformer. the output embedding corresponding to the first sub-token of w i is treated as the embedding representation of w i , say e i , and fed into four types of multilayer perceptron (mlp) layers to extract features for w i being a head (*-h) or a dependent (*-d) for the arc relations (arc-*) and the labels (rel-*) (k and l are the dimensions of the arc and label representations, respectively): all feature vectors, h * 1 , . . . , h * n , from each representation are stacked into a matrix (n is the number a j e k a v 6 e 1 7 0 l 6 0 d + 1 j 1 l r s i p l 9 9 a f a 5 w + e / j g @xmath0 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath1 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath2 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath3 mlp (rel-d) < l a t e x i t s h a 1 _ b a s e 6 4 = " r i m 4 m 6 c q c 6 g q i s t n h i x j x p i q p w h e m 6 c w e + q p j 8 w e 6 u z g @xmath4 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath5 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath6 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath7 t o e j b f j 3 f i y t c 4 y 0 5 k 9 8 g f g 5 w + 0 o z g @xmath8 t o e j b f j 3 f i y t c 4 y 0 5 k 9 8 g f g 5 w + 0 o z g @xmath9 t o e j b f j 3 f i y t c 4 y 0 5 k 9 8 g f g 5 w + 0 o z g @xmath10 a q u h 6 2 r + n s i j p 5 s a 8 / v n c o t 1 b q 3 f p / z 6 h f 2 j p o x 9 8 m i m u / h i z q r s d c w h o l y b s 4 z r t h q h f d j 9 a 0 w 7 r f j k o r c t b 2 c m / 3 y l k k c f b y 7 4 f z z u e i p j j g g l o x a h h w 4 h c k c q w n k q o e b n u a f x o 1 h 4 9 l 4 m 9 7 h r s l j m r m n f 2 b 8 / a a u w 5 n @xmath11 a j e k a v 6 e 1 7 0 l 6 0 d + 1 j 1 l r s i p l 9 9 a f a 5 w + e / j g @xmath12 a j e k a v 6 e 1 7 0 l 6 0 d + 1 j 1 l r s i p l 9 9 a f a 5 w + e / j g @xmath13 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath14 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath15 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath16 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath17 a q u h 6 2 r + n s i j p 5 s a 8 / v n c o t 1 b q 3 f p / z 6 h f 2 j p o x 9 8 m i m u / h i z q r s d c w h o l y b s 4 z r t h q h f d j 9 a 0 w 7 r f j k o r c t b 2 c m / 3 y l k k c f b y 7 4 f z z u e i p j j g g l o x a h h w 4 h c k c q w n k q o e b n u a f x o 1 h 4 9 l 4 m 9 7 h r s l j m r m n f 2 b 8 / a a u w 5 n @xmath18 a j e k a v 6 e 1 7 0 l 6 0 d + 1 j 1 l r s i p l 9 9 a f a 5 w + e / j g @xmath19 a j e k a v 6 e 1 7 0 l 6 0 d + 1 j 1 l r s i p l 9 9 a f a 5 w + e / j g @xmath20 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath21 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath22 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath23 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath24, . . . , u rel m ) c 6 p 7 w e + / v 7 u j q h 8 h z h e b q d 2 p e 8 x x l f i 3 x s g / j d u m g h p g c i q j w o 0 p m w h l h a s f n u 0 z 3 a w s t e j o t j d j w o / d x q h i w o q c i t v o u m 5 0 b y t q h c z g s k u v a s i a j s s p v q n f t s a h q 7 m d y f o s d g 6 t q 9 u j m s 6 e z u 3 x m j d b q e b b 7 p d c g o 9 l w 3 f v / z m j h 2 l t s j l 1 g m i n l 0 u s 8 w d o b o o a y n y x u w f c n d k f p c 3 o q w a v w u o y k s z 0 l w 5 l 9 e j l w z o u c w v z v z f o l q f k e w h j i j u i a e u s a l u i y v u i w m p j j n 8 k r e r c f r x x q 3 p q a t g w s 2 s 0 / + w p r 8 a q j j l h @xmath25 c 6 p 7 w e + / v 7 u j q h 8 h z h e b q d 2 p e 8 x x l f i 3 x s g / j d u m g h p g c i q j w o 0 p m w h l h a s f n u 0 z 3 a w s t e j o t j d j w o / d x q h i w o q c i t v o u m 5 0 b y t q h c z g s k u v a s i a j s s p v q n f t s a h q 7 m d y f o s d g 6 t q 9 u j m s 6 e z u 3 x m j d b q e b b 7 p d c g o 9 l w 3 f v / z m j h 2 l t s j l 1 g m i n l 0 u s 8 w d o b o o a y n y x u w f c n d k f p c 3 o q w a v w u o y k s z 0 l w 5 l 9 e j l w z o u c w v z v z f o l q f k e w h j i j u i a e u s a l u i y v u i w m p j j n 8 k r e r c f r x x q 3 p q a t g w s 2 s 0 / + w p r 8 a q j j l h @xmath26 c 6 p 7 w e + / v 7 u j q h 8 h z h e b q d 2 p e 8 x x l f i 3 x s g / j d u m g h p g c i q j w o 0 p m w h l h a s f n u 0 z 3 a w s t e j o t j d j w o / d x q h i w o q c i t v o u m 5 0 b y t q h c z g s k u v a s i a j s s p v q n f t s a h q 7 m d y f o s d g 6 t q 9 u j m s 6 e z u 3 x m j d b q e b b 7 p d c g o 9 l w 3 f v / z m j h 2 l t s j l 1 g m i n l 0 u s 8 w d o b o o a y n y x u w f c n d k f p c 3 o q w a v w u o y k s z 0 l w 5 l 9 e j l w z o u c w v z v z f o l q f k e w h j i j u i a e u s a l u i y v u i w m p j j n 8 k r e r c f r x x q 3 p q a t g w s 2 s 0 / + w p r 8 a q j j l h @xmath27 m m / p 8 m d m f t v 1 t f j a w v 1 b x i u u l j c 2 t 7 r 1 z d 6 + l o k q s 2 i q r j 2 t h x 4 p y j m g t g h d a i s x f o c 9 p 2 x 9 d t / z 2 p z w k r e i w x j h 1 q j w q l g a e g 5 z 6 5 k h 9 l n w b p k b a w z k c d k 8 y f 6 i 4 6 5 l l u 2 p p y s 0 s j y d l l k p r m 7 / c f k s s k a o g h c v v d e w y v b r l y i t t r o q m i s a y j p c a d j u v o k t k s 6 f 3 z 9 a x v v p w e e l d a q y p + n s i x a f s 4 9 d x n s g g o z r 3 j u j / x j e b 4 n j l m y g t o i l m f g u j t y c y j m f y f s y p a t 7 w b b p j 9 k 0 w g w k j c e j i s j o e z / 7 l r d i 6 q z p 2 1 b k 5 l 9 e u 8 j i k 6 b a d o q p y 0 a w q o t p q o c y i 6 b e 9 o 1 f 0 z j w z l 8 a 7 8 t f r l r j 5 z d 7 6 a + p z b + 9 u l g @xmath28 m m / p 8 m d m f t v 1 t f j a w v 1 b x i u u l j c 2 t 7 r 1 z d 6 + l o k q s 2 i q r j 2 t h x 4 p y j m g t g h d a i s x f o c 9 p 2 x 9 d t / z 2 p z w k r e i w x j h 1 q j w q l g a e g 5 z 6 5 k h 9 l n w b p k b a w z k c d k 8 y f 6 i 4 6 5 l l u 2 p p y s 0 s j y d l l k p r m 7 / c f k s s k a o g h c v v d e w y v b r l y i t t r o q m i s a y j p c a d j u v o k t k s 6 f 3 z 9 a x v v p w e e l d a q y p + n s i x a f s 4 9 d x n s g g o z r 3 j u j / x j e b 4 n j l m y g t o i l m f g u j t y c y j m f y f s y p a t 7 w b b p j 9 k 0 w g w k j c e j i s j o e z / 7 l r d i 6 q z p 2 1 b k 5 l 9 e u 8 j i k 6 b a d o q p y 0 a w q o t p q o c y i 6 b e 9 o 1 f 0 z j w z l 8 a 7 8 t f r l r j 5 z d 7 6 a + p z b + 9 u l g @xmath29 m m / p 8 m d m f t v 1 t f j a w v 1 b x i u u l j c 2 t 7 r 1 z d 6 + l o k q s 2 i q r j 2 t h x 4 p y j m g t g h d a i s x f o c 9 p 2 x 9 d t / z 2 p z w k r e i w x j h 1 q j w q l g a e g 5 z 6 5 k h 9 l n w b p k b a w z k c d k 8 y f 6 i 4 6 5 l l u 2 p p y s 0 s j y d l l k p r m 7 / c f k s s k a o g h c v v d e w y v b r l y i t t r o q m i s a y j p c a d j u v o k t k s 6 f 3 z 9 a x v v p w e e l d a q y p + n s i x a f s 4 9 d x n s g g o z r 3 j u j / x j e b 4 n j l m y g t o i l m f g u j t y c y j m f y f s y p a t 7 w b b p j 9 k 0 w g w k j c e j i s j o e z / 7 l r d i 6 q z p 2 1 b k 5 l 9 e u 8 j i k 6 b a d o q p y 0 a w q o t p q o c y i 6 b e 9 o 1 f 0 z j w z l 8 a 7 8 t f r l r j 5 z d 7 6 a + p z b + 9 u l g @xmath30 m m / p 8 m d m f t v 1 t f j a w v 1 b x i u u l j c 2 t 7 r 1 z d 6 + l o k q s 2 i q r j 2 t h x 4 p y j m g t g h d a i s x f o c 9 p 2 x 9 d t / z 2 p z w k r e i w x j h 1 q j w q l g a e g 5 z 6 5 k h 9 l n w b p k b a w z k c d k 8 y f 6 i 4 6 5 l l u 2 p p y s 0 s j y d l l k p r m 7 / c f k s s k a o g h c v v d e w y v b r l y i t t r o q m i s a y j p c a d j u v o k t k s 6 f 3 z 9 a x v v p w e e l d a q y p + n s i x a f s 4 9 d x n s g g o z r 3 j u j / x j e b 4 n j l m y g t o i l m f g u j t y c y j m f y f s y p a t 7 w b b p j 9 k 0 w g w k j c e j i s j o e z / 7 l r d i 6 q z p 2 1 b k 5 l 9 e u 8 j i k 6 b a d o q p y 0 a w q o t p q o c y i 6 b e 9 o 1 f 0 z j w z l 8 a 7 8 t f r l r j 5 z d 7 6 a + p z b + 9 u l g @xmath31 the bilinear and biaffine classifiers are then used for the arc and label predictions respectively, where u (arc) , u (rel) i and v (rel) are trainable parameters, and m is the number of dependency labels. in particular, a separate weight matrix u (rel) i is dedicated for the prediction of each label: once the arc score matrix s (arc) and the label score tensor s (rel) are generated by those classifiers, the chu-liu-edmond's maximum spanning tree (mst) algorithm is applied to s (arc) for the arc prediction, then the label with largest score in s (rel) corresponding to the arc is taken for the label prediction: @xmath32, parsing models are separately developed on pkt-ud v2018 and v2020; in other words, v2018 models are trained and evaluated on pkt-ud v2018 whereas v2020 models are trained and evaluated on pkt-ud v2020. the transformer-based parsing approach in section 5 is used to develop all models. for each version of the corpus, three models are developed by initializing neural weights with different random seeds and the average accuracy and its standard deviation is reported for each version. the entire corpus is divided into the training (trn), development (dev), and evaluation (tst) sets by following the 80/10/10% split ( the multilingual bert foot_2 is used as the transformer encoder in our parsing models @xcite . all models are optimized by the sum of softmax cross-entropy losses on the gold dependency heads and labels. adamw @xcite is used as the optimizer with the learning rate of 5e-06 for the bert weights and 5e-05 for the rest. the learning rate is scheduled as a combination of both linear warm-up and decay phases. the models are trained for 100 epochs with a batch size of 150. following the standard practice, we evaluate our best models with the unicode punctuation ignored using the unlabeled attachment score (uas) and the labeled attachment score (las). ## error analysis pkt-ud v2018 we perform an error analysis on the parsing outputs generated by the v2018 model. our analysis shows that the head error occurred in 1,360 eojeols and the label error occurred in 4,292 eojeols. ## conclusion in this study, we revise the penn korean universal dependency treebank (pkt-ud) and compare parsing performance between models trained on the original and revised versions of pkt. our new guidelines follow the udv2 guidelines. upos and deprel are revised to reflect korean morphological features and flexible word-order aspects with reference to korean ud studies such as @xcite , @xcite . in upos, adj, noun, and verb are revised extensively. in deprel, iobj, acl, advcl, and goeswith are thoroughly revised. the revision results showing the percentage change of each label are presented in as a result of the parsing experiment, the v2020 model improves uas by 0.6% and las by 3.0% over the v2018 model. in particular, obl, acl, nummod, and advcl errors are significantly reduced. this study, which improves parsing accuracy by applying characteristics of korean, can also contribute to improve the quality of other korean ud treebanks. in the future, we will explore the possibility of extending pkt-ud with enhanced dependency types foot_3 by incorporating empty categories from the original pkt.