Adaptation of Multilingual Transformer Encoder for Robust Enhanced U niversal D ependency Parsing (2020)

Abstract: This paper presents our enhanced dependency parsing approach using transformer encoders, coupled with a simple yet powerful ensemble algorithm that takes advantage of both tree and graph dependency parsing. Two types of transformer encoders are compared, a multilingual encoder and language-specific encoders. Our dependency tree parsing (DTP) approach generates only primary dependencies to form trees whereas our dependency graph parsing (DGP) approach handles both primary and secondary dependencies to form graphs. Since DGP does not guarantee the generated graphs are acyclic, the ensemble algorithm is designed to add secondary arcs predicted by DGP to primary arcs predicted by DTP. Our results show that models using the multilingual encoder outperform ones using the language specific encoders for most languages. The ensemble models generally show higher labeled attachment score on enhanced dependencies (ELAS) than the DTP and DGP models. As the result, our best models rank the third place on the macro-average ELAS over 17 languages.

## introduction dependency parsing can generate computational structures for a wide range of typologically different languages, which provides structural relations that have been found to be useful for various nlp applications. however, these applications often require richer dependency relations carrying on deep semantics, which are missing in traditional dependency trees. thus, enhanced dependencies emerge to explicitly capture deep semantic relations over surface structures @xcite . recently, there has been lots of interests in constructing and parsing advanced graph structures beyond tree representations. @xcite introduce deep dependency graphs that address several limitations in ud tree structures. @xcite analyze gapping constructions in the enhanced ud representation. @xcite evaluate both rule-based and data-driven systems for adding enhanced dependencies to existing treebanks. apart from syntactic relations, researchers are moving towards semantic dependency parsing @xcite for more direct analysis of entities and events. the efforts of treebank construction stimulates the interest of many researchers in improving the performance of semantic parsers @xcite @xcite . this paper presents our parsing approach to the shared task on enhanced universal dependencies at iwpt 2020 @xcite ). 1 our system is a simplified version of the transformer-based dependency parsers presented by @xcite , which employs the deep biaffine dependency parsing decoder @xcite over the transformer encoder, bert @xcite . we simplify their network by removing the lstm and fine-tuning their static transformer encoder. in order to effectively predict the enhanced dependencies, we also ensemble the dependency tree parser with an dependency graph parser through a greedy searching algorithm. at last, we perform extensive experiments on the effects of transformer encoders to perform a detailed analysis. our experiments show that the multilingual encoder has a substantial advantage over the languagespecific encoders. moreover, our analysis shows that tree parsing model can accurately predict primary dependencies in long sentences, while graph parsing model excels at label prediction. by taking advantages from both sides, our ensemble models outperform individual models in most languages. 2 mlp (rel-h) < l a t e x i t s h a 1 _ b a s e 6 4 = " i j g z h n a e v d m 1 u s w w 9 w m q o d k m 1 n @xmath0 e m n m p j i n s k r e t o e j b f j 3 f i y t c 4 y 0 5 k 9 8 g f g 5 w + 0 o z g @xmath1 a a 6 s t a p a q j l 1 e j t r n a j e k a v 6 e 1 7 0 l 6 0 d + 1 j 1 l r s i p l 9 9 a f a 5 w + e / j g @xmath2 a a 6 s t a p a q j l 1 e j t r n a j e k a v 6 e 1 7 0 l 6 0 d + 1 j 1 l r s i p l 9 9 a f a 5 w + e / j g @xmath3 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath4 e m n m p j i n s k r e t o e j b f j 3 f i y t c 4 y 0 5 k 9 8 g f g 5 w + 0 o z g @xmath5 a j e k a v 6 e 1 7 0 l 6 0 d + 1 j 1 l r s i p l 9 9 a f a 5 w + e / j g @xmath6 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath7 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath8 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath9 w h e m 6 c w e + q p j 8 w e 6 u z g @xmath10 ## analysis this section analyzes factors that affect our models the most, common error made across languages and what possible improvement that can be made. ## conclusion this paper describes our parsing approach to enhanced universal dependencies for the iwpt 2020 shared task. we find that the multilingual bert encoder is able to parse various languages without language specific network design. our proposed ensemble method is shown to be beneficial for the secondary dependency prediction. in the future, we will improve the secondary dependency prediction in a more systematic way. we believe our current approach generating dependency graphs satisfying the tree constraint of primary dependencies can be further improved if the constraint can be applied to biaffine attention during training time, and the msdags constraint can be relaxed for better performance. we leave these exciting topics for future work.