Enhanced U niversal D ependency Parsing with Automated Concatenation of Embeddings (2021)

Abstract: This paper describe the system used in our submission to the IWPT 2021 Shared Task . Our system is a graph-based parser with the technique of Automated Concatenation of Embeddings (ACE). Because recent work found that better word representations can be obtained by concatenating different types of embeddings, we use ACE to automatically find the better concatenation of embeddings for the task of enhanced universal dependencies. According to official results averaged on 17 languages, our system rank 2nd over 9 teams.

## introduction compared to the universal dependencies (ud) @xcite , the enhanced universal dependencies (eud) @xcite 1 makes some of the implicit relations between words more explicit and augments some of the dependency labels to facilitate the disambiguation of types of arguments and modifiers. the representation of eud is an enhanced graph with reentrancies, cycles, and empty nodes. such representation can represent richer grammatical relations than rooted trees, but it is harder to learn. to make the learning process relatively easy, we transfer the enhanced graph to a bi-lexical structure like annotation of semantic dependency parsing (sdp) @xcite by reducing reentrancies and empty nodes into new labels. therefore, many approaches for sdp can be adopted by eud. instead of the second-order parser that was used in previous work @xcite @xcite , we apply the biaffine parser @xcite which is one of the state-of-theart approaches of sdp for simplicity. recent developments on pre-trained contextualized embeddings have significantly improved the performance of structured prediction tasks in natural language processing. a lot of work has also shown that word representations based on the concatenation of multiple pre-trained contextualized embeddings and traditional non-contextualized embeddings (such as word2vec @xcite and character embeddings (santos and zadrozny, 2014)) can further improve performance @xcite @xcite . @xcite proposed automated concatenation of embeddings to automate the process of finding better concatenations of embeddings and further improved performance of many tasks. we utilize their method to find concatenations of pretrained embeddings as the input of the biaffine parser for eud. because there are many contextualized embeddings, such as xlmr @xcite , bert @xcite and flair @xcite , non-contextualized embeddings, such as word2vec @xcite , glove @xcite , and fasttext @xcite , and character embeddings @xcite . the search space of embeddings concatenation is large in size, besides, we need to train models of 17 languages respectively. following @xcite , we use reinforcement learning to efficiently find the better embeddings concatenation for each language. experimental results averaged on 17 languages show the effectiveness of our approach. our system is ranked 2nd over 9 teams in the official evaluation. ## conclusion our system is a parser with automated embeddings concatenation and a biaffine encoder. empirical results show the effectiveness of ace to enhanced universal dependencies. our system ranks 2nd over 9 teams according to the official elas.