The DCU - EPFL Enhanced Dependency Parser at the IWPT 2021 Shared Task (2021)

Abstract: We describe the DCU-EPFL submission to the IWPT 2021 Parsing Shared Task: From Raw Text to Enhanced Universal Dependencies. The task involves parsing Enhanced UD graphs, which are an extension of the basic dependency trees designed to be more facilitative towards representing semantic structure. Evaluation is carried out on 29 treebanks in 17 languages and participants are required to parse the data from each language starting from raw strings. Our approach uses the Stanza pipeline to preprocess the text files, XLM-RoBERTa to obtain contextualized token representations, and an edge-scoring and labeling model to predict the enhanced graph. Finally, we run a postprocessing script to ensure all of our outputs are valid Enhanced UD graphs. Our system places 6th out of 9 participants with a coarse Enhanced Labeled Attachment Score (ELAS) of 83.57. We carry out additional post-deadline experiments which include using Trankit for pre-processing, XLM-RoBERTa LARGE, treebank concatenation, and multitask learning between a basic and an enhanced dependency parser. All of these modifications improve our initial score and our final system has a coarse ELAS of 88.04.

## introduction the iwpt 2021 parsing shared task: from raw text to enhanced universal dependencies @xcite is the second task involving the prediction of enhanced universal dependencies (eud) graphs foot_0 following the 2020 task @xcite . eud graphs are an extension of basic ud trees, designed to be more useful in shallow natural language understanding tasks @xcite and lend themselves more easily to the representation of semantic structure than strict surface structure dependency trees. in the shared task, the enhanced graphs must be predicted from raw text, i.e. participants must segment the input into sentences and tokens. participants are encouraged to predict lemmas, part-of-speech (pos) tags, morphological features and basic dependency trees as well. our system, dcu-epfl, uses a single multilingual transformer @xcite encoder, namely xlm-roberta (xlm-r) @xcite , which is a multilingual roberta model @xcite , to obtain contextualized token encodings. these are then passed to the enhanced dependency parsing model. the system is straightforward to apply to new languages with enhanced ud annotations. in the official submission, we use the same hyper-parameters for all languages. our parsing component can produce arbitrary graphs, including graph structures where words may have multiple heads and cyclic graphs. our system uses the following three components: 1. stanza @xcite for sentence segmentation, tokenization and the prediction of all ud features apart from the enhanced graph. our official system placed 6th out of 9 teams with a coarse enhanced labeled attachment score (elas) of 83.57. in a number of unofficial postevaluation experiments, we make four incremental changes to our pipeline approach: all of these additional steps improved our evaluation scores, and for our final system, which incorporates all additional modifications, our evaluation score increases from 83.57 to 88.04. our code is publicly available. foot_1 in this section, we discuss the relevant literature related to enhanced universal dependencies. ## official system overview this section describes our official system, which is the system we submitted prior to the competition deadline. the architecture of our system is cls s1 s2 s3 s4 s5 sep z1 z2 z3 z4 z5 z6 z7 xlm-roberta ... ... eud score function eud decoding mechanism eud decoder basic score function basic decoding mechanism basic decoder input sentence pre-processing post-processing eud output graph basic output graph multi-task shown in figure 1 . 3 the raw text test files for each language contain a mixture of test data covering multiple treebanks, so participants do not know their exact domain. for our official system, we choose the model trained on the treebank with the most amount of training data in terms of sentences for each language to process the test files. this heuristic corresponds to using czech-pdt for czech, dutch-alpino for dutch, english-ewt for english, estonian-edt for estonian and polish-pdb for polish. ## experiments in this section, we discuss our official results and then describe post-deadline experiments that improved our submission's score. model hyperparameters are listed in ## conclusion we have described the dcu-epfl submission to the iwpt 2021 shared task on parsing into enhanced universal dependencies. our approach uses a single multilingual transformer encoder as well as an enhanced dependency parsing component. our official system placed 6th out of 9 teams. in post-deadline experiments, we show how our submission can be improved by leveraging better upstream pre-processing, a larger encoder, concatenating treebanks as well as introducing a multitask parser that can parse the basic tree and enhanced graphs jointly.