Overview of the IWPT 2020 Shared Task on Parsing into Enhanced U niversal D ependencies (2020)

Abstract: This overview introduces the task of parsing into enhanced universal dependencies, describes the datasets used for training and evaluation, and evaluation metrics. We outline various approaches and discuss the results of the shared task.

## introduction universal dependencies (ud) @xcite is a framework for cross-linguistically consistent treebank annotation that has so far been applied to over 90 languages. ud defines two levels of annotation, the basic trees and the enhanced graphs (eud). in 2017 @xcite and 2018 @xcite there were conll shared tasks on multilingual ud parsing that attracted a substantial number of participants. while the previous tasks evaluated morphology and prediction of basic dependencies on the ud data, the current task's focus is on predicting enhanced dependency representations. the evaluation was done on datasets covering 17 languages from four language families. the current task was organized as a part of the 16th international conference on parsing technologies foot_0 (iwpt), collocated with acl 2020, as a follow-up to stimulate research on parsing natural language into richly annotated structures. ## motivation the basic dependency annotation in the universal dependencies format introduces labeled edges between tokens in the input string, where each token is a dependent of exactly one other token, with the exception of the root token. while such an annotation layer supports many downstream tasks, there are also phenomena that are hard to capture using single edges between tokens only. the enhanced dependency layer therefore supports a richer level of annotation, where tokens may have more than one parent, and where additional 'empty' tokens may be added to the input string. the enhanced level can be used to account for a range of linguistic phenomena (see section 3) and to support downstream applications that require representations that capture more aspects of the semantic interpretation of the input. there are now a number of treebanks that include enhanced dependency annotation. furthermore, the recent shared tasks on dependency parsing and subsequent work have shown that considerable progress has been made in multilingual dependency parsing. it remains to be seen, however, whether the same is true for enhanced dependency parsing. the challenge is both formal and practical. first, the enhanced representation is a connected graph, possibly containing cycles, while previous work on dependency parsing mostly dealt with rooted trees. second, as some dependency labels incorporate the lemma of certain dependents and other additional information, the set of labels to be predicted is much larger and language-dependent. on the other hand, it has been shown that much of the enhanced annotation can be predicted on the basis of the basic ud annotation @xcite . moreover, most state of the art work in dependency parsing uses a graph-based approach, where the assumption that the output must form a tree is only used in the final step from predicted links to final output. and finally, work on deep-syntax and semantic parsing has shown that accurate mapping of strings into rich graph representations is possible @xcite @xcite and could even lead to state of the art performance for downstream applications as shown by the results of the extrinsic evaluation parsing shared-task @xcite . all enhancements are optional, so a ud treebank may contain enhanced graphs with one type of enhancement and still lack the other types. ## data the evaluation was done on 17 languages from 4 language families: arabic, bulgarian, czech, dutch, english, estonian, finnish, french, italian, latvian, lithuanian, polish, russian, slovak, swedish, tamil, ukrainian. the language selection is driven simply by the fact that at least partial enhanced representation is available for the given language. sue has 5 euros , pat 6 and kim 3 nsubj conj conj obj nummod punct orphan cc orphan sue has 5 euros , pat 6 and kim 3 nsubj conj conj obj nummod punct nsubj obj cc nsubj obj figure 2: the correct enhanced graph of the gapping structure from figure 1. " " are empty nodes. training and development data were based on the ud release 2.5 @xcite but for several treebanks the enhanced annotation is richer than in ud 2.5. our goal was to have annotations as uniform and complete as possible. there are only 6 treebanks of 3 languages in ud 2.5 that contain all types of enhancements: dutch (alpino and lassysmall), english (ewt and pud), and swedish (talbanken and pud). for several other languages we obtained new annotations that became part of ud from the next release (2.6) on. for the remaining languages, we applied simple heuristics and added at least some enhancements for the purpose of the shared task, but these annotations are not yet part of the regular ud releases. we only applied our heuristics to the missing enhancement types; we did not attempt to modify the enhancements provided by the data providers. the enhancements differ in how easily and accurately they can be inferred from the basic ud annotation: treebank ud 2.5 task 2.6 arabic padt ps gps rc bulgarian btb psxrc gpsxrc czech cac ps ## task as in the previous dependency parsing shared tasks, participants were expected to go from raw, untokenized, strings to full dependency annotation. the evaluation focused on the enhanced annotation layer, but the participants were encouraged to predict all annotation layers, and the evaluation of the other layers is available on the shared task website. 3 the task was open, in the sense that participants were allowed to use any additional resources they deemed fit (with the exception of ud 2.5 test data) as long as this was announced in advance and the additional resource was freely available to everybody. the submitted system outputs had to be valid conll-u files; if a file was invalid, its score would be zero. 4 the official ud validation script 5 was used to check validity, although only at 'level 2', which means that only basic file format was checked and not the annotation guidelines (e.g., an unknown relation label would not render the file invalid). still, certain aspects of level-2 validity complicate the prediction of the enhanced graphs, and as the participants were not alerted to individual restrictions beforehand, these restrictions were an unwelcome surprise to them. so the relations can be unknown but can only contain characters from a limited set. the enhanced graph can contain cycles, but not self-loops (a node depending on itself). and most crucially, there must be at least one root node and every node must be reachable via a directed path from at least one root node (rootedness and connectedness). when we saw during the test phase that some teams might not be able to comply with these restrictions, we created a quick-fix script that tries to make the submission valid; however, the solution the script provided for unconnected graphs is not optimal. in addition to conll-u validity, we also required that systems do not alter any non-whitespace characters when processing the input. this is a pre-requisite for the evaluation, where systempredicted tokens must be aligned with goldstandard tokens; files with modified word forms would be rejected. ## evaluation metrics the main evaluation metric is elas (labeled attachment score on enhanced dependencies), where elas is defined as f1-score over the set of enhanced dependencies in the system output and the gold standard. complete edge labels are taken into account, i.e. obl:on differs from obl. a second metric is eulas, which differs from elas in that only the universal part of the dependency relation label is taken into account. relation subtypes are ignored, i.e., obl:on, obl:auf, and obl are treated as identical. as is apparent from a final issue we address is the evaluation of empty nodes. a consequence of the treatment of gapping and ellipsis is that some sentences contain additional nodes (numbered 1.1 etc.). it is not guaranteed that gold and system agree on the position in the string where these should appear, but the information encoded by these additional nodes might nevertheless be identical. thus, such empty nodes should be considered equal even if their string index differs. to ensure that this is the case, we have opted for a solution that basically compiles the information expressed by empty nodes into the dependency label of its dependents. i.e. if a dependent with dependency label l2 has an empty node i2.1 as parent which itself is an l1 dependent of i1, its dependency label will be expanded into a path i1:@xmath0. this preserves the infor-mation that the dependent was an l2 dependent of 'something' that was itself an l1 dependent of i1, while at the same time removing the potentially conflicting i2.1 (figure 5 ). 6 ## approaches there is quite a bit of variation in the way various teams have addressed the task. for the initial stages of the analysis (tokenization, lemmatization, postagging) some version of udpipe foot_4 @xcite , udify foot_5 @xcite , and/or stanza foot_6 @xcite ) is often involved. several teams (orange (heinecke, 2020), fast-parse @xcite , unipi @xcite , clasp @xcite , adapt @xcite ) concentrate on parsing into standard ud, and then add hand-written enhancement rules, sometimes in combination with data-driven heuristics to improve robustness. turkunlp @xcite transforms eud into a representation that is compatible with standard ud by combining multiple edges into a single edge with a complex label, and compiling edges involving empty nodes into complex edge labels (as is done by the evaluation script as well). the total number of edge-labels is reduced by de-lexicalising enhanced edge labels and storing a pointer to the dependent from which the lemma of an enhancement originates in the de-lexicalized edge label. a wide range of parsers (graph-based biaffine, transitionbased), and pre-trained embeddings (xlm-r or mbert or language specific berts) is used. finally, several teams (emory nlp @xcite , shanghaitech @xcite , adapt, køpsala @xcite , robertnlp @xcite ) do not use conversion (or only to restore de-lexicalized labels), but instead use a graph-based parser that can directly produce enhanced dependency graphs. the output of the graph-based parser is often combined with information from a standard ud parser to ensure well-formedness and connectedness of the resulting graph. ## results we include two baseline results: foot_7 baseline1 was obtained by taking gold basic ud trees and copying these into the enhanced layer without any modifications. baseline2 uses udpipe 1.2 trained on ud 2.5 treebanks foot_8 and again copies basic ud to the enhanced layer. both baselines give an impression of how much the enhanced layer differs from the basic layer, where baseline1 makes the unrealistic assumption that parsing into basic ud is perfect. if we compare scores for las, eulas, and elas, it can be observed that usually there is a small drop in accuracy when going from las to eulas to elas, although the drop from las/eulas to elas seems to be larger for some of the systems in the lower half of the table. this suggests that predicting the correct label enhancement is problematic for some approaches. the eulas and elas scores for the qualitative evaluation (which takes into account differences in the enhancement level of treebanks) are only slightly higher than in the coarse evaluation. it should be noted though, that scores cannot be compared directly, as the coarse evaluation is a macro average over languages, whereas most scores in the qualitative evaluation are macro averages over treebanks. this implies that the data is weighted slightly differently in both averages, which plays a role in the las scores being generally a bit higher in the qualitative evaluation. when the qualitative elas is averaged over languages (the elas-l column in difference between coarse and qualitative evaluation is small. this is due to (a) the fact that this makes a difference for 9 of 28 treebanks only and (b) the fact that some of the phenomena that are ignored in the qualitative evaluation are relatively rare in the data (e.g. ellipsis). a number of teams have submitted runs on the test data after the deadline for the official evaluation, an overview in given in ## conclusions this shared task was the first attempt at a coordinated evaluation effort on parsing enhanced universal dependencies. while a large part of the methodology could be adopted from the previous conll shared tasks on parsing into ud, a number of issues did require attention. first, providing training and test data is complicated by the fact that not all treebanks in the ud repository include the same level of enhancements. this makes training a single, multilingual, model, harder than it ought to be, as annotation style differs per treebank. for evaluation, different enhancement levels pose a problem as it is unclear to what extent 'overannotating' data should be considered an error. as another issue for validation is the status of 'empty' nodes. the position in the string of such nodes is not defined by the guidelines, and therefore one may expect mismatches between gold and system data. our solution to this issue is described in section 6. for future tasks, however, it might be worthwhile to investigate whether a different representation of such nodes in the data files or an alternative evaluation strategy is needed. several systems struggled with the validation requirements of enhanced ud. while an enhanced graph may contain nodes with more than one parent, may contain cycles, and may have multiple root nodes, there are still constraints that an enhanced ud graph must comply with, such as that the graph must be connected and that there should be one or more 'root' nodes from which all other nodes are reachable. in future tasks, the restrictions should be more carefully described in advance. the results of the shared task illustrate that there is quite a wide variety in the way that the problem of parsing into enhanced universal dependencies can be approached, with some systems sticking closer to traditional approaches for parsing ud, and dealing with the enhancements in a conversion script, while other systems output a graph directly. the scores indicate that while parsing into enhanced ud is harder than parsing into ud, the drop in performance is minimal for most systems, which suggests that the challenges posed by the annotation format of enhanced ud are not an obstacle for accurate parsing.