From Raw Text to Enhanced U niversal D ependencies: The Parsing Shared Task at IWPT 2021 (2021)

Abstract: We describe the second IWPT task on end-to-end parsing from raw text to Enhanced Universal Dependencies. We provide details about the evaluation metrics and the datasets used for training and evaluation. We compare the approaches taken by participating teams and discuss the results of the shared task, also in comparison with the first edition of this task.

## introduction universal dependencies (ud) @xcite is a framework for cross-linguistically consistent treebank annotation that has so far been applied to 114 languages. ud defines two levels of annotation, the basic trees and the enhanced graphs (eud) @xcite . there are several good parsers that can predict the basic trees (including tokenization and morphology) for previously unseen text @xcite . two large shared tasks on basic ud parsing were organized at conll @xcite . enhanced ud parsing attracted comparatively less attention until the shared task organized at iwpt 2020 @xcite . the present paper describes a second instance of that task, organized as a part of the 17th international conference on parsing technologies foot_0 (iwpt), collocated with acl-ijcnlp 2021. like in the previous year, the evaluation was done on datasets covering 17 languages from four language familiies. this paper is a follow-up of the overview paper of the previous instance of the shared task @xcite . to make the paper self-contained, we include updated versions of some sections of that paper, in particular describing the enhanced annotation format, the task, and the evaluation metric. the data section now documents the modifications we made to the data from ud release 2.7. ## motivation the basic dependency annotation in the universal dependencies format introduces labeled edges between nodes that represent tokens in the input string, where each node is a dependent of exactly one other node, with the exception of the node token. while this tree structure supports many downstream tasks, there are also phenomena that are hard to capture using single-parent edges only. the enhanced dependency layer therefore supports richer annotation where nodes may have more than one parent, and where additional 'empty' nodes represent elided material that is not overtly expressed in the input string. the enhanced level can be used to account for a range of linguistic phenomena (see section 3) and to support downstream applications that rely on the semantic interpretation of the input. there are now a number of treebanks that include enhanced dependency annotation. furthermore, the recent shared tasks on dependency parsing and subsequent work have shown that considerable progress has been made in multilingual dependency parsing. for enhanced dependency parsing, there are additional challenges. the enhanced representation is a connected directed graph, possibly containing cycles, while the bulk of dependency parsing work still focuses on rooted trees. the set of labels to be predicted is also much larger, as some enhanced dependency labels incorporate the lemma of certain dependents. on the other hand, it has been shown that much of the enhanced annotation can be predicted on the basis of the basic ud annotation @xcite @xcite . moreover, most state-of-the-art work in dependency parsing uses a graph-based approach, where the assumption that the output must form a tree is only used in the final step from predicted links to final output. and finally, work on deep-syntax and semantic parsing has shown that accurate mapping of strings into rich graph representations is possible @xcite @xcite and could even lead to state-of-the-art performance for downstream applications as shown by the results of the extrinsic parsing evaluation shared task @xcite . the previous iwpt shared task @xcite reflected this development quite well: some submissions took the way of direct text-to-graph mapping, some of them predicted a rooted tree and then employed heuristics to enhance it; and one submission encoded graphs as trees, then used a tree parser to predict them. since it was the first task of its kind on large scale multilingual enhanced dependencies parsing and some teams may not have been able to successfully implement all their ideas in time (or new ideas may have occurred after seeing what other teams had done), a second round of the task is a natural next step to see whether we can do even better. ## enhanced universal dependencies ud version 2 foot_1 states that apart from the morphological and basic dependency annotation layers, strings may be annotated with an additional, enhanced, dependency layer, where the following phenomena can be captured: all enhancements are optional, so a ud treebank may contain enhanced graphs with one type of enhancement and still lack the other types. ## data the evaluation was done on 17 languages from 4 language families: arabic, bulgarian, czech, dutch, english, estonian, finnish, french, italian, latvian, lithuanian, polish, russian, slovak, swedish, tamil, ukrainian. the language selection is driven simply by the fact that at least partial enhanced representation is available for the given language. training and development data were based on the ud release 2.7 (zeman et al., 2020) but for several treebanks the enhanced annotation is richer than in ud 2.7. besides improvements in the officially released versions of the individual treebanks, a few other things have changed in comparison to the iwpt 2020 task. the english data now includes the gum treebank (its enhanced annotation was not present in ud 2.7 but it was being prepared for ud 2.8 and it was ready in time for the shared task). as in 2020, we include two french treebanks whose enhanced annotation is still not included in the official ud releases, but the annotation is more conservative this year, omitting the extra labels for diathesis neutralization @xcite and surface vs deep syntax markers. still, some enhancements in french go slightly beyond the official ud guidelines (see below for details). in polish, we now harmonize the relation subtypes in the three treebanks so that merging them into one dataset is no longer an issue. finally, we omit the chukchi treebank, which is new in ud 2.7 and has enhanced graphs, but the graphs are there only sue has 5 euros , pat _ 6 and kim _ 3 nsubj conj conj obj nummod punct nsubj obj cc nsubj obj figure 2: the correct enhanced graph of the gapping structure from figure 1. "_" are empty nodes. to provide empty nodes to capture incorporated modifiers (rather than gapping); furthermore, the treebank is too small and has no training data. there are 13 treebanks of 7 languages in ud 2.7 that contain all types of enhancements: czech (cac, fictree, pdt, and pud), dutch (alpino and lassysmall), english (ewt and pud), italian (isdt), lithuanian (alksnis), slovak (snk), and swedish (talbanken and pud). for the remaining languages, we applied simple heuristics and added at least some enhancements for the purpose of the shared task, but these annotations are not yet part of the regular ud releases. we only applied our heuristics to the missing enhancement types; we did not attempt to modify the enhancements provided by the data providers. the enhancements differ in how easily and accurately they can be inferred from the basic ud annotation: although the ud releases distinguish several different treebanks for some languages, for the purpose of the shared task evaluation we merged all test sets of each language. we wanted to promote robust parsers that are not tightly tied to one particular dataset. merging treebanks of one language was possible because for almost all languages it holds that treebanks participating in the present task are maintained by the same team, hence no significant treebank-specific annotation decisions are expected. the exceptions are english and polish but there should not be any significant divergence in these languages either. in english, the gum corpus is maintained by other people than ewt and pud; nevertheless, the corpora use the same set of relations, and there are ongoing efforts to harmonize the way the relations are used. in polish, the lfg treebank uses a different set of relation subtypes than pdb and pud; however, this year we removed the subtypes that are not used in all three treebanks, so it should be possible to train a parser on one treebank and successfully apply it to another. for instance, the percentage of basic dependencies that have a different label in the enhanced graph (mostly because of adding the case information to obl and other relations), ranges from 0 to 27%. enhanced dependencies that introduce truly novel edges are rarer. in the table they are again expressed relatively to the number of basic dependencies, and the figure varies between 2 and 13%. up to 2% basic edges are omitted in the enhanced graph. there are slight differences in how individual languages implement particular enhancement types. some languages follow earlier proposals for enhanced relation subtypes that are not supported by the current ud guidelines, e.g., external subjects are labeled nsubj:xsubj, antecedents of relative clauses are nsubj:relsubj or obj:relobj, the "case" information is extended to showing conjunction lemma with conjuncts (conj:and, conj:or etc.) empty nodes are occasionally used for other ellipsis types than gapping or stripping. the adding of relations from relative clauses to modified nouns is further extended in french to infinitival and participial adnominal clauses, as in figure 5 . 3 upon completion of the shared task, the data has been made publicly available at the permanent address http://hdl.handle.net/11234/1-3728 . ## task as in the previous dependency parsing shared tasks, participants were expected to go from raw, untokenized strings to full dependency annotation. the evaluation focused on the enhanced annotation layer, but the participants were encouraged to pre-dict all annotation layers, and the evaluation of the other layers is available on the shared task website. 4 the task was open, in the sense that participants were allowed to use any additional resources they deemed fit (with the exception of ud 2.7 test data) as long as this was announced in advance and the additional resource was freely available to everybody. the submitted system outputs had to be valid conll-u files; if a file was invalid, its score would be zero. 5 the official ud validation script foot_5 was used to check validity, although only at 'level 2', which means that only basic file format was checked and not the annotation guidelines (e.g., an unknown relation label would not render the file invalid). constraints that have to be met at this level are that there must be at least one root node and every node must be reachable via a directed path from at least one root node (rootedness and connectedness), that the enhanced graph can contain cycles, but not self-loops (a node depending on itself), and that dependency labels can only contain characters from a limited set. in addition to conll-u validity, we also required that systems do not alter any non-whitespace characters when processing the input. this is a pre-requisite for the evaluation, where systempredicted tokens must be aligned with goldstandard tokens; files with modified word forms would be rejected. ## evaluation metrics the main evaluation metric is elas (labeled attachment score on enhanced dependencies), where elas is defined as f 1 -score over the set of enhanced dependencies in the system output and the gold standard. complete edge labels are taken into account, i.e. obl:on differs from obl. a second metric is eulas, which differs from elas in that only the universal part of the dependency relation label is taken into account. relation subtypes are ignored, i.e., obl:on, obl:auf, and obl are treated as identical. another issue we address is the evaluation of empty nodes. a consequence of the treatment of gapping and ellipsis is that some sentences contain additional nodes (numbered 1.1 etc.). it is not guaranteed that gold and system agree on the position in the string where these should appear, but the information encoded by these additional nodes might nevertheless be identical. thus, such empty nodes should be considered equal even if their string index differs. to ensure that this is the case, we have opted for a solution that basically compiles the information expressed by empty nodes into the dependency label of its dependents. i.e. if a dependent with dependency label l2 has an empty node i2.1 as parent which itself is an l1 dependent of i1, its dependency label will be expanded into a path i1:@xmath6. this preserves the information that the dependent was an l2 dependent of 'something' that was itself an l1 dependent of i1, while at the same time removing the potentially conflicting i2.1 (figure 6 ). 7 finally, to analyze results, we computed elas scores per phenomenon. this should be seen as a diagnostic only, and is intended to gain further insights into the capability of various systems to deal with challenging phenomena, such as the proper analysis of phenomena occurring in the context of coordination and ellipsis. ## approaches the predominant approach to obtaining the enhanced dependency graph is to use a biaffine function, i.e., predicting for each pair of nodes how likely it is that they are in a parent-child relation. there is wide variety in the way the final annota-tion graph is obtained, and ensuring that the result is valid (i.e. connected). grew @xcite uses manually constructed rewrite rules to map basic ud into eud, while fast-parse @xcite and nuig @xcite reformulate the task as a sequence-labeling task. for the initial stages of the analysis (sentence splitting, tokenization, lemmatization, postagging) most teams use stanza @xcite or trankit @xcite or similar methods. in a post-evaluation experiment, the dcu-epfl team @xcite obtained improved scores using trankit instead of stanza, while the tgif team @xcite uses a variation of the trankit and stanza systems to obtain the best pre-processing results, especially for sentencesplitting. a wide variety of monolingual and multilingual pre-trained language models is used, with xml-r @xcite being the most popular. the shanghaitech system @xcite learns an input representation from a combination of pretrained language models where the various representations are concatenated into a single vector and masking is used to learn a weighting for various components of the combined vector. both combo @xcite and unipi @xcite use a method that learns weights for the scores obtained from various layers of the bert model to be used as input for the biaffine parser. most teams reduce the number of edge labels during training by de-lexicalizing edge labels. dependency paths involving an empty node are usually also replaced by concatenating the path labels into a single path, as is also done in the evaluation script, thereby removing the need to predict empty nodes. ## results the nuig submission was incomplete, in that the results for some languages were missing. the submissions of tgif and shanghaitech contain dummy annotations for all annotation layers except eud, so no las is provided. las and elas correlate strongly, with elas generally being 3-4% lower than las, except for dcu-epfl, whose elas beats las. the best system in the first edition of this shared task @xcite obtained a elas of 84.50, while the current highest scoring system obtains an elas of 89.24. the average of elas of the top-5 was 78.75 for the first edition, while the current top-5 has an average of 86.14. the higher scores are most likely both due to more uniform annotations across treebanks as described in section 4 and improvements in approaches. for the first edition of this task @xcite we provided a qualitative evaluation, where scores were computed per treebank, while taking into account that some treebanks do not include all enhancements stated in the guidelines in their enhanced layer. this year, as the annotation is considerable more uniform across treebanks, we decided to concentrate on performance per enhancement type. we used a script that labeled each edge in all best scores for 2021 were obtained by tgif except for arabic (shanghaitech). 1 : english compares the score for the ewt and pud treebanks (2020) with ewt+pud+gum (2021). 2 : french compares the scores between the 2021 more simple annotation scheme and the 2020 more complex original proposal. the enhanced annotation as belonging to one of the phenomena or enhancement types listed in ## conclusions the second edition of the shared task for parsing into enhanced universal dependencies shows improvements at various levels. first of all, the same set of languages was included as for the first edition, b basic this enhanced edge is identical to an edge in the basic tree (including the label) c cased case-enhanced relation (the relation with the shorter label may or may not exist in the basic tree) l relabeled the same two nodes are also connected in the basic tree but the label is different and the difference does not look like a case enhancement g gapping the parent or the child is an empty node note that for systems that only provide dummy annotations for basic ud, some of the scores cannot be computed in a meaningful way. the nuig system was not included as it lacked results for some languages. but now we were using treebanks of ud release 2.7 @xcite . this eud annotation of this release is more consistent and according to guidelines than the data of release 2.5, but we still had to harmonize some of the annotations so that differences in annotation would not have a negative effect on system performance. second, the requirement that submitted annotations should be minimally valid according to the guidelines, was now more easily met by all participating teams. teams ensured that graphs would be connected, for instance, by applying several heuristics that introduce the minimal amount of additional edges to meet connectedness. third, while the best performing system in the first shared task used a method that pre-compiled the enhanced annotation graph into a tree, compatible with basic ud, and used a standard dependency parsing algorithm for learning to produce such annotations, almost all systems in this years shared task went for a graph-based approach. there still is quite a bit of variation in the way the graph is constructed though, with some systems first producing a tree, and then adding additional edges, where others try to produce the graph directly. at the same time, most systems do apply some form of pre-compilation to make the data more suitable for learning. in particular, case-enhanced dependency labels are replaced by de-lexicalized labels that can be easily reconstructed in postprocessing. similarly, most teams adopt a method that removes 'empty' nodes and instead expresses the information in incoming and outgoing edges from these nodes in the form of complex dependency labels (as is done in the evaluation script as well). finally, a very positive outcome of this evaluation is that scores have increased considerably, not only for the top performing system, but also for the top-5 systems. in particular, lower performance now seems to be restricted to languages for which very limited amounts of data is available, and, as