A Corpus Study of Creating Rule-Based Enhanced U niversal D ependencies for G erman (2021)

Abstract: In this paper, we present a first attempt at enriching German Universal Dependencies (UD) treebanks with enhanced dependencies. Similarly to the converter for English (Schuster and Manning, 2016), we develop a rule-based system for deriving enhanced dependencies from the basic layer, covering three linguistic phenomena: relative clauses, coordination, and raising/control. For quality control, we manually correct or validate a set of 196 sentences, finding that around 90% of added relations are correct. Our data analysis reveals that difficulties arise mainly due to inconsistencies in the basic layer annotations. We show that the English system is in general applicable to German data, but that adapting to the particularities of the German treebanks and language increases precision and recall by up to 10%. Comparing the application of our converter on gold standard dependencies vs. automatic parses, we find that F1 drops by around 10% in the latter setting due to error propagation. Finally, an enhanced UD parser trained on a converted treebank performs poorly when evaluated against our annotations, indicating that more work remains to be done to create gold standard enhanced German treebanks.

## introduction universal dependencies (ud; de @xcite are a widely used framework for syntactic dependency annotation. aiming to provide a crosslingual representation that focuses on relations between content words, nearly 200 ud treebanks for over 100 languages are currently available. 1 an extension of ud are enhanced universal dependencies @xcite , which have since become an official part of the ud guidelines @xcite . enhanced ud augments "basic" ud syntax trees by including additional dependency edges in the presence of linguistic phenomena such as coordination, relative clauses, and raising/control. as a result, enhanced ud structures are not necessarily trees, but more general syntax graphs. for example, in figure 1b , the basic dependency tree (black edges on top) is augmented by an enhanced nsubj link (blue edge at bottom) in order to represent that the subject of "warten", "eltern", has been raised to the object position of "empfahl." enhanced ud representations have been shown to be useful for downstream nlp tasks @xcite , and parsing them has experienced a rise of attention in the scope of two recent shared tasks @xcite . yet, annota-tions for this dependency representation are still only available for a small subset of ud treebanks. 2 many of these existing annotations were created using rule-based systems operating on the basic layer with little to no human supervision/evaluation, or via conversion from other syntactic representations @xcite . in this paper, we take a first step towards extending enhanced ud coverage to german treebanks, for which no enhanced annotations exist to date. we give details on our findings when generating enhanced ud representations from the basic layer of german ud treebanks in a rule-based way, similarly to the work of @xcite for english. we focus on three linguistic phenomena (relative clauses, raising/control, and coordination) and present rule-based algorithms for each phenomenon that take into account the peculiarities of the german language, which has less strict rules for word order compared to english. we assess the accuracy of the extractions made by our system with the help of expert human annotators, evaluating system performance on both gold as well as automatically parsed basic dependencies. in the gold setting, we find that around 90 % of added relations are correct, outperforming the system by @xcite . on automatically parsed text, performance drops by around 10 %, indicating a reliance on correct basic dependencies. a graph parser trained on automatically extracted enhanced dependencies achieves low accuracy, indicating that more work remains to be done to automatically create high-quality enhanced german treebanks. to the best of our knowledge, our work presents the first investigation into creating enhanced ud annotations for german treebanks. our system achieves accuracies which we believe may already be useful in downstream applications (e.g., to increase recall in information extraction tasks). furthermore, we hope that our findings may serve as a starting point for the creation of high-quality enhanced ud corpora for the german language. to facilitate further research, our code and annotations will be made freely available. foot_1 converter_law_dmr_2021 2 related work in this section, we briefly review work on extracting enhanced ud from the basic representation, as well as enhanced ud parsing and related graph-based meaning representations. ## basic-to-enhanced converter in this section, we describe the implementation of our basic-to-enhanced ud converter for german. along the way, we discuss important differences w.r.t. the converter for english @xcite . ## analysis and evaluation in this section, we first describe our manual corpus study to measure the performance of our converter. we compute precision, recall and f1 on enhanced links and draw comparisons to the english converter by @xcite . in addition, we evaluate the performance of the converter in automatic parsing settings. ## conclusion and outlook in this paper, we have described our approach of enriching german ud treebanks with enhanced dependencies by implementing a rule-based converter. evaluating on a set of manually annotated sentences, our system achieves an f1 score of around 90% when operating on gold basic dependencies, outperforming the english-based system by @xcite . in an automatic parsing setting, accuracy is lower with an f1 score of around 80 %, which may still be useful in practice. training an enhanced parser on treebanks created using our converter yields poor results. in our annotation study, we found that our system fails mainly due to errors in the basic layer, which have been derived semi-automatically from german treebanks annotated with different dependency frameworks. for example, an xcomp relation in the wrong direction leads to severe errors in the enhanced layer when following our rule-based approach. we further observed divergences in regards to the annotation schemes applied across german treebanks, for instance the usage of different relation types for the same sentence structures. we therefore conclude that measures to increase consistency across german treebanks are an important step to facilitate the conversion to enhanced dependencies. outlook. there remains a number of ambiguous cases for which we have not found a solution thus far, such as the propagation of objects in case of two conjoined verbs (see sec. 3.2). regarding raising and control structures, examples like "wir lassen den jungen gehen" ("we let the boy leave") vs. "wir lassen den reifen montieren" ("we have the tyre mounted") are identical regarding their basic dependency structure and pos tags, yet should be treated differently by the converter. in control structures such as "wir entschieden, hier nicht zu schwimmen" ("we decided not to swim here,") the subject of the embedded verb "schwimmen" ("swim") is the same as that of "entschieden" ("decided"), while in the sentence "wir empfehlen, hier nicht zu schwimmen" ("we recommend not to swim here,") the subject does not co-refer with the speaker. a possibility to improve here might be the usage of word lists containing control/raising verbs for which the subject of the controlled/embedded predicate can not co-refer with the speaker (as for "recommend"). to conclude, our work demonstrates that rulebased conversion is a suitable method for creating enhanced dependencies for german. however, our analysis also shows that taking language-specific phenomena into account is important to ensure correctness of the extractions. hence, creating targeted enhanced ud converters for more languages is a promising avenue, but will likely require further manual work for each instance.