Evaluating U niversal D ependency Parser Recovery of Predicate Argument Structure via C omp C hain Analysis (2021)

Abstract: Accurate recovery of predicate-argument structure from a Universal Dependency (UD) parse is central to downstream tasks such as extraction of semantic roles or event representations. This study introduces compchains, a categorization of the hierarchy of predicate dependency relations present within a UD parse. Accuracy of compchain classification serves as a proxy for measuring accurate recovery of predicate-argument structure from sentences with embedding. We analyzed the distribution of compchains in three UD English treebanks, EWT, GUM and LinES, revealing that these treebanks are sparse with respect to sentences with predicate-argument structure that includes predicate-argument embedding. We evaluated the CoNLL 2018 Shared Task UDPipe (v1.2) baseline (dependency parsing) models as compchain classifiers for the EWT, GUMS and LinES UD treebanks. Our results indicate that these three baseline models exhibit poorer performance on sentences with predicate-argument structure with more than one level of embedding; we used compchains to characterize the errors made by these parsers and present examples of erroneous parses produced by the parser that were identified using compchains. We also analyzed the distribution of compchains in 58 non-English UD treebanks and then used compchains to evaluate the CoNLL’18 Shared Task baseline model for each of these treebanks. Our analysis shows that performance with respect to compchain classification is only weakly correlated with the official evaluation metrics (LAS, MLAS and BLEX). We identify gaps in the distribution of compchains in several of the UD treebanks, thus providing a roadmap for how these treebanks may be supplemented. We conclude by discussing how compchains provide a new perspective on the sparsity of training data for UD parsers, as well as the accuracy of the resulting UD parses.

## introduction the universal dependencies (ud) project @xcite is a multilingual annotation scheme for dependency grammars that has gained wide usage @xcite @xcite . to this extent, automatically identifying whether a dependency parse foot_0 is correct or incorrect, as well as the potential source of such errors, becomes an important part of nlp pipelines. for example, such identification can prevent errors from propagating to downstream applications such as the identification of predicate-argument structure, involved in semantic role labeling and sentiment analysis. 2 furthermore, embedding of sentences within sentences, and in particular embedding of predicate-argument structures within one another, is one of the ways in which humans have the capability to generate an infinity of @xmath0, @xmath1, and so it is important to evaluate whether a ud parser can accurately recover the predicate-argument structure of sentences with embedding. thus, characterizing the limits of how accurately and consistently ud parsers assign predicate-argument structure in the context of correct ud annotation also becomes important @xcite @xcite @xcite . that is the goal of this study. in this study we introduce compchains, a categorization of the hierarchy of predicate dependency relations present within a universal dependency (ud) parse; this categorization serves as a proxy for predicate-argument structure. we use compchains to evaluate the accuracy of three (english) conll 2018 shared task baseline models for the udpipe dependency parser @xcite . we found that the baseline model for the ewt ud treebank was more accurate than the baseline models for the lines and gum ud treebanks. we then use compchains to characterize the errors (relevant to predicate-argument structure) made by these models. we found that the accuracy of all three models dropped significantly when restricting the test set to samples with predicate-argument structure with embedding. finally, we extended the analysis above to languages other than english, computing the distribution of compchains in 58 ud treebanks and evaluating the performance of the corresponding conll 2018 shared task baseline models (for the udpipe parser) as compchain classifiers. we conclude by discussing deficiencies in the distribution of predicate-argument structure with embedding present in the ud treebanks, as identified by our analysis. ## related work this section reviews prior work on the evaluation of (universal) dependency parsers and the characterization of the errors these parsers make. the conll shared task is a well established benchmark for evaluating the performance of multilingual (universal) dependency parsers @xcite @xcite . the task uses a number of metrics to evaluate the accuracy of the parser including: uas (unlabeled attached score), las (labeled attachment score), clas (content-word las) @xcite , mlas (morphologically-aware las) and blex (bilexical dependency score). however, these metrics rely on the attachment accuracy (of dependency relations) foot_2 and do not take into account that errors cascade -i.e. if the parser incorrectly attaches a dependency relation, it may then be forced to make yet another incorrect attachment @xcite , thus making it difficult to identify the provenance of the error. in light of this, efforts to further characterize the errors have proceeded in several directions. one direction involves studying whether and how the parsing errors are a result of the design of the dependency parser: @xcite characterizes and compares the errors produced by graph-based dependency parsers (e.g. the mst-parser by @xcite ; see also @xcite @xcite ) and transition-based dependency parsers (e.g. the maltparser by @xcite ); @xcite shows how the two approaches to dependency parsing may be combined and documents the resulting improvement in performance. an alternative direction involves characterizing the errors in the context of linguistic theory -e.g. @xcite has introduced a method for classifying erroneous parse trees by repairing the tree with a series of tree-transformations, with each tree-transformation having a linguistic interpretation; @xcite has shown that it is possible to systematically break nlp systems for sentiment analysis by editing sentences with linguistically interpretable transformations. in this study we pursue this latter direction, opting to characterize erroneous parse trees by classifying their predicate-argument structure using compchains. ## compchains within a ud parse tree, predicate-argument structure foot_3 is encoded by core argument dependency relations, along with the special dependency relation root. 5 the core-argument dependency relations fall into two classes: predicate relations and nominal relations. in this study, we limit our attention to the two predicate dependency relations that encode embedding of clausal complements: (i) ccomp -a dependent, clausal complement, and (ii) xcomp -a clausal complement lacking a subject; the subject is determined by an argument that is external to the xcomp, usually the object (or otherwise subject) of the next higher clause. 6 we will focus on categorizing sequences of these two dependency relations (with pos marked as verb) that originate from the root of a dependency tree (intuitively, the spine of the predicate-argument structure). this notion is formalized as follows: definition. a compchain is a finite sequence of dependency relations that traces a path starting at the root node of a dependency parse tree and passing through only xcomp and ccomp dependency relations, subject to the constraints that: (i) every node in a compchain must have the pos tag of verb; (ii) no node in a compchain should have a child dependency relation with pos verb that is either an xcomp or ccomp and is not in the compchain as well. 7 we denote a compchain by listing the sequence of dependency relations, starting from the root of the tree, using the notation: @xmath2; @xmath3; @xmath4. e.g. we would denote the compchain [@xmath5. see figure -1 for examples of ud parses and their compchain classifications. one way to evaluate (indirectly) how well a ud parser can identify predicate-argument structure for sentences in a ud treebank is to evaluate whether the ud parse assigned by the parser to a sentence in the treebank has the same compchain as the compchain associated with the gold 7 this constraint serves to ensure that if a ud parse tree has a compchain, it is unique and may be derived deterministically. this constraint also implies that some valid ud parse trees do not have a compchain -e.g. a parse in which there are two xcomp dependency relations that are both children of the same node. we use the symbol / 0 to denote that a ud parse tree has no compchain. ud parse listed for that sentence (in the treebank); we refer to this task as compchain classification. performance on the compchain classification task is a proxy for performance on the task of classifying predicate-argument structure that includes predicate-argument embedding. if a ud parser performs poorly on the compchain classification task, predicate-argument structure cannot be reliably recovered from an (output) ud parse tree via top-down traversal of the sequence of dependency relations that forms the associated compchain. see figure -2 for examples of incorrect compchain classifications that reflect the parser recovering incorrect predicate-argument structure. ## conclusion in this study, we defined compchains and used them to evaluate how accurately a ud parser can parse sentences with predicate-argument structure that contains embedded clauses. we also used compchains to classify the errors, relevant to predicateargument structure with embedding, made by a ud parser. overall model performance on the compchain classification task (as measured by weighted f-measure) was found to be dominated by parse trees in the training set with no embedding (compchain r); closer inspection of per-compchain performance revealed that parser accuracy dropped precipitously as the degree of embedding in the predicate argument structure (i.e. length of compchain) increased. finally, our results indicate that ud treebanks have very few parse trees with degree of embedding (i.e. length of compchain) greater than two. this presents an opportunity: if the test sets of the ud treebanks were augmented with parses with predicate-argument structure with degree of embeddings greater than two, then ud parsers can be evaluated in terms of their capacity to generalize from constructions (in the training set) with (mostly) low degree of embedding, just as a child must in some models of first language acquisition @xcite @xcite .