A U niversal D ependencies Corpora Maintenance Methodology Using Downstream Application (2021)

Abstract: This paper investigates updates of Universal Dependencies (UD) treebanks in 23 languages and their impact on a downstream application. Numerous people are involved in updating UD’s annotation guidelines and treebanks in various languages. However, it is not easy to verify whether the updated resources maintain universality with other language resources. Thus, validity and consistency of multilingual corpora should be tested through application tasks involving syntactic structures with PoS tags, dependency labels, and universal features. We apply the syntactic parsers trained on UD treebanks from multiple versions (2.0 to 2.7) to a clause-level sentiment extractor. We then analyze the relationships between attachment scores of dependency parsers and performance in application tasks. For future UD developments, we show examples of outputs that differ depending on version.

## introduction universal dependencies (ud) @xcite is a worldwide project that provides cross-linguistic treebank annotations. ud defined 17 pos tags and 37 dependency labels to annotate multilingual sentences in a uniform manner, allowing language-specific extension to be represented by features. the resources and documents are updated every six months. the latest version 2.7, as of november 2020, consists of 183 treebanks in 104 languages. the ud corpora are consistently annotated in multiple languages and are extensively used to train and evaluate taggers and parsers @xcite . @xcite trained a single dependency model for many languages relying on ud corpora. @xcite used universal pos (upos) labels to evaluate multilingual sentence representations. however, few studies have focused on the contributions of syn-figure 1 : our methodology to get insights from the difference of the corpora on the flow of the multilingual sentiment annotator. the components in red dashed lines are variable, while solid ones are fixed. tactic parsers trained by ud corpora to real-world applications. extrinsic evaluation of dependency parser has been already studied in a series of shared tasks @xcite using tasks of event extraction, negation detection and opinion analysis for english documents. in addition to extrinsic evaluation of parsers, @xcite established a method for evaluating the universality of ud-based parsers and ud corpora by using a clause-level sentiment extractor, which detects positive and negative predicates and targets on top of ud-based syntactic trees. they showed that language-universal syntactic structures and features are effective in their multilingual systems. in this paper we investigate how the ud corpora and underlying guidelines are updated and how they contribute to the parser and sentiment extractor which consumes the output of the parser. we compared ud versions 2.0 to 2.7 foot_0 in 23 languages from diverse language families. figure 1 shows the proposed methodology. the idea is to use corpora with sentence-level sentiment annotations (sa) in two ways: 1) we can compare sa results considering different syntactic models; 2) we can compare the sa annotation with the golden sentiment annotation. the first one is useful for qualitative analysis. the second one is useful for quantitative analysis, given that we can measure the sa efficiency. first, we trained a dependency parsing model for each ud corpora version (ud release) using a fixed syntactic parser. using the models, we produce as many syntactic analyses as models for each corpus with sentence-level sentiment annotations. later, we applied a deterministic rule-based sentiment annotator for each syntactic tree. the advantage of this methodology is that it is much easier to find sentiment annotation errors than syntactic annotation errors, and those errors often show the essential aspects of syntax. comparing the gold sentiment annotation of input and final output, we can quantitatively estimate the usefulness of parsing models, moreover, a qualitative analysis of system outputs provides practical insights for corpora maintainers. in particular, inspecting the output of a sentiment analysis system for discovering possible annotation inconsistencies is one important additional advantage. we found examples where improvements in the corpus have led to improvements in the output of the sentiment annotator (reducing the number of uses of the dep relation and minimizing the errors reported by the ud validator). but some examples can be also found where change in the corpus had made a negative impact in the sentiment analysis (section 5). we use a different measure (f 2 ) to extrinsically evaluate the ud corpora. it is not directly related to the intrinsic ud measures such as star rating for ud corpora and las for the dependency parser. section 2 summarizes the changes of the ud corpora in versions 2.0-2.7. section 3 describes the sentiment analysis methodology which is used for benchmarking dependency parsers. in section 4 we show how to evaluate multilingual systems, and in section 5, we discuss the differences of multiple versions of ud corpora with multilingual instances of changes in syntactic structures and downstream results. ## universal dependencies universal dependencies is a framework for designing and maintaining consistent syntactic annotations across multiple languages. the ud corpora are updated every six months by numerous contributors. however, few studies have focused on the changes in outputs of ud-trained parsers used for application tasks. labeled attachment score (las) and the ud star rating are two commonly used metrics to evaluate the update of the ud corpora. las is a measure of the performance of dependency parsers, where the universal dependency labels are taken into account in the measurement. the star rating is a measure designed by ud organizers, which quantifies the qualities of the corpora themselves, such as usability of corpora and variety of genres. while the ud corpora and the parsers have been evaluated, there is a need for an external evaluation of ud in application tasks. to explore the impacts caused by updates of ud corpora on the sentiment analysis task, we first investigate the changes in the ud corpora listed in figure 2 shows the ud treebanks updates for versions 2.0 (march 2017) to 2.7 (november 2020) in 23 languages. inspecting the amount of changes between versions for each treebank was done regarding six out of the ten fields in the conll-u files (form, lemma, upostag, feats, head, and deprel). most languages have been actively updated in versions 2.0-2.7. in versions 2.0-2.4, most of the modifications in the ud corpora focused on fundamental syntactic elements such as pos tags and dependency labels, and universal features were incrementally appended. on the other hand, in versions 2.4-2.6, the major updates shifted towards language-specific features. through discussion across languages, the ud's annotation policy is gradually becoming more consistent among close languages. pos tags for copulas and auxiliary verbs are one typical examples of this: "be" in "have been" and "will be" were changed from verb to aux in english v2.5, as well as "hebben" in dutch v2.4. in addition, there is a movement to make aux a closed set. in portuguese v2.5, many aux were changed to verb, e.g., "continuar" ('continue'), "deixar" ('leave'). similarly, french v2.1 and onward limit aux to "être", "avoir" and "faire,". "pouvoir" ('can') and other words in the same category are tagged as verb, even though english modal verbs are tagged as aux. ## multilingual clause-level sentiment analysis we investigate changes in ud corpora and their impact on an application task. we use clause-level sentiment analysis designed for fine-grained sentiment detection with high precision. kanayama and iwamoto (2020) demonstrated that a system which fully utilizes ud-based syntactic structures can easily handle many languages, making it an effective platform for evaluating ud corpora and parsing models trained on them. the main objective of clause-level sentiment analysis is to detect polar clauses associated with a predicate and a target. for example, the sentence (1) below conveys two polarities: (1a) a positive polarity regarding the hotel (which is loved) and (1b) a negative polarity about the waiters (who are not friendly). figure 3 illustrates the top-down process of detecting sentiment clauses in the dependency tree. the main clause is headed by the root node of the dependency tree. when the node has child nodes labeled conj and parataxis, those nodes are recursively scanned as potential sentiment clauses. when a node is a verb that takes a ccomp (clausal compliment) child, e.g., "say", the child node is also examined. in example (1), two clauses, headed by "love" and "friendly" are detected. after detecting the clauses, the predicates are compared with lexical entries associated with a lemma, a pos tag and its polarity and the case frame, as exemplified in in all of the languages, detecting negation is the key to detecting polarities with high precision. the basic types of negation are direct negation of the verb and noun in (3) and ( 4 ). to multilingualize the clause-level sentiment detector, the english polarity lexicon shown in ## experimental settings to extrinsically evaluate the ud corpora, we combine a ud-compliant dependency parser trained with multiple versions of ud corpora to the senti-ment extractor. since the syntactic structure is the only factor that changes the output of sentiment detection, we can easily find the effects of parsing to the downstream application. ## conclusion we observed updates of the ud corpora versions 2.0-2.7 in 23 languages and extrinsically evaluated the parsing models trained by the corpora in a realworld scenario. the evaluation using the sentiment extractor with ud-trained parsers do not correlate clearly with existing evaluations such as las and star rating, indicating that evaluation using an application task is useful to measure ud corpus from a new perspective. we showed examples where the updates of ud corpora have either adversely or positively affected the output of dependency parsing and sentiment clause detection. our methodology is easier to find the changes of sentiment detection. those changes often show the important aspects of syntax. we identified issues in multilingual applications of the ud platform. for example, some corpora have less diverse writing styles for informal sentences which are more common in review documents. in some languages, ud corpora updates have been slowed down after version 2.4 and shifted towards language-specific features and augmented dependencies, but there are still open problems in fundamental syntactic structures. we anticipate continuous improvements to multilingual corpora for ud communities worldwide. we hope the emergence of other applications that utilize ud's syntactic structures will lead to further discussions and enhancements of multilingual corpora.